I figured out my problem, it's the whole confusing what's part of src.fedoraproject.org and what's part of pagure.io... You need both tokens for different actions but one goes in:
[fedpkg.pagure] token = <paguire.io token>
[fedpkg.distgit] token = <src.fp.o token>
Thanks, Richard