I figured out my problem, it's the whole confusing what's part of src.fedoraproject.org and what's part of pagure.io... You need both tokens for different actions but one goes in:

[fedpkg.pagure]
token = <paguire.io token>

[fedpkg.distgit]
token = <src.fp.o token>

Thanks,

Richard