Authenticate `su -` through PAM and SSH Agent

[Justin W]

I'd appreciate it if anybody would tell me if there is a way to use PAM 
to authenticate a user when using `su -`. After reading about ssh agent 
a while ago, I've had this idea that it'd be a perfect way to add 
another layer of security around my server. Currently I use ssh keys to 
log in remotely as my user. In my mind, using ssh agent instead of 
passwords would be the next logical step.

I'm not very familiar with ssh agent and rsa authentication methods 
though. I've got a rough idea of what I'm looking for, but one of you 
may be able to direct to me why my plan isn't safe/wouldn't work. This 
is what I've been thinking:

I currently have my ssh key for my user allowing me to login to my 
server (encrypted with a password of course). I enable ssh agent 
forwarding in PuTTY or ssh (depending on whether I'm in Windows or FC6 
at the time). I'd then issue a `su -` at the prompt and it'd 
authenticate against the ssh agent I have running locally, and if I 
didn't have it running, I'd be denied access.

Thanks for any suggestions or help,
Justin W