I assume you know the chances that an average linux user actually get exploited in that way is very low.
Don't expect that to last. Windows users get hosed by web browser and plugin bugs a lot. Right now it isn't worth the trouble to set up similar exploits for Linux systems.
So, right now I do not need SELinux even if I "use a web browser to view more than a short list of trusted sites".
Expect that to change as the malware toolkits get better.
I don't.