So a lot of servers only come with 2d only video cards which requires that a llvm emulator of 3d to be around for various things that KDE and/or Gnome need. Having recently had to deal with an app that needed a local X this was extremely painful to try (the Raspberry pi without closed source drivers had better video performance than this). The app ran fine since it relied on gtk2 but the rest of the desktop to try and get the app working was visible redraw city. It was in the end faster to X11 forward to the laptop even though buttons didn't redraw correctly etc. 

The speed issues are a problem that the desktop team aren't interested in which is part of the reason we have different products. With the fact that server admins end up using oddball desktops (i3, xfce, lxde, e18, etc etc.) for lack of 3d reasons..  I am not sure the long fights over which one gets selected because its the ligthest weight are going to be short either. 

Thus I am more for the let the admin install it via a yum command when they need it and choose what they think is best.