I had promised to do this and post my results a week ago and got thoroughly tied up over the holidays - sorry about that. It was a good Christmas for us though! :)
So - I did get around to loading up a server with the latest version of F10 (32-bit in this case) to run the 32-bit version of VMware Server 2.0 (build 122956) to try and answer the burning question: Does selinux need to be disabled for VMware Server to run properly on F10?
I know the inpatient out there can't wait to read the whole post, so here's the answer:
Yes.
According to our testing (a friend of mine who also frequents this list was here too), the current version of VMware Server DOES NOT RUN on F10 (32-bit) unless selinux is DISABLED. Permissive mode doesn't cut it - it still causes VMware Server to not run.
Here are the details: Server: "Whitebox" Supermicro 1U chassis, dual 2.4GHz Pentium Xeon processors, 4GB RAM, Dual Gig-E NICs, dual 250GB IDE drives
OS: F10 32-bit, with all patches as of 12-28-08 Kernel: 2.6.27.9-159.fc10 (PAE version - required to see the full 4GB)
We loaded a fresh copy F10 with all of the required development tools and supporting stuff VMware Server needs to compile, and left selinux in its default (enforcing) mode and targeted policy. The system was intentionally updated with all of the latest available patches. After rebooting (kernel update that included a switch to the PAE kernel), we then installed VMware Server from the RPM via Package Kit. The initial RPM install went as expected with no errors or issues beyond the warning that the RPM is not signed (Request to VMware: Please, PLEASE make sure that you always sign your RPMs!).
Next up was to configure the system. We fired up a terminal window, switched user to root, and then launched vmware-config.pl as normal. The script properly found everything it needed, set up the virtual networks, and compiled all of the modules against the PAE kernel with no errors at all. All of the services reported in as having started successfully when the script exited, which was when the trouble started.
We immediately picked up an selinux error saying that one of the modules required the ability to use text relocation. No big deal here, which is why I don't remember off hand which module committed the offense. I'll go back and pull it up next chance - I'm on a different system right now. The selinux troubleshooter gave us the required command to address this issue, so we fixed the problem and off we went.
...Or so we thought.
It seems that something else in selinux is interfering with a new VMware Server 2.0 service called VirtualMachines. I'm not sure what the problem is, how it happens, or why. What happens is that you can launch Firefox to talk to VMware server (http://localhost:8222 in this case) and get the VMware Server login page. However, from there you are unable to login. The system times out with a message basically saying that communication with the back-end server processes has been lost. Further checking (service vmware status) shows that several VMware Server services are actually NOT running.
Upon trying to restart the vmware services (service vmware restart), we see that the VirtualMachines service has failed. There are no errors I can see, and nothing in dmesg out of the ordinary.
Next, we placed selinux into permissive mode to see if anything might pop up or change, and then rebooted the system. We saw exactly the same behavior from VMware Server as before when selinux was in enforcing mode.
Finally, we disabled selinux altogether and rebooted once more. This time, VMware Server came up and ran flawlessly. In fact, it was impressively fast given the age of the hardware.
Just for grins, we then completely erased VMware Server, rebooted, and double-checked to make sure everything about it was completely gone from the system. We then re-installed it using the exact same procedure as before. VMware Server installed and ran flawlessly. In fact, just to be sure again, we rebooted the server one more time. Again VMware Server came up and ran without issues.
Thus, in our testing of this, it is clear there are multiple issues with VMware Server and selinux. One of the issues is that a specific module requires text relocation, which is easily solved. The other issue is going to be a little more difficult to troubleshoot, but clearly there is something that conflicts between selinux and one of the new VMware Server services, and the only way to get around it at this point is to disable selinux.
I'll have the system handy for the next day or so to do some additional testing, but then I have to put it back into production. Let me know what specifics I should look for next to find the source of the problem.
Cheers,
Chris
-- ================================== By all means marry; If you get a good wife, you'll be happy. If you get a bad one, you'll become a philosopher.
--Socrates