Has anyone come across problem with directory server having too many
fds open, which then causes it to not receive any new connections?

This typically indicates that you have a client application that is leaking connections.

The sockets on CLOSE_WAIT shouldn't be a problem. It's active open connections
that bung up your server.

If you really do need gazillions of active connections, then you'll need to
increase the file descriptor limit as detailed in the documentation.