Suitability of Python for daemon processes

[Mike McGrath]

On Sun, 25 Oct 2009, Ben Boeckel wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Mike McGrath wrote:
>
> > On Sun, 25 Oct 2009, Ben Boeckel wrote:
> >
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA256
> >>
> >> Hi,
> >>
> >> I am working on a project designed to mirror large, changing archives of
> >> software in a manner that ensures data integrity and atomic updates using a
> >> peer-to-peer protocol.
> >>
> >> My team and I would like to design this software such that it provides a
> >> suitable replacement for rsync.  As Fedora is among the leading
> >> distributions, I would like to solicit your opinions on the implementation.
> >> We have designed much of the architecture, but have not implemented anything
> >> yet.  One issue we wish to address currently is that of interpreted vs.
> >> compiled languages.
> >>
> >> My team and I would like to know whether the community would be accepting of
> >> such a project (which includes a daemon) if it were written in Python rather
> >> than C or C++.  If so, it would greatly simplify the implementation and
> >> allow it to be more robust.  Python's built-in libraries and facilities
> >> provide much of the path and network manipulation that the daemon requires.
> >> Using the Python standard libraries allows us to rely upon a well-tested
> >> base and focus on higher-level issues.
> >>
> >> What are your opinions, as system administrators, on using Python for
> >> long-running daemon processes when the developers are explicitly mindful of
> >> memory considerations?
> >>
> >
> > I've generally had better luck with C/C++ based daemons from a systems
> > admin point of view.  But we've used plenty of python based daemons that
> > worked just fine.  I think the problem is the lower barrier to using
> > python means it's easier for less experienced developers to create python
> > daemons.
> >
> > Really though, if it's done right, python daemons can be just as good as C
> > daemons.  And extending Python is very easy as you mentioned with using
> > pythons built in libraries.  If you're concerned about performance though
> > it shouldn't take much to do the basics of what you want and compare C to
> > python.  Sometimes they're identical, sometimes C wins but I don't think
> > I've seen python win yet (with performance).  But it is much easier to
> > work with python :)
> >
> > 	-Mike
>
> Thanks for your reply.
>
> Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't
> then either you're sitting next door, our implementation is bad, or the
> hardware shouldn't be a mirror in the first place.
>

With all my babbling I forgot to mention we do already run python daemons
in Fedora Infrastructure.  Func is one, TurboGears (though it's wrapped in
mod_wsgi) and one that we wrote ourselves[1] is our mirrorlist server.
It's the backend that powers:

http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-11&arch=i386

	-Mike

[1] By 'ourselves' I mean Matt Domsch who, quite honestly, has designed
the most robust, fault tolerant dynamic content system we run.