The move to git!

Phil Knirsch pknirsch at redhat.com
Fri Jul 30 07:48:04 UTC 2010


On 07/30/2010 05:55 AM, Jesse Keating wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> This evening we opened up dist-git for business.  Dist-git is our git
> based replacement for dist-cvs, the source control system we were using
> for our package specs, patches, and source files.  This has been a long
> time coming and a massive effort.  I want to take a little time here to
> outline what we've done and where we are going.
>
> First a brief outline of how our CVS system worked.  CVS is a daemon of
> sorts, and all repos typically live within a single CVSROOT.  Within
> this CVSROOT we had an 'avail' system to make use of, that we could
> populate with data from Fedora Account System and dump into this file.
> Avail worked on path names, relative to the CVSROOT.  Since we used
> directories for each Fedora release as pseudo branches we could set
> avail info on each release "branch".  CVS also used some filesystem
> symlink tricks to create a "common" subdir for every package module, and
> this is where we stuffed common scripts and Makefile content.  Pretty
> clever on one hand, we can make updates to the make system without
> touching every single package, but it is pretty hackish and we had
> constant issues where somebody would attempt an action using old common
> content and stuff would fall over.
>
> Now we look at git.  Git is for the most part a daemonless system.  Each
> repository is completely stand alone and generally does not require any
> other infrastructure to be useful.  You can interact with a repository
> directly on the filesystem using /usr/bin/git or you can interact with
> it through say ssh, again using /usr/bin/git (your local /usr/bin/git
> will call a remote /usr/bin/git).  Generally there is no running daemon
> to connect to and authenticate with.  Basic authentication of who can
> check out what and commit where with git happens at the filesystem
> permissions level.  One twist with this is that with git, we wanted to
> use real branches within a package repo to reflect the different
> Fedora/EPEL releases.  In repo branches are not reflected with path
> names that we could set filesystem ACLs upon, so this posed a problem
> for our conversion.
>
> Enter gitolite.  Gitolite ( http://github.com/sitaramc/gitolite )is an
> addon system to git that provides ACL functionality including different
> rights for different branches within a given repository, and more!  By
> using gitolite as a replacement for /usr/bin/git when a user connects to
> our git server we can again utilize the information we have within the
> Fedora Package Database and properly allow / restrict changes on
> specific branches for specific developers.  The gitolite upstream (
> Sitaram Chamarty, sitaram at atc.tcs.com ) has been fantastically
> responsive to our needs, which are admittedly a little unique.  We have
> a very large set of repositories (over 10.5K) and a largish number of
> contributors (1050).  The combination of the two leads to a very very
> large and complex ACL structure that at first broke the system quite
> badly.  Upstream was very quick to create a "bigconfig" method of
> compiling the ACLs without crashing the box.  Our other unique needs
> involve having individual accounts for each committer instead of a
> shared account with a large list of allowed SSH keys.  Add to that some
> of our accounts need to be able to ssh shell into the git server for
> administrative duties.  Throughout our trials and testing with gitolite
> every time we've ran into some issue that just didn't fit the mold,
> Sitaram has been there with a smile and a fix.  At this point our
> production server is a whopping one line different from current gitolite
> upstream.  This is a fantastic win for us, for our sustainability, and
> for the next large group that wants to make use of gitolite.
>
> Once we had a plan for ACLs and for branches, we had to decide if we
> were going to replace the Make system and with what.  I had never been a
> fan of Make, it was entirely too difficult to modify and innovate with.
>   Since I was the one pushing this transition forward, I decided to write
> a new tool in my favorite language, python.  The fedpkg tool was born
> and took off.  fedpkg was born around January 4th, 2010 and has since
> grown into 1,523 (via sloccount) lines of code.  While far from
> complete, it is a great start (if I do say so myself!) at replacing the
> make system.  Because it is written in python (or maybe just !Make) it
> seems to be easier to contribute to, and I've already gotten a number of
> contributions.  More will come as it starts to be more widely used.  The
> biggest challenge with fedpkg is removing the need to update something
> on the end user's system every single time we added a new Fedora release
> and changed what happens when you build for rawhide.  I'll spare you the
> details but I'm fairly happy with what we have.  The end result should
> be far fewer misfires and end user confusion.
>
> The last major piece of the puzzle was how to actually convert the
> existing CVS repositories, including the fun pseudo branches, into git
> repositories.  I tried a number of options over the years (I've been
> working on this off and on for nearly 4 years!) ranging from the built
> in git cvsimport to git-svn to parsecvs and a few others.  In the end,
> we took a page from the gnome project and used parsecvs (
> http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of
> our repositories.  There were a few that gave parsecvs fits and recent
> versions of git cvsimport were able to handle them.  The git system is
> fantastic enough that we were able to merge our pseudo cvs branches into
> actual git branches complete with a real shared history, but again I'll
> spare you the details of the scripting to do this.  All but the kernel
> repo  seems to have converted successfully which is a pretty good
> success rate in my book.  We may yet get the kernel converted, but in
> the interest of time we opted to start fresh with dist-git for now.
>
> Without the help of many others, this project would never have gotten
> done.  Folks helped out with Koji modifications, with fedpkg
> contributions, with repeated testing of attempted conversions, with
> logic checking of my plans, of helping me understand more of git
> internals and deciphering error output, and most of all with being
> patient while we worked through the transition and very positive along
> the way.  Things will be bumpy over the next few weeks as we really
> start putting distgit to the test.  No amount of staging and testing can
> really replace production use.  There will be many more updates to
> fedpkg as bugs are found and fixed and features are contributed.  Wiki
> pages will get filled out as knowledge of how to interact with dist-git
> starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a
> good start ).
>
> Once again I want to thank everybody who helped out and for all the
> (continued) patience!  I'll be available via email and IRC as much as
> possible the next few days to help anybody with dist-git issues.  Look
> for Oxf13 on freenode.  Happy gitting!
>

I only have 2 words for you and everyone who has put so insanely much 
work into this effort for such a long time:

  T H A N K   Y O U !

Wow, i never thought i'd ever do that again with all caps and spaces in 
between. :)

But in all seriousness, feel free to imagine the above is repeated 1001 
times (at least) as this is going to be such a tremendous help for 
everyone working on and with Fedora packages.

Rock on!

Regards, Phil

-- 
Philipp Knirsch              | Tel.:  +49-711-96437-470
Supervisor Core Services     | Fax.:  +49-711-96437-111
Red Hat GmbH                 | Email: Phil Knirsch <pknirsch at redhat.com>
Hauptstaetterstr. 58         | Web:   http://www.redhat.com/
D-70178 Stuttgart, Germany
Motd:  You're only jealous cos the little penguins are talking to me.


More information about the devel mailing list