The move to git!

Fri Jul 30 20:20:34 UTC 2010

  On 07/30/2010 03:02 PM, Jon Ciesla wrote:
> On 7/29/2010 10:55 PM, Jesse Keating wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> This evening we opened up dist-git for business.  Dist-git is our git
>> based replacement for dist-cvs, the source control system we were using
>> for our package specs, patches, and source files.  This has been a long
>> time coming and a massive effort.  I want to take a little time here to
>> outline what we've done and where we are going.
>>
>> First a brief outline of how our CVS system worked.  CVS is a daemon of
>> sorts, and all repos typically live within a single CVSROOT.  Within
>> this CVSROOT we had an 'avail' system to make use of, that we could
>> populate with data from Fedora Account System and dump into this file.
>> Avail worked on path names, relative to the CVSROOT.  Since we used
>> directories for each Fedora release as pseudo branches we could set
>> avail info on each release "branch".  CVS also used some filesystem
>> symlink tricks to create a "common" subdir for every package module, and
>> this is where we stuffed common scripts and Makefile content.  Pretty
>> clever on one hand, we can make updates to the make system without
>> touching every single package, but it is pretty hackish and we had
>> constant issues where somebody would attempt an action using old common
>> content and stuff would fall over.
>>
>> Now we look at git.  Git is for the most part a daemonless system.  Each
>> repository is completely stand alone and generally does not require any
>> other infrastructure to be useful.  You can interact with a repository
>> directly on the filesystem using /usr/bin/git or you can interact with
>> it through say ssh, again using /usr/bin/git (your local /usr/bin/git
>> will call a remote /usr/bin/git).  Generally there is no running daemon
>> to connect to and authenticate with.  Basic authentication of who can
>> check out what and commit where with git happens at the filesystem
>> permissions level.  One twist with this is that with git, we wanted to
>> use real branches within a package repo to reflect the different
>> Fedora/EPEL releases.  In repo branches are not reflected with path
>> names that we could set filesystem ACLs upon, so this posed a problem
>> for our conversion.
>>
>> Enter gitolite.  Gitolite ( http://github.com/sitaramc/gitolite )is an
>> addon system to git that provides ACL functionality including different
>> rights for different branches within a given repository, and more!  By
>> using gitolite as a replacement for /usr/bin/git when a user connects to
>> our git server we can again utilize the information we have within the
>> Fedora Package Database and properly allow / restrict changes on
>> specific branches for specific developers.  The gitolite upstream (
>> Sitaram Chamarty, sitaram at atc.tcs.com ) has been fantastically
>> responsive to our needs, which are admittedly a little unique.  We have
>> a very large set of repositories (over 10.5K) and a largish number of
>> contributors (1050).  The combination of the two leads to a very very
>> large and complex ACL structure that at first broke the system quite
>> badly.  Upstream was very quick to create a "bigconfig" method of
>> compiling the ACLs without crashing the box.  Our other unique needs
>> involve having individual accounts for each committer instead of a
>> shared account with a large list of allowed SSH keys.  Add to that some
>> of our accounts need to be able to ssh shell into the git server for
>> administrative duties.  Throughout our trials and testing with gitolite
>> every time we've ran into some issue that just didn't fit the mold,
>> Sitaram has been there with a smile and a fix.  At this point our
>> production server is a whopping one line different from current gitolite
>> upstream.  This is a fantastic win for us, for our sustainability, and
>> for the next large group that wants to make use of gitolite.
>>
>> Once we had a plan for ACLs and for branches, we had to decide if we
>> were going to replace the Make system and with what.  I had never been a
>> fan of Make, it was entirely too difficult to modify and innovate with.
>>    Since I was the one pushing this transition forward, I decided to write
>> a new tool in my favorite language, python.  The fedpkg tool was born
>> and took off.  fedpkg was born around January 4th, 2010 and has since
>> grown into 1,523 (via sloccount) lines of code.  While far from
>> complete, it is a great start (if I do say so myself!) at replacing the
>> make system.  Because it is written in python (or maybe just !Make) it
>> seems to be easier to contribute to, and I've already gotten a number of
>> contributions.  More will come as it starts to be more widely used.  The
>> biggest challenge with fedpkg is removing the need to update something
>> on the end user's system every single time we added a new Fedora release
>> and changed what happens when you build for rawhide.  I'll spare you the
>> details but I'm fairly happy with what we have.  The end result should
>> be far fewer misfires and end user confusion.
>>
>> The last major piece of the puzzle was how to actually convert the
>> existing CVS repositories, including the fun pseudo branches, into git
>> repositories.  I tried a number of options over the years (I've been
>> working on this off and on for nearly 4 years!) ranging from the built
>> in git cvsimport to git-svn to parsecvs and a few others.  In the end,
>> we took a page from the gnome project and used parsecvs (
>> http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of
>> our repositories.  There were a few that gave parsecvs fits and recent
>> versions of git cvsimport were able to handle them.  The git system is
>> fantastic enough that we were able to merge our pseudo cvs branches into
>> actual git branches complete with a real shared history, but again I'll
>> spare you the details of the scripting to do this.  All but the kernel
>> repo  seems to have converted successfully which is a pretty good
>> success rate in my book.  We may yet get the kernel converted, but in
>> the interest of time we opted to start fresh with dist-git for now.
>>
>> Without the help of many others, this project would never have gotten
>> done.  Folks helped out with Koji modifications, with fedpkg
>> contributions, with repeated testing of attempted conversions, with
>> logic checking of my plans, of helping me understand more of git
>> internals and deciphering error output, and most of all with being
>> patient while we worked through the transition and very positive along
>> the way.  Things will be bumpy over the next few weeks as we really
>> start putting distgit to the test.  No amount of staging and testing can
>> really replace production use.  There will be many more updates to
>> fedpkg as bugs are found and fixed and features are contributed.  Wiki
>> pages will get filled out as knowledge of how to interact with dist-git
>> starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a
>> good start ).
>>
>>
> When I do fedpkg clone -B libfoo, to get the old-style layout, all the
> branches are identical to devel.  The branches on the standard clone
> seem fine.  I've not made any changes with the -B layout, but I thought
> this should be corrected or deprecated.
>
> fedora-packager-0.5.1.0-1.fc13.noarch
>
> -J
Oh, and since I neglected to say so, thanks a huge heaping ton to Jesse 
and everyone else involved in making this happen, I've been excited to 
see it happen, and I have to say after using it today that the 
improvements in speed and usability are better than I expected.  Kudos 
all around.

-J
>> Once again I want to thank everybody who helped out and for all the
>> (continued) patience!  I'll be available via email and IRC as much as
>> possible the next few days to help anybody with dist-git issues.  Look
>> for Oxf13 on freenode.  Happy gitting!
>>
>> - -- 
>> Jesse Keating
>> Fedora -- Freedom² is a feature!
>> identi.ca: http://identi.ca/jkeating
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.14 (GNU/Linux)
>> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
>>
>> iEYEARECAAYFAkxSTR0ACgkQ4v2HLvE71NXEswCeOAu+EG/porsvkjMVq+4lAchy
>> VMgAn1DNwqyYcSSC3bwX/MQ/cfwx7qjs
>> =fBGf
>> -----END PGP SIGNATURE-----
>> _______________________________________________
>> devel-announce mailing list
>> devel-announce at lists.fedoraproject.org
>> https://admin.fedoraproject.org/mailman/listinfo/devel-announce
>>
>


-- 
- in your fear, speak only peace
   in your fear, seek only love

-d. bowie