The move to git!
Jon Ciesla
limb at jcomserv.net
Fri Jul 30 20:02:39 UTC 2010
On 7/29/2010 10:55 PM, Jesse Keating wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> This evening we opened up dist-git for business. Dist-git is our git
> based replacement for dist-cvs, the source control system we were using
> for our package specs, patches, and source files. This has been a long
> time coming and a massive effort. I want to take a little time here to
> outline what we've done and where we are going.
>
> First a brief outline of how our CVS system worked. CVS is a daemon of
> sorts, and all repos typically live within a single CVSROOT. Within
> this CVSROOT we had an 'avail' system to make use of, that we could
> populate with data from Fedora Account System and dump into this file.
> Avail worked on path names, relative to the CVSROOT. Since we used
> directories for each Fedora release as pseudo branches we could set
> avail info on each release "branch". CVS also used some filesystem
> symlink tricks to create a "common" subdir for every package module, and
> this is where we stuffed common scripts and Makefile content. Pretty
> clever on one hand, we can make updates to the make system without
> touching every single package, but it is pretty hackish and we had
> constant issues where somebody would attempt an action using old common
> content and stuff would fall over.
>
> Now we look at git. Git is for the most part a daemonless system. Each
> repository is completely stand alone and generally does not require any
> other infrastructure to be useful. You can interact with a repository
> directly on the filesystem using /usr/bin/git or you can interact with
> it through say ssh, again using /usr/bin/git (your local /usr/bin/git
> will call a remote /usr/bin/git). Generally there is no running daemon
> to connect to and authenticate with. Basic authentication of who can
> check out what and commit where with git happens at the filesystem
> permissions level. One twist with this is that with git, we wanted to
> use real branches within a package repo to reflect the different
> Fedora/EPEL releases. In repo branches are not reflected with path
> names that we could set filesystem ACLs upon, so this posed a problem
> for our conversion.
>
> Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an
> addon system to git that provides ACL functionality including different
> rights for different branches within a given repository, and more! By
> using gitolite as a replacement for /usr/bin/git when a user connects to
> our git server we can again utilize the information we have within the
> Fedora Package Database and properly allow / restrict changes on
> specific branches for specific developers. The gitolite upstream (
> Sitaram Chamarty, sitaram at atc.tcs.com ) has been fantastically
> responsive to our needs, which are admittedly a little unique. We have
> a very large set of repositories (over 10.5K) and a largish number of
> contributors (1050). The combination of the two leads to a very very
> large and complex ACL structure that at first broke the system quite
> badly. Upstream was very quick to create a "bigconfig" method of
> compiling the ACLs without crashing the box. Our other unique needs
> involve having individual accounts for each committer instead of a
> shared account with a large list of allowed SSH keys. Add to that some
> of our accounts need to be able to ssh shell into the git server for
> administrative duties. Throughout our trials and testing with gitolite
> every time we've ran into some issue that just didn't fit the mold,
> Sitaram has been there with a smile and a fix. At this point our
> production server is a whopping one line different from current gitolite
> upstream. This is a fantastic win for us, for our sustainability, and
> for the next large group that wants to make use of gitolite.
>
> Once we had a plan for ACLs and for branches, we had to decide if we
> were going to replace the Make system and with what. I had never been a
> fan of Make, it was entirely too difficult to modify and innovate with.
> Since I was the one pushing this transition forward, I decided to write
> a new tool in my favorite language, python. The fedpkg tool was born
> and took off. fedpkg was born around January 4th, 2010 and has since
> grown into 1,523 (via sloccount) lines of code. While far from
> complete, it is a great start (if I do say so myself!) at replacing the
> make system. Because it is written in python (or maybe just !Make) it
> seems to be easier to contribute to, and I've already gotten a number of
> contributions. More will come as it starts to be more widely used. The
> biggest challenge with fedpkg is removing the need to update something
> on the end user's system every single time we added a new Fedora release
> and changed what happens when you build for rawhide. I'll spare you the
> details but I'm fairly happy with what we have. The end result should
> be far fewer misfires and end user confusion.
>
> The last major piece of the puzzle was how to actually convert the
> existing CVS repositories, including the fun pseudo branches, into git
> repositories. I tried a number of options over the years (I've been
> working on this off and on for nearly 4 years!) ranging from the built
> in git cvsimport to git-svn to parsecvs and a few others. In the end,
> we took a page from the gnome project and used parsecvs (
> http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of
> our repositories. There were a few that gave parsecvs fits and recent
> versions of git cvsimport were able to handle them. The git system is
> fantastic enough that we were able to merge our pseudo cvs branches into
> actual git branches complete with a real shared history, but again I'll
> spare you the details of the scripting to do this. All but the kernel
> repo seems to have converted successfully which is a pretty good
> success rate in my book. We may yet get the kernel converted, but in
> the interest of time we opted to start fresh with dist-git for now.
>
> Without the help of many others, this project would never have gotten
> done. Folks helped out with Koji modifications, with fedpkg
> contributions, with repeated testing of attempted conversions, with
> logic checking of my plans, of helping me understand more of git
> internals and deciphering error output, and most of all with being
> patient while we worked through the transition and very positive along
> the way. Things will be bumpy over the next few weeks as we really
> start putting distgit to the test. No amount of staging and testing can
> really replace production use. There will be many more updates to
> fedpkg as bugs are found and fixed and features are contributed. Wiki
> pages will get filled out as knowledge of how to interact with dist-git
> starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a
> good start ).
>
>
When I do fedpkg clone -B libfoo, to get the old-style layout, all the
branches are identical to devel. The branches on the standard clone
seem fine. I've not made any changes with the -B layout, but I thought
this should be corrected or deprecated.
fedora-packager-0.5.1.0-1.fc13.noarch
-J
> Once again I want to thank everybody who helped out and for all the
> (continued) patience! I'll be available via email and IRC as much as
> possible the next few days to help anybody with dist-git issues. Look
> for Oxf13 on freenode. Happy gitting!
>
> - --
> Jesse Keating
> Fedora -- Freedom² is a feature!
> identi.ca: http://identi.ca/jkeating
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.14 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkxSTR0ACgkQ4v2HLvE71NXEswCeOAu+EG/porsvkjMVq+4lAchy
> VMgAn1DNwqyYcSSC3bwX/MQ/cfwx7qjs
> =fBGf
> -----END PGP SIGNATURE-----
> _______________________________________________
> devel-announce mailing list
> devel-announce at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel-announce
>
More information about the devel
mailing list