-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This evening we opened up dist-git for business. Dist-git is our git based replacement for dist-cvs, the source control system we were using for our package specs, patches, and source files. This has been a long time coming and a massive effort. I want to take a little time here to outline what we've done and where we are going.
First a brief outline of how our CVS system worked. CVS is a daemon of sorts, and all repos typically live within a single CVSROOT. Within this CVSROOT we had an 'avail' system to make use of, that we could populate with data from Fedora Account System and dump into this file. Avail worked on path names, relative to the CVSROOT. Since we used directories for each Fedora release as pseudo branches we could set avail info on each release "branch". CVS also used some filesystem symlink tricks to create a "common" subdir for every package module, and this is where we stuffed common scripts and Makefile content. Pretty clever on one hand, we can make updates to the make system without touching every single package, but it is pretty hackish and we had constant issues where somebody would attempt an action using old common content and stuff would fall over.
Now we look at git. Git is for the most part a daemonless system. Each repository is completely stand alone and generally does not require any other infrastructure to be useful. You can interact with a repository directly on the filesystem using /usr/bin/git or you can interact with it through say ssh, again using /usr/bin/git (your local /usr/bin/git will call a remote /usr/bin/git). Generally there is no running daemon to connect to and authenticate with. Basic authentication of who can check out what and commit where with git happens at the filesystem permissions level. One twist with this is that with git, we wanted to use real branches within a package repo to reflect the different Fedora/EPEL releases. In repo branches are not reflected with path names that we could set filesystem ACLs upon, so this posed a problem for our conversion.
Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an addon system to git that provides ACL functionality including different rights for different branches within a given repository, and more! By using gitolite as a replacement for /usr/bin/git when a user connects to our git server we can again utilize the information we have within the Fedora Package Database and properly allow / restrict changes on specific branches for specific developers. The gitolite upstream ( Sitaram Chamarty, sitaram@atc.tcs.com ) has been fantastically responsive to our needs, which are admittedly a little unique. We have a very large set of repositories (over 10.5K) and a largish number of contributors (1050). The combination of the two leads to a very very large and complex ACL structure that at first broke the system quite badly. Upstream was very quick to create a "bigconfig" method of compiling the ACLs without crashing the box. Our other unique needs involve having individual accounts for each committer instead of a shared account with a large list of allowed SSH keys. Add to that some of our accounts need to be able to ssh shell into the git server for administrative duties. Throughout our trials and testing with gitolite every time we've ran into some issue that just didn't fit the mold, Sitaram has been there with a smile and a fix. At this point our production server is a whopping one line different from current gitolite upstream. This is a fantastic win for us, for our sustainability, and for the next large group that wants to make use of gitolite.
Once we had a plan for ACLs and for branches, we had to decide if we were going to replace the Make system and with what. I had never been a fan of Make, it was entirely too difficult to modify and innovate with. Since I was the one pushing this transition forward, I decided to write a new tool in my favorite language, python. The fedpkg tool was born and took off. fedpkg was born around January 4th, 2010 and has since grown into 1,523 (via sloccount) lines of code. While far from complete, it is a great start (if I do say so myself!) at replacing the make system. Because it is written in python (or maybe just !Make) it seems to be easier to contribute to, and I've already gotten a number of contributions. More will come as it starts to be more widely used. The biggest challenge with fedpkg is removing the need to update something on the end user's system every single time we added a new Fedora release and changed what happens when you build for rawhide. I'll spare you the details but I'm fairly happy with what we have. The end result should be far fewer misfires and end user confusion.
The last major piece of the puzzle was how to actually convert the existing CVS repositories, including the fun pseudo branches, into git repositories. I tried a number of options over the years (I've been working on this off and on for nearly 4 years!) ranging from the built in git cvsimport to git-svn to parsecvs and a few others. In the end, we took a page from the gnome project and used parsecvs ( http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of our repositories. There were a few that gave parsecvs fits and recent versions of git cvsimport were able to handle them. The git system is fantastic enough that we were able to merge our pseudo cvs branches into actual git branches complete with a real shared history, but again I'll spare you the details of the scripting to do this. All but the kernel repo seems to have converted successfully which is a pretty good success rate in my book. We may yet get the kernel converted, but in the interest of time we opted to start fresh with dist-git for now.
Without the help of many others, this project would never have gotten done. Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way. Things will be bumpy over the next few weeks as we really start putting distgit to the test. No amount of staging and testing can really replace production use. There will be many more updates to fedpkg as bugs are found and fixed and features are contributed. Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).
Once again I want to thank everybody who helped out and for all the (continued) patience! I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues. Look for Oxf13 on freenode. Happy gitting!
- -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
_______________________________________________ devel-announce mailing list devel-announce@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel-announce
On 07/30/2010 05:55 AM, Jesse Keating wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This evening we opened up dist-git for business. Dist-git is our git based replacement for dist-cvs, the source control system we were using for our package specs, patches, and source files. This has been a long time coming and a massive effort. I want to take a little time here to outline what we've done and where we are going.
First a brief outline of how our CVS system worked. CVS is a daemon of sorts, and all repos typically live within a single CVSROOT. Within this CVSROOT we had an 'avail' system to make use of, that we could populate with data from Fedora Account System and dump into this file. Avail worked on path names, relative to the CVSROOT. Since we used directories for each Fedora release as pseudo branches we could set avail info on each release "branch". CVS also used some filesystem symlink tricks to create a "common" subdir for every package module, and this is where we stuffed common scripts and Makefile content. Pretty clever on one hand, we can make updates to the make system without touching every single package, but it is pretty hackish and we had constant issues where somebody would attempt an action using old common content and stuff would fall over.
Now we look at git. Git is for the most part a daemonless system. Each repository is completely stand alone and generally does not require any other infrastructure to be useful. You can interact with a repository directly on the filesystem using /usr/bin/git or you can interact with it through say ssh, again using /usr/bin/git (your local /usr/bin/git will call a remote /usr/bin/git). Generally there is no running daemon to connect to and authenticate with. Basic authentication of who can check out what and commit where with git happens at the filesystem permissions level. One twist with this is that with git, we wanted to use real branches within a package repo to reflect the different Fedora/EPEL releases. In repo branches are not reflected with path names that we could set filesystem ACLs upon, so this posed a problem for our conversion.
Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an addon system to git that provides ACL functionality including different rights for different branches within a given repository, and more! By using gitolite as a replacement for /usr/bin/git when a user connects to our git server we can again utilize the information we have within the Fedora Package Database and properly allow / restrict changes on specific branches for specific developers. The gitolite upstream ( Sitaram Chamarty, sitaram@atc.tcs.com ) has been fantastically responsive to our needs, which are admittedly a little unique. We have a very large set of repositories (over 10.5K) and a largish number of contributors (1050). The combination of the two leads to a very very large and complex ACL structure that at first broke the system quite badly. Upstream was very quick to create a "bigconfig" method of compiling the ACLs without crashing the box. Our other unique needs involve having individual accounts for each committer instead of a shared account with a large list of allowed SSH keys. Add to that some of our accounts need to be able to ssh shell into the git server for administrative duties. Throughout our trials and testing with gitolite every time we've ran into some issue that just didn't fit the mold, Sitaram has been there with a smile and a fix. At this point our production server is a whopping one line different from current gitolite upstream. This is a fantastic win for us, for our sustainability, and for the next large group that wants to make use of gitolite.
Once we had a plan for ACLs and for branches, we had to decide if we were going to replace the Make system and with what. I had never been a fan of Make, it was entirely too difficult to modify and innovate with. Since I was the one pushing this transition forward, I decided to write a new tool in my favorite language, python. The fedpkg tool was born and took off. fedpkg was born around January 4th, 2010 and has since grown into 1,523 (via sloccount) lines of code. While far from complete, it is a great start (if I do say so myself!) at replacing the make system. Because it is written in python (or maybe just !Make) it seems to be easier to contribute to, and I've already gotten a number of contributions. More will come as it starts to be more widely used. The biggest challenge with fedpkg is removing the need to update something on the end user's system every single time we added a new Fedora release and changed what happens when you build for rawhide. I'll spare you the details but I'm fairly happy with what we have. The end result should be far fewer misfires and end user confusion.
The last major piece of the puzzle was how to actually convert the existing CVS repositories, including the fun pseudo branches, into git repositories. I tried a number of options over the years (I've been working on this off and on for nearly 4 years!) ranging from the built in git cvsimport to git-svn to parsecvs and a few others. In the end, we took a page from the gnome project and used parsecvs ( http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of our repositories. There were a few that gave parsecvs fits and recent versions of git cvsimport were able to handle them. The git system is fantastic enough that we were able to merge our pseudo cvs branches into actual git branches complete with a real shared history, but again I'll spare you the details of the scripting to do this. All but the kernel repo seems to have converted successfully which is a pretty good success rate in my book. We may yet get the kernel converted, but in the interest of time we opted to start fresh with dist-git for now.
Without the help of many others, this project would never have gotten done. Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way. Things will be bumpy over the next few weeks as we really start putting distgit to the test. No amount of staging and testing can really replace production use. There will be many more updates to fedpkg as bugs are found and fixed and features are contributed. Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).
Once again I want to thank everybody who helped out and for all the (continued) patience! I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues. Look for Oxf13 on freenode. Happy gitting!
I only have 2 words for you and everyone who has put so insanely much work into this effort for such a long time:
T H A N K Y O U !
Wow, i never thought i'd ever do that again with all caps and spaces in between. :)
But in all seriousness, feel free to imagine the above is repeated 1001 times (at least) as this is going to be such a tremendous help for everyone working on and with Fedora packages.
Rock on!
Regards, Phil
Dne 30.7.2010 09:48, Phil Knirsch napsal(a):
I only have 2 words for you and everyone who has put so insanely much work into this effort for such a long time:
T H A N K Y O U !
Everybody was bitching about CVS for years, so now we all should say to this
+1000
Thank you
On Fri, Jul 30, 2010 at 2:17 PM, Matěj Cepl mcepl@redhat.com wrote:
Everybody was bitching about CVS for years,
I'm pretty sure I wasn't. But I'll still pile on with a +1 for the switch over to git
-jef"getting git to compile on qnx is no fun"spaleta
On Friday 30 of July 2010 05:55:09 Jesse Keating wrote:
... Wiki pages
will get filled out as knowledge of how to interact with dist-git
starts
to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a
good
start ).
Thanks for your hard work! Could you describe this in more details: OLD CVS | NEW GIT | Notes make tag | N/A | Explicitly tagging source states for package builds is no longer necessary.
how this exactly works? What will happen in following cases? : 1) commit some changes with release number change, commit another changes, build
2) commit some changes with release number change, scratch build, commit rest of the changes, build
3) commit some changes with release number change, someone else starts build, commit rest of the changes, build
4) commit some changes with release number change, build - fails because of typo/missing updated sources, commit fix, build
Michal
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 01:02 AM, Michal Hlavinka wrote:
On Friday 30 of July 2010 05:55:09 Jesse Keating wrote:
... Wiki pages
will get filled out as knowledge of how to interact with dist-git
starts
to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a
good
start ).
Thanks for your hard work! Could you describe this in more details: OLD CVS | NEW GIT | Notes make tag | N/A | Explicitly tagging source states for package builds is no longer necessary.
how this exactly works? What will happen in following cases? :
- commit some changes with
release number change, commit another changes, build
The build attempt will use the latest content you pushed. If you've previously built something with that n-v-r the buildsystem will reject the build, otherwise it'll continue.
- commit some changes
with release number change, scratch build, commit rest of the changes, build
See above, the build would happen.
- commit some changes with release number change, someone else
starts build, commit rest of the changes, build
The buildsystem will reject the build because one with the same n-v-r is either in progress or completed successfully.
- commit some changes with
release number change, build - fails because of typo/missing updated sources, commit fix, build
Build will go through because there are no successful builds with that n-v-r
Michal
- -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
Am 30.07.10 10:02, schrieb Michal Hlavinka:
Thanks for your hard work! Could you describe this in more details: OLD CVS | NEW GIT | Notes make tag | N/A | Explicitly tagging source states for package builds is no longer necessary.
how this exactly
Each commit has an unique number in git. If you submit an build to koji, this unique number wil be transmitted to koji. So you can be sure, that koji use your personally HEAD even another people has done a push to mein repository.
Best Regards:
Jochen Schmitt
On Fri, Jul 30, 2010 at 4:55 AM, Jesse Keating jkeating@redhat.com wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This evening we opened up dist-git for business. Dist-git is our git based replacement for dist-cvs, the source control system we were using for our package specs, patches, and source files. This has been a long time coming and a massive effort. I want to take a little time here to outline what we've done and where we are going.
First a brief outline of how our CVS system worked. CVS is a daemon of sorts, and all repos typically live within a single CVSROOT. Within this CVSROOT we had an 'avail' system to make use of, that we could populate with data from Fedora Account System and dump into this file. Avail worked on path names, relative to the CVSROOT. Since we used directories for each Fedora release as pseudo branches we could set avail info on each release "branch". CVS also used some filesystem symlink tricks to create a "common" subdir for every package module, and this is where we stuffed common scripts and Makefile content. Pretty clever on one hand, we can make updates to the make system without touching every single package, but it is pretty hackish and we had constant issues where somebody would attempt an action using old common content and stuff would fall over.
Now we look at git. Git is for the most part a daemonless system. Each repository is completely stand alone and generally does not require any other infrastructure to be useful. You can interact with a repository directly on the filesystem using /usr/bin/git or you can interact with it through say ssh, again using /usr/bin/git (your local /usr/bin/git will call a remote /usr/bin/git). Generally there is no running daemon to connect to and authenticate with. Basic authentication of who can check out what and commit where with git happens at the filesystem permissions level. One twist with this is that with git, we wanted to use real branches within a package repo to reflect the different Fedora/EPEL releases. In repo branches are not reflected with path names that we could set filesystem ACLs upon, so this posed a problem for our conversion.
Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an addon system to git that provides ACL functionality including different rights for different branches within a given repository, and more! By using gitolite as a replacement for /usr/bin/git when a user connects to our git server we can again utilize the information we have within the Fedora Package Database and properly allow / restrict changes on specific branches for specific developers. The gitolite upstream ( Sitaram Chamarty, sitaram@atc.tcs.com ) has been fantastically responsive to our needs, which are admittedly a little unique. We have a very large set of repositories (over 10.5K) and a largish number of contributors (1050). The combination of the two leads to a very very large and complex ACL structure that at first broke the system quite badly. Upstream was very quick to create a "bigconfig" method of compiling the ACLs without crashing the box. Our other unique needs involve having individual accounts for each committer instead of a shared account with a large list of allowed SSH keys. Add to that some of our accounts need to be able to ssh shell into the git server for administrative duties. Throughout our trials and testing with gitolite every time we've ran into some issue that just didn't fit the mold, Sitaram has been there with a smile and a fix. At this point our production server is a whopping one line different from current gitolite upstream. This is a fantastic win for us, for our sustainability, and for the next large group that wants to make use of gitolite.
Once we had a plan for ACLs and for branches, we had to decide if we were going to replace the Make system and with what. I had never been a fan of Make, it was entirely too difficult to modify and innovate with. Since I was the one pushing this transition forward, I decided to write a new tool in my favorite language, python. The fedpkg tool was born and took off. fedpkg was born around January 4th, 2010 and has since grown into 1,523 (via sloccount) lines of code. While far from complete, it is a great start (if I do say so myself!) at replacing the make system. Because it is written in python (or maybe just !Make) it seems to be easier to contribute to, and I've already gotten a number of contributions. More will come as it starts to be more widely used. The biggest challenge with fedpkg is removing the need to update something on the end user's system every single time we added a new Fedora release and changed what happens when you build for rawhide. I'll spare you the details but I'm fairly happy with what we have. The end result should be far fewer misfires and end user confusion.
The last major piece of the puzzle was how to actually convert the existing CVS repositories, including the fun pseudo branches, into git repositories. I tried a number of options over the years (I've been working on this off and on for nearly 4 years!) ranging from the built in git cvsimport to git-svn to parsecvs and a few others. In the end, we took a page from the gnome project and used parsecvs ( http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of our repositories. There were a few that gave parsecvs fits and recent versions of git cvsimport were able to handle them. The git system is fantastic enough that we were able to merge our pseudo cvs branches into actual git branches complete with a real shared history, but again I'll spare you the details of the scripting to do this. All but the kernel repo seems to have converted successfully which is a pretty good success rate in my book. We may yet get the kernel converted, but in the interest of time we opted to start fresh with dist-git for now.
Without the help of many others, this project would never have gotten done. Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way. Things will be bumpy over the next few weeks as we really start putting distgit to the test. No amount of staging and testing can really replace production use. There will be many more updates to fedpkg as bugs are found and fixed and features are contributed. Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).
Once again I want to thank everybody who helped out and for all the (continued) patience! I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues. Look for Oxf13 on freenode. Happy gitting!
Excellent news! Thanks for all the work.
One thing I've never got around to working out how to do in git which is different from previous is dealing with branches. Where previously it was as simple as changing directories to deal with the various fedora releases obviously with real branches now we need to do something a little differnet. Could someone update the docs with details how to do this? I retried "fedpkg switch-branch f-13" (and various other possible branch names) and using a "git branch" didn't give me any branches other than master. Could we also add some extra branch related commands to indicate things like listing the current branch, a list of all branches, and how we would commit a new spec to more than one branch.
Cheers, Peter
On Fri, 2010-07-30 at 09:49 +0100, pbrobinson@gmail.com wrote:
Excellent news! Thanks for all the work.
One thing I've never got around to working out how to do in git which is different from previous is dealing with branches. Where previously it was as simple as changing directories to deal with the various fedora releases obviously with real branches now we need to do something a little differnet. Could someone update the docs with details how to do this? I retried "fedpkg switch-branch f-13" (and various other possible branch names) and using a "git branch" didn't give me any branches other than master. Could we also add some extra branch related commands to indicate things like listing the current branch, a list of all branches, and how we would commit a new spec to more than one branch.
I'm not sure how it works if you do "fedpkg clone -B libfoo", but probably similar to the CVS setup. If you do just "fedpkg clone" the branches are managed in git only so the "fedpkg switch-branch f13" indeed switches you to f13 branch even though it looks like master ;-)
Personally I use "git merge <branch-to-merge-from>" to sync the branches (if I have same spec for more than one branch).
Martin
2010/7/30 Martin Sourada martin.sourada@gmail.com:
On Fri, 2010-07-30 at 09:49 +0100, pbrobinson@gmail.com wrote:
Excellent news! Thanks for all the work.
One thing I've never got around to working out how to do in git which is different from previous is dealing with branches. Where previously it was as simple as changing directories to deal with the various fedora releases obviously with real branches now we need to do something a little differnet. Could someone update the docs with details how to do this? I retried "fedpkg switch-branch f-13" (and various other possible branch names) and using a "git branch" didn't give me any branches other than master. Could we also add some extra branch related commands to indicate things like listing the current branch, a list of all branches, and how we would commit a new spec to more than one branch.
I'm not sure how it works if you do "fedpkg clone -B libfoo", but probably similar to the CVS setup. If you do just "fedpkg clone" the branches are managed in git only so the "fedpkg switch-branch f13" indeed switches you to f13 branch even though it looks like master ;-)
Personally I use "git merge <branch-to-merge-from>" to sync the branches (if I have same spec for more than one branch).
I think using git merge or git cherry-pick will be better than coping files from other branch like cvs, git is much smarter than cvs.
Regards, Chen Lei
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 01:49 AM, pbrobinson@gmail.com wrote:
One thing I've never got around to working out how to do in git which is different from previous is dealing with branches. Where previously it was as simple as changing directories to deal with the various fedora releases obviously with real branches now we need to do something a little differnet. Could someone update the docs with details how to do this? I retried "fedpkg switch-branch f-13" (and various other possible branch names) and using a "git branch" didn't give me any branches other than master. Could we also add some extra branch related commands to indicate things like listing the current branch, a list of all branches, and how we would commit a new spec to more than one branch.
https://fedoraproject.org/wiki/Using_Fedora_GIT#Merging is a good start for this, and eventually we'll have tooling in fedpkg to facilitate things like this.
- -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
On Thu, 2010-07-29 at 20:55 -0700, Jesse Keating wrote: <snip>
Without the help of many others, this project would never have gotten done. Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way. Things will be bumpy over the next few weeks as we really start putting distgit to the test. No amount of staging and testing can really replace production use. There will be many more updates to fedpkg as bugs are found and fixed and features are contributed. Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).
Ok, so trying to update a package (libass) I've noticed a tiny problem: the fedpkg switch-branch does not seem to set up proper tracking of remote branches which then results in (uhm, I'm using git commit -a and git push, so you maybe handle this in fedpkg commit -p):
pyfedpkg.FedpkgError: There are unpushed changes in your repo
fedora-packager-0.5.0.1-3.fc12.noarch
Martin
2010/7/30 Martin Sourada martin.sourada@gmail.com:
On Thu, 2010-07-29 at 20:55 -0700, Jesse Keating wrote:
<snip> > Without the help of many others, this project would never have gotten > done. Folks helped out with Koji modifications, with fedpkg > contributions, with repeated testing of attempted conversions, with > logic checking of my plans, of helping me understand more of git > internals and deciphering error output, and most of all with being > patient while we worked through the transition and very positive along > the way. Things will be bumpy over the next few weeks as we really > start putting distgit to the test. No amount of staging and testing can > really replace production use. There will be many more updates to > fedpkg as bugs are found and fixed and features are contributed. Wiki > pages will get filled out as knowledge of how to interact with dist-git > starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a > good start ).
Ok, so trying to update a package (libass) I've noticed a tiny problem: the fedpkg switch-branch does not seem to set up proper tracking of remote branches which then results in (uhm, I'm using git commit -a and git push, so you maybe handle this in fedpkg commit -p):
pyfedpkg.FedpkgError: There are unpushed changes in your repo
fedora-packager-0.5.0.1-3.fc12.noarch
Martin
--
I now use 'git checkout -t origin/f14/master' instead of 'fedpkg switch-branch f14' temporarily.
Chen Lei
On Fri, 2010-07-30 at 17:17 +0800, Chen Lei wrote:
2010/7/30 Martin Sourada martin.sourada@gmail.com:
On Thu, 2010-07-29 at 20:55 -0700, Jesse Keating wrote:
<snip> > Without the help of many others, this project would never have gotten > done. Folks helped out with Koji modifications, with fedpkg > contributions, with repeated testing of attempted conversions, with > logic checking of my plans, of helping me understand more of git > internals and deciphering error output, and most of all with being > patient while we worked through the transition and very positive along > the way. Things will be bumpy over the next few weeks as we really > start putting distgit to the test. No amount of staging and testing can > really replace production use. There will be many more updates to > fedpkg as bugs are found and fixed and features are contributed. Wiki > pages will get filled out as knowledge of how to interact with dist-git > starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a > good start ).
Ok, so trying to update a package (libass) I've noticed a tiny problem: the fedpkg switch-branch does not seem to set up proper tracking of remote branches which then results in (uhm, I'm using git commit -a and git push, so you maybe handle this in fedpkg commit -p):
pyfedpkg.FedpkgError: There are unpushed changes in your repo
fedora-packager-0.5.0.1-3.fc12.noarch
Martin
--
I now use 'git checkout -t origin/f14/master' instead of 'fedpkg switch-branch f14' temporarily.
I've just find a core of the problem. Looking through the git config files I noticed nothing wrong, so I wondered why the push isn't working. So I tried pushing from git gui which threw some error, which lead me to try renaming the branch to f12/master. And lo! git push now works!
Martin
On Fri, 2010-07-30 at 11:20 +0200, Martin Sourada wrote:
On Fri, 2010-07-30 at 17:17 +0800, Chen Lei wrote:
2010/7/30 Martin Sourada martin.sourada@gmail.com:
On Thu, 2010-07-29 at 20:55 -0700, Jesse Keating wrote:
<snip> > Without the help of many others, this project would never have gotten > done. Folks helped out with Koji modifications, with fedpkg > contributions, with repeated testing of attempted conversions, with > logic checking of my plans, of helping me understand more of git > internals and deciphering error output, and most of all with being > patient while we worked through the transition and very positive along > the way. Things will be bumpy over the next few weeks as we really > start putting distgit to the test. No amount of staging and testing can > really replace production use. There will be many more updates to > fedpkg as bugs are found and fixed and features are contributed. Wiki > pages will get filled out as knowledge of how to interact with dist-git > starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a > good start ).
Ok, so trying to update a package (libass) I've noticed a tiny problem: the fedpkg switch-branch does not seem to set up proper tracking of remote branches which then results in (uhm, I'm using git commit -a and git push, so you maybe handle this in fedpkg commit -p):
pyfedpkg.FedpkgError: There are unpushed changes in your repo
fedora-packager-0.5.0.1-3.fc12.noarch
Martin
--
I now use 'git checkout -t origin/f14/master' instead of 'fedpkg switch-branch f14' temporarily.
I've just find a core of the problem. Looking through the git config files I noticed nothing wrong, so I wondered why the push isn't working. So I tried pushing from git gui which threw some error, which lead me to try renaming the branch to f12/master. And lo! git push now works!
Ugh, I forgot to quote the error:
Pushing to ssh://mso@pkgs.fedoraproject.org/libass error: there are still refs under 'refs/heads/f14' remote: error: failed to lock refs/heads/f14[K To ssh://mso@pkgs.fedoraproject.org/libass ! [remote rejected] f14 -> f14 (failed to lock) error: failed to push some refs to 'ssh://mso@pkgs.fedoraproject.org/libass'
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 02:13 AM, Martin Sourada wrote:
Ok, so trying to update a package (libass) I've noticed a tiny problem: the fedpkg switch-branch does not seem to set up proper tracking of remote branches which then results in (uhm, I'm using git commit -a and git push, so you maybe handle this in fedpkg commit -p):
pyfedpkg.FedpkgError: There are unpushed changes in your repo
fedora-packager-0.5.0.1-3.fc12.noarch
This is my fault, I did all my design and testing with a git setting of "push.default tracking" which instructs git to "push" to whatever branch I'm tracking (which makes perfect sense to me, why isn't it a defaut?). Until fedpkg takes this into account, it's easiest to just do:
git config --global --add push.default tracking
Then you can easily push (and pull) from fedpkg switch-branch setups.
- -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
On Fri, 2010-07-30 at 10:42 -0700, Jesse Keating wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 02:13 AM, Martin Sourada wrote:
Ok, so trying to update a package (libass) I've noticed a tiny problem: the fedpkg switch-branch does not seem to set up proper tracking of remote branches which then results in (uhm, I'm using git commit -a and git push, so you maybe handle this in fedpkg commit -p):
pyfedpkg.FedpkgError: There are unpushed changes in your repo
fedora-packager-0.5.0.1-3.fc12.noarch
This is my fault, I did all my design and testing with a git setting of "push.default tracking" which instructs git to "push" to whatever branch I'm tracking (which makes perfect sense to me, why isn't it a defaut?).
This is probably for historical reasons. Apparently (from what I've quickly googled up) the only behaviour in older git was to push *all* matching branches (not sure how it decides it's matching, but probably it's 'merge' in branch config + same name), but newer git has more options[1]: 'nothing' : Do not push anything 'matching' : Push all matching branches (default) 'tracking' : Push the current branch to whatever it is tracking 'current' : Push the current branch
while defaulting (as hinted above) to the old behaviour.
Martin
References: [1] http://pivotallabs.com/users/alex/blog/articles/883-git-config-push-default-...
Thank you for all the work.
I am currently in the process of convincing my employer to change from svn to git and know how much work this is. Running git-svn clone alone takes a few weeks for us.
I will check out all my packages on the weekend.
Happy Branching!
On 07/30/2010 02:44 PM, Christof Damian wrote:
Thank you for all the work.
I am currently in the process of convincing my employer to change from svn to git and know how much work this is. Running git-svn clone alone takes a few weeks for us.
FYI, git-svn is for users who want to continue to use Subversion on the server but interact using Git from clients. It is not a tool for conversion. It is a common mistake.
Rahul
On Fri, Jul 30, 2010 at 11:18, Rahul Sundaram metherid@gmail.com wrote:
On 07/30/2010 02:44 PM, Christof Damian wrote:
Thank you for all the work.
I am currently in the process of convincing my employer to change from svn to git and know how much work this is. Running git-svn clone alone takes a few weeks for us.
FYI, git-svn is for users who want to continue to use Subversion on the server but interact using Git from clients. It is not a tool for conversion. It is a common mistake.
I know, I have used different tools too. But git-svn deals best with our set-up and it allows incremental syncs which I need at the moment. The others are usually for one-off conversions.
Rahul Sundaram wrote:
FYI, git-svn is for users who want to continue to use Subversion on the server but interact using Git from clients. It is not a tool for conversion. It is a common mistake.
Not so much. Many folks use git-svn to convert from svn to git. Once you have cloned an svn repo with git-svn, you can simply publish the git repo and use it as your new upstream repo. I converted the upstream libgpod and gtkpod repositories this way.
Great news!
Few questions still remains unanswered for me:
* Should I use git now? * May I still use cvs? * If I'll add some changes right now using git/cvs will they be available in cvs/git respectively? * I just noticed that some freshly cloned git repositories are about one month old (namely, couchdb) - do I need to wait until final synchronization before using git for them?
On Fri, Jul 30, 2010 at 11:06 AM, Peter Lemenkov lemenkov@gmail.com wrote:
Great news!
Few questions still remains unanswered for me:
- Should I use git now?
- May I still use cvs?
- If I'll add some changes right now using git/cvs will they be
available in cvs/git respectively?
- I just noticed that some freshly cloned git repositories are about
one month old (namely, couchdb) - do I need to wait until final synchronization before using git for them?
I think you need to get the latest package from koji so it points to the live git repos and not the test ones. I had similar issues when I tested it first up but fedora-packager-0.5.1.0-1 seems to have fixed it.
I believe its all git now and cvs is read only.
Peter
On Fri, Jul 30, 2010 at 11:08:37AM +0100, pbrobinson@gmail.com wrote:
I think you need to get the latest package from koji so it points to the live git repos and not the test ones.
yes, the old version (which is still in F-12) uses pkgs.stg.fedoraproject.org, but the right URL is
pkgs.fedoraproject.org
(see "git config remote.origin.url" for more details).
I had similar issues when I tested it first up but fedora-packager-0.5.1.0-1 seems to have fixed it.
yes
Karel
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 03:06 AM, Peter Lemenkov wrote:
Great news!
Few questions still remains unanswered for me:
- Should I use git now?
Yes.
- May I still use cvs?
read-only
- If I'll add some changes right now using git/cvs will they be
available in cvs/git respectively?
You won't be able to add anything to cvs, so use git.
- I just noticed that some freshly cloned git repositories are about
one month old (namely, couchdb) - do I need to wait until final synchronization before using git for them?
You need a newer fedora-packager.
- -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
On 07/30/2010 05:55 AM, Jesse Keating wrote:
starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).
I made:
fedpkg clone python-debian cd python-debian fedpkg switch-branch f14 edit spec file fedpkg commit -m 'edit changelog entry' -p fedpkg push
And the last thep did not updated that remote branch. I had to manualy run:
git push origin f14:f14/master
which done what I really wanted. This is bug or I done something wrong?
Additionaly: how the new tags in git are going to appear. Is is supposed to create fedpkg? Or should I manually create it using "git tag" and "git push --tags"?
And btw: Thx. Big thx. Now I can really see what I did, which was in cvs impossible (at least with my level of laziness).
On 07/30/2010 02:43 PM, Miroslav Suchý wrote:
On 07/30/2010 05:55 AM, Jesse Keating wrote:
starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).
I made:
fedpkg clone python-debian cd python-debian fedpkg switch-branch f14 edit spec file fedpkg commit -m 'edit changelog entry' -p fedpkg push
And the last thep did not updated that remote branch. I had to manualy run:
git push origin f14:f14/master
or if you have tracking branch `git push origin HEAD'
which done what I really wanted. This is bug or I done something wrong?
Additionaly: how the new tags in git are going to appear. Is is supposed to create fedpkg? Or should I manually create it using "git tag" and "git push --tags"?
And btw: Thx. Big thx. Now I can really see what I did, which was in cvs impossible (at least with my level of laziness).
I am trying to update one of my packages with git for the first time. Works good, but "fedpkg build" fails for me.
It complains that it can't retrieve the sources: http://koji.stg.fedoraproject.org/koji/getfile?taskID=2239266&name=root....
When I open the URL open with firefox/curl they work and the file is the correct one.
Is there any reason why the build system can't access http://cvs.fedoraproject.org/repo/pkgs/php-pdepend-PHP-Depend/PHP_Depend-0.9... ?
Excerpts from Christof Damian's message of Fri Jul 30 16:10:44 +0200 2010:
I am trying to update one of my packages with git for the first time. Works good, but "fedpkg build" fails for me.
It complains that it can't retrieve the sources: http://koji.stg.fedoraproject.org/koji/getfile?taskID=2239266&name=root....
Your fedora-packager package is too old. Install version 0.5.x (from koji, but should be available in the repos now too I believe)
On Fri, Jul 30, 2010 at 16:18, Stanislav Ochotnicky sochotnicky@redhat.com wrote:
Excerpts from Christof Damian's message of Fri Jul 30 16:10:44 +0200 2010:
I am trying to update one of my packages with git for the first time. Works good, but "fedpkg build" fails for me.
It complains that it can't retrieve the sources: http://koji.stg.fedoraproject.org/koji/getfile?taskID=2239266&name=root....
Your fedora-packager package is too old. Install version 0.5.x (from koji, but should be available in the repos now too I believe)
That did the trick. I got fedora-packager-0.5.1.0-1.fc13.noarch from updates testing.
Cheers, Christof
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
fedpkg build fedpkg build Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in <module> args.command(args) File "/usr/bin/fedpkg", line 297, in build mymodule.init_koji(args.user, kojiconfig) File "/usr/lib/python2.7/site-packages/pyfedpkg/__init__.py", line 1102, in init_koji defaults['serverca']) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1628, in ssl_login sinfo = self.callMethod('sslLogin', proxyuser) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1673, in callMethod return self._callMethod(name, args, opts) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1698, in _callMethod return proxy.__getattr__(name)(*args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1570, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in single_request response = h.getresponse(buffering=True) AttributeError: PlgHTTPS instance has no attribute 'getresponse' [Exit 1]
Seems to be broken in Rawhide.
rpm -q fedora-packager fedora-packager-0.5.1.0-1.fc14.noarch
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 08:52 AM, Daniel J Walsh wrote:
fedpkg build fedpkg build Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in <module> args.command(args) File "/usr/bin/fedpkg", line 297, in build mymodule.init_koji(args.user, kojiconfig) File "/usr/lib/python2.7/site-packages/pyfedpkg/__init__.py", line 1102, in init_koji defaults['serverca']) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1628, in ssl_login sinfo = self.callMethod('sslLogin', proxyuser) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1673, in callMethod return self._callMethod(name, args, opts) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1698, in _callMethod return proxy.__getattr__(name)(*args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1570, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in single_request response = h.getresponse(buffering=True) AttributeError: PlgHTTPS instance has no attribute 'getresponse' [Exit 1]
Seems to be broken in Rawhide.
rpm -q fedora-packager fedora-packager-0.5.1.0-1.fc14.noarch
Yeah, this is unfortunate, but something is broken in the xmlrpc stuff in or around koji.
- -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
On 07/30/2010 08:13 PM, Jesse Keating wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 08:52 AM, Daniel J Walsh wrote:
fedpkg build fedpkg build Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in<module> args.command(args) File "/usr/bin/fedpkg", line 297, in build mymodule.init_koji(args.user, kojiconfig) File "/usr/lib/python2.7/site-packages/pyfedpkg/__init__.py", line 1102, in init_koji defaults['serverca']) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1628, in ssl_login sinfo = self.callMethod('sslLogin', proxyuser) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1673, in callMethod return self._callMethod(name, args, opts) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1698, in _callMethod return proxy.__getattr__(name)(*args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1570, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in single_request response = h.getresponse(buffering=True) AttributeError: PlgHTTPS instance has no attribute 'getresponse' [Exit 1]
Seems to be broken in Rawhide.
rpm -q fedora-packager fedora-packager-0.5.1.0-1.fc14.noarch
Yeah, this is unfortunate, but something is broken in the xmlrpc stuff in or around koji.
Any workaround, fix for that?
On 08/02/2010 07:02 PM, Harald Hoyer wrote:
On 07/30/2010 08:13 PM, Jesse Keating wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 08:52 AM, Daniel J Walsh wrote:
fedpkg build fedpkg build Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in<module> args.command(args) File "/usr/bin/fedpkg", line 297, in build mymodule.init_koji(args.user, kojiconfig) File "/usr/lib/python2.7/site-packages/pyfedpkg/__init__.py", line 1102, in init_koji defaults['serverca']) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1628, in ssl_login sinfo = self.callMethod('sslLogin', proxyuser) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1673, in callMethod return self._callMethod(name, args, opts) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1698, in _callMethod return proxy.__getattr__(name)(*args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1570, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in single_request response = h.getresponse(buffering=True) AttributeError: PlgHTTPS instance has no attribute 'getresponse' [Exit 1]
Seems to be broken in Rawhide.
rpm -q fedora-packager fedora-packager-0.5.1.0-1.fc14.noarch
Yeah, this is unfortunate, but something is broken in the xmlrpc stuff in or around koji.
Any workaround, fix for that?
Here is one attached
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 05:43 AM, Miroslav Suchý wrote:
On 07/30/2010 05:55 AM, Jesse Keating wrote:
starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).
I made:
fedpkg clone python-debian cd python-debian fedpkg switch-branch f14 edit spec file fedpkg commit -m 'edit changelog entry' -p fedpkg push
And the last thep did not updated that remote branch. I had to manualy run:
git push origin f14:f14/master
which done what I really wanted. This is bug or I done something wrong?
One part bug, one part user configuration.
If you do "git config --add --global push.default tracking" then fedpkg push would do as you expect. I need to make fedpkg push work regardless of that local setting.
Additionaly: how the new tags in git are going to appear. Is is supposed to create fedpkg? Or should I manually create it using "git tag" and "git push --tags"?
You don't need tags to build anymore, so fedpkg doesn't do anything with them at this point.
And btw: Thx. Big thx. Now I can really see what I did, which was in cvs impossible (at least with my level of laziness).
- -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
Many thanks!
Thanks for dist-git -- one step closer to banishing CVS from my life! :)
A git trick I'd like fedpkg to learn is to use separate url/pushurl, e.g. in .git/config:
[remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* url = git://pkgs.fedoraproject.org/foo pushurl = ssh://user@pkgs.fedoraproject.org/foo
This way it will fetch over the faster git protocol and still push over ssh. I expect that will lessen the load on Fedora infrastructure too.
Thanks!
Josh
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 09:14 AM, Josh Stone wrote:
Thanks for dist-git -- one step closer to banishing CVS from my life! :)
A git trick I'd like fedpkg to learn is to use separate url/pushurl, e.g. in .git/config:
[remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* url = git://pkgs.fedoraproject.org/foo pushurl = ssh://user@pkgs.fedoraproject.org/foo
This way it will fetch over the faster git protocol and still push over ssh. I expect that will lessen the load on Fedora infrastructure too.
Thanks!
Josh
Hrm, not sure this is something we should do by default, are there any drawbacks?
- -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
On Fri, 2010-07-30 at 11:15 -0700, Jesse Keating wrote:
On 07/30/2010 09:14 AM, Josh Stone wrote:
A git trick I'd like fedpkg to learn is to use separate url/pushurl, e.g. in .git/config:
[remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* url = git://pkgs.fedoraproject.org/foo pushurl = ssh://user@pkgs.fedoraproject.org/foo
This way it will fetch over the faster git protocol and still push over ssh. I expect that will lessen the load on Fedora infrastructure too.
Hrm, not sure this is something we should do by default, are there any drawbacks?
You need two network protocols to do work on the package. That might be difficult in some restricted networks.
For everyone with actual proper internet access, this looks like it could work quite well.
On 07/30/2010 11:36 AM, Hans Ulrich Niedermann wrote:
On Fri, 2010-07-30 at 11:15 -0700, Jesse Keating wrote:
On 07/30/2010 09:14 AM, Josh Stone wrote:
A git trick I'd like fedpkg to learn is to use separate url/pushurl, e.g. in .git/config:
[remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* url = git://pkgs.fedoraproject.org/foo pushurl = ssh://user@pkgs.fedoraproject.org/foo
This way it will fetch over the faster git protocol and still push over ssh. I expect that will lessen the load on Fedora infrastructure too.
Hrm, not sure this is something we should do by default, are there any drawbacks?
You need two network protocols to do work on the package. That might be difficult in some restricted networks.
Good point, I didn't think of that.
For everyone with actual proper internet access, this looks like it could work quite well.
It's quite an improvement for me on other projects. For now I've made that change manually in my fedpkg checkout. If we don't make this by default, perhaps it could still be a command-line option or a ~/.fedpkgrc entry.
Josh
Josh Stone jistone@redhat.com writes:
A git trick I'd like fedpkg to learn is to use separate url/pushurl, e.g. in .git/config:
[remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* url = git://pkgs.fedoraproject.org/foo pushurl = ssh://user@pkgs.fedoraproject.org/foo
Or more general:
[url "ssh://pkgs.fedoraproject.org/"] pushinsteadof = git://pkgs.fedoraproject.org/
Andreas.
On Fri, 2010-07-30 at 09:14 -0700, Josh Stone wrote:
A git trick I'd like fedpkg to learn is to use separate url/pushurl, e.g. in .git/config:
[remote "origin"] fetch = +refs/heads/*:refs/remotes/origin/* url = git://pkgs.fedoraproject.org/foo pushurl = ssh://user@pkgs.fedoraproject.org/foo
This way it will fetch over the faster git protocol and still push over ssh. I expect that will lessen the load on Fedora infrastructure too.
The ssh:// URL does use the git protocol over ssh, so the git:// URL is only faster by the lack of encryption, which is hardly a good thing. I don't want malware landing on my machine because someone did a MITM attack on a Fedora maintainer's unencrypted "git fetch" and inserted some extra patches to get pushed back to the real repository later.
On Tue, Aug 3, 2010 at 11:16 AM, Matt McCutchen matt@mattmccutchen.net wrote:
don't want malware landing on my machine because someone did a MITM attack on a Fedora maintainer's unencrypted "git fetch" and inserted some extra patches to get pushed back to the real repository later.
The git protocol makes it extremely hard to inject malware successfully. It would have to match sha1, _and_ match resulting filesize _and_ be meaningful code, all without the benefits of preimaging.
Even for crypto hashes that have been "broken" for a while, doing the above is a huge challenge.
If you do consider this a real risk, here's someone who wants to want to play with you, and build a bunker, 5 miles underground... http://marc.info/?l=git&m=111375923219555&w=2
:-)
martin (formerly, a git hacker)
On Tue, 2010-08-03 at 11:29 -0400, Martin Langhoff wrote:
On Tue, Aug 3, 2010 at 11:16 AM, Matt McCutchen matt@mattmccutchen.net wrote:
don't want malware landing on my machine because someone did a MITM attack on a Fedora maintainer's unencrypted "git fetch" and inserted some extra patches to get pushed back to the real repository later.
The git protocol makes it extremely hard to inject malware successfully. It would have to match sha1, _and_ match resulting filesize _and_ be meaningful code, all without the benefits of preimaging.
No. If the attacker MITMs the entire connection, they can lie about the values of the remote refs too, so there is no need to find a hash collision.
Matt McCutchen matt@mattmccutchen.net wrote:
No. If the attacker MITMs the entire connection, they can lie about the values of the remote refs too, so there is no need to find a hash collision.
And how would you then be allowed to push? The git server would see that your history doesn't match the history it has and will refuse the commits. If they MITM your SSH push connection, I believe you have bigger fish to fry.
--Ben
On Tue, 2010-08-03 at 22:09 +0000, Ben Boeckel wrote:
Matt McCutchen matt@mattmccutchen.net wrote:
No. If the attacker MITMs the entire connection, they can lie about the values of the remote refs too, so there is no need to find a hash collision.
And how would you then be allowed to push? The git server would see that your history doesn't match the history it has and will refuse the commits.
When the maintainer fetches, the attacker adds malicious commits on top of the real remote ref value, and then the maintainer pushes those commits as if he committed them himself. But IMNSHO, malicious alteration of a fetch is something maintainers shouldn't have to deal with, regardless of what the consequences might or might not be.
(I should have changed the subject two round trips ago. Oh well...)
On Wed, 2010-08-04 at 01:33 -0700, Matt McCutchen wrote:
On Tue, 2010-08-03 at 22:09 +0000, Ben Boeckel wrote:
Matt McCutchen matt@mattmccutchen.net wrote:
No. If the attacker MITMs the entire connection, they can lie about the values of the remote refs too, so there is no need to find a hash collision.
And how would you then be allowed to push? The git server would see that your history doesn't match the history it has and will refuse the commits.
When the maintainer fetches, the attacker adds malicious commits on top of the real remote ref value, and then the maintainer pushes those commits as if he committed them himself. But IMNSHO, malicious alteration of a fetch is something maintainers shouldn't have to deal with, regardless of what the consequences might or might not be.
(I should have changed the subject two round trips ago. Oh well...)
I suspect it might short-circuit the 'ahhh, but what about...' 'oooh, but then I can...' nature of the conversation if you just put together a proof-of-concept attack and document it somewhere. I suspect the git maintainers might be interested at that point as well. :)
On Wed, Aug 04, 2010 at 09:42:01AM -0700, Adam Williamson wrote:
I suspect it might short-circuit the 'ahhh, but what about...' 'oooh, but then I can...' nature of the conversation if you just put together a proof-of-concept attack and document it somewhere. I suspect the git maintainers might be interested at that point as well. :)
The attack is quite trivial: 1) clone the git pkg Fedora repos 2) commit some nasty change 3) publish the repo on some server 4) if the victim wants to fetch from the Fedora pkg repo, use the MITM attack to make him fetch from the server set up in step 3. Steps 1-3 can obviously be done on-demand.
If this is e.g. done on a conference / FUDCon / Fedora Action Day, the attack can easily targeted to make the change in step 2 be expected to be fast forward. E.g. if packages simply need to be bumped for a rebuild, a upload of a bad tarball and modification of the sources file might be unnoticed.
Regards Till
On Wed, 04 Aug 2010 22:03:14 +0200 Till Maas opensource@till.name wrote:
On Wed, Aug 04, 2010 at 09:42:01AM -0700, Adam Williamson wrote:
I suspect it might short-circuit the 'ahhh, but what about...' 'oooh, but then I can...' nature of the conversation if you just put together a proof-of-concept attack and document it somewhere. I suspect the git maintainers might be interested at that point as well. :)
The attack is quite trivial:
- clone the git pkg Fedora repos
- commit some nasty change
- publish the repo on some server
- if the victim wants to fetch from the Fedora pkg repo, use the MITM
attack to make him fetch from the server set up in step 3. Steps 1-3 can obviously be done on-demand.
If this is e.g. done on a conference / FUDCon / Fedora Action Day, the attack can easily targeted to make the change in step 2 be expected to be fast forward. E.g. if packages simply need to be bumped for a rebuild, a upload of a bad tarball and modification of the sources file might be unnoticed.
Just to clarify, as this is a long thread:
This only works if people are using git:// urls, not the default for fedora ssh: ones, right? (provided you have connected before to pkgs.fedoraproject.org and have the known_hosts entry?)
kevin
On Thu, Aug 05, 2010 at 01:11:24PM -0600, Kevin Fenzi wrote:
On Wed, 04 Aug 2010 22:03:14 +0200 Till Maas opensource@till.name wrote:
The attack is quite trivial:
- clone the git pkg Fedora repos
- commit some nasty change
- publish the repo on some server
- if the victim wants to fetch from the Fedora pkg repo, use the MITM
attack to make him fetch from the server set up in step 3. Steps 1-3 can obviously be done on-demand.
If this is e.g. done on a conference / FUDCon / Fedora Action Day, the attack can easily targeted to make the change in step 2 be expected to be fast forward. E.g. if packages simply need to be bumped for a rebuild, a upload of a bad tarball and modification of the sources file might be unnoticed.
Just to clarify, as this is a long thread:
This only works if people are using git:// urls, not the default for fedora ssh: ones, right? (provided you have connected before to pkgs.fedoraproject.org and have the known_hosts entry?)
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
Regards Till
On Thu, 5 Aug 2010, Till Maas wrote:
On Thu, Aug 05, 2010 at 01:11:24PM -0600, Kevin Fenzi wrote:
On Wed, 04 Aug 2010 22:03:14 +0200 Till Maas opensource@till.name wrote:
The attack is quite trivial:
- clone the git pkg Fedora repos
- commit some nasty change
- publish the repo on some server
- if the victim wants to fetch from the Fedora pkg repo, use the MITM
attack to make him fetch from the server set up in step 3. Steps 1-3 can obviously be done on-demand.
If this is e.g. done on a conference / FUDCon / Fedora Action Day, the attack can easily targeted to make the change in step 2 be expected to be fast forward. E.g. if packages simply need to be bumped for a rebuild, a upload of a bad tarball and modification of the sources file might be unnoticed.
Just to clarify, as this is a long thread:
This only works if people are using git:// urls, not the default for fedora ssh: ones, right? (provided you have connected before to pkgs.fedoraproject.org and have the known_hosts entry?)
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
We also use SSHFP records for those of you that want to enable VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts have it but many of our 'user' based external hosts do (pkgs, fedorapeople, fedorahosted, etc)
-Mike
On Thu, Aug 05, 2010 at 04:32:36PM -0500, Mike McGrath wrote:
On Thu, 5 Aug 2010, Till Maas wrote:
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
We also use SSHFP records for those of you that want to enable VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts have it but many of our 'user' based external hosts do (pkgs, fedorapeople, fedorahosted, etc)
Afaik the SSHFP records are not protected against tampering by an MITM attacker.
Regards Till
On Fri, 6 Aug 2010, Till Maas wrote:
On Thu, Aug 05, 2010 at 04:32:36PM -0500, Mike McGrath wrote:
On Thu, 5 Aug 2010, Till Maas wrote:
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
We also use SSHFP records for those of you that want to enable VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts have it but many of our 'user' based external hosts do (pkgs, fedorapeople, fedorahosted, etc)
Afaik the SSHFP records are not protected against tampering by an MITM attacker.
They're better then ssh alone. They're only used for the first initation. So you'd have to be MITM'ed on the first connection in which case you're right, they wouldn't protect against that.
-Mike
On Fri, 6 Aug 2010, Mike McGrath wrote:
On Fri, 6 Aug 2010, Till Maas wrote:
On Thu, Aug 05, 2010 at 04:32:36PM -0500, Mike McGrath wrote:
On Thu, 5 Aug 2010, Till Maas wrote:
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
We also use SSHFP records for those of you that want to enable VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts have it but many of our 'user' based external hosts do (pkgs, fedorapeople, fedorahosted, etc)
Afaik the SSHFP records are not protected against tampering by an MITM attacker.
They're better then ssh alone. They're only used for the first initation. So you'd have to be MITM'ed on the first connection in which case you're right, they wouldn't protect against that.
Side note on this: Combined with DNSSEC, this actually would protect against a MITM attacker.
-Mike
On Fri, Aug 06, 2010 at 04:34:19AM -0500, Mike McGrath wrote:
On Fri, 6 Aug 2010, Mike McGrath wrote:
On Fri, 6 Aug 2010, Till Maas wrote:
On Thu, Aug 05, 2010 at 04:32:36PM -0500, Mike McGrath wrote:
We also use SSHFP records for those of you that want to enable VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts have it but many of our 'user' based external hosts do (pkgs, fedorapeople, fedorahosted, etc)
Afaik the SSHFP records are not protected against tampering by an MITM attacker.
<snip>
Side note on this: Combined with DNSSEC, this actually would protect against a MITM attacker.
Not only that, but they *are* signed, and the signing key is in the ISC DLV registry. You just need a name server that has DNSSEC enabled and uses the ISC DLV registry (output trimmed for relevance):
$ dig +adflag -t sshfp pkgs.fedoraproject.org ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
;; QUESTION SECTION: ;pkgs.fedoraproject.org. IN SSHFP
;; ANSWER SECTION: pkgs.fedoraproject.org. 3600 IN SSHFP 1 1 2AB094461D34656B4704B4EDBCF002A09EBCE4F7
Note the ad flag, which indicates that the answer is authentic.
On Fri, Aug 06, 2010 at 04:31:00AM -0500, Mike McGrath wrote:
On Fri, 6 Aug 2010, Till Maas wrote:
On Thu, Aug 05, 2010 at 04:32:36PM -0500, Mike McGrath wrote:
On Thu, 5 Aug 2010, Till Maas wrote:
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
We also use SSHFP records for those of you that want to enable VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts have it but many of our 'user' based external hosts do (pkgs, fedorapeople, fedorahosted, etc)
Afaik the SSHFP records are not protected against tampering by an MITM attacker.
They're better then ssh alone. They're only used for the first initation. So you'd have to be MITM'ed on the first connection in which case you're right, they wouldn't protect against that.
Afaik using the SSHFP records make SSH not warn the user that the host key is not verified. If SSH would e.g. warn that the host key is unknown, but at least matches the SSHFP record, then it might be a little better. But actually it makes MITM attacks easier, because if one tampers the DNS response and the SSH connection, then the user does not even get a warning on the first attempt, making the situation even worse IMHO.
And SSH is only vulnerable to MITM attacks on the first connection in general and I guess that SSHFP records are not used anymore after the first connection. What would they be good for when the host key is already known to SSH?
Regards Till
On Fri, Aug 06, 2010 at 12:54:23PM +0200, Till Maas wrote:
On Fri, Aug 06, 2010 at 04:31:00AM -0500, Mike McGrath wrote:
On Fri, 6 Aug 2010, Till Maas wrote:
On Thu, Aug 05, 2010 at 04:32:36PM -0500, Mike McGrath wrote:
On Thu, 5 Aug 2010, Till Maas wrote:
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
We also use SSHFP records for those of you that want to enable VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts have it but many of our 'user' based external hosts do (pkgs, fedorapeople, fedorahosted, etc)
Afaik the SSHFP records are not protected against tampering by an MITM attacker.
They're better then ssh alone. They're only used for the first initation. So you'd have to be MITM'ed on the first connection in which case you're right, they wouldn't protect against that.
Afaik using the SSHFP records make SSH not warn the user that the host key is not verified. If SSH would e.g. warn that the host key is unknown, but at least matches the SSHFP record, then it might be a little better. But actually it makes MITM attacks easier, because if one tampers the DNS response and the SSH connection, then the user does not
SSH considers SSHFP valid only when it is verified via DNSSEC. So it's not possible to perform DNS spoofing (or MITM) when DNSSEC is used.
When ssh receives SSHFP record which is not verified then it falls back to "standard" (i.e. yes/no method) verification.
even get a warning on the first attempt, making the situation even worse IMHO.
And SSH is only vulnerable to MITM attacks on the first connection in general and I guess that SSHFP records are not used anymore after the first connection. What would they be good for when the host key is already known to SSH?
SSHFP records are useless when fingerprint is already known.
Regards, Adam
On Fri, Aug 06, 2010 at 12:54:23PM +0200, Till Maas wrote:
On Fri, Aug 06, 2010 at 04:31:00AM -0500, Mike McGrath wrote:
On Fri, 6 Aug 2010, Till Maas wrote:
On Thu, Aug 05, 2010 at 04:32:36PM -0500, Mike McGrath wrote:
We also use SSHFP records for those of you that want to enable VerifyHostKeyDNS yes in their ~/.ssh/config files. Not all of our hosts have it but many of our 'user' based external hosts do (pkgs, fedorapeople, fedorahosted, etc)
Afaik the SSHFP records are not protected against tampering by an MITM attacker.
They're better then ssh alone. They're only used for the first initation. So you'd have to be MITM'ed on the first connection in which case you're right, they wouldn't protect against that.
Afaik using the SSHFP records make SSH not warn the user that the host key is not verified. If SSH would e.g. warn that the host key is unknown, but at least matches the SSHFP record, then it might be a little better. But actually it makes MITM attacks easier, because if one tampers the DNS response and the SSH connection, then the user does not even get a warning on the first attempt, making the situation even worse IMHO.
And SSH is only vulnerable to MITM attacks on the first connection in general and I guess that SSHFP records are not used anymore after the first connection. What would they be good for when the host key is already known to SSH?
openssh is more paranoid than that. An unsigned, unvalidated SSHFP record will be treated just like the server response is today -- the user will be shown the fingerprint and asked if it's correct.
Only if the response is marked with the AD flag (indicating that the resolver has checked the signature and it's ok) does openssh trust it automatically. I know this because I've tried all three: unsigned, signed but no AD flag on the response, signed with AD flag.
If you really want to argue about security, I'll point out that this isn't completely bulletproof the way Fedora works today. The host (your computer) runs a stub resolver that communicates with the recursive DNS server (typically another computer) over an unauthenticated connection. The stub resolver trusts the recursive resolver to only set the AD flag in its response when the answer is trusted, and to not set it when it's not. If you're talking directly to the authoritative nameserver, it's not allowed to validate its own response, so it'll never set the AD flag.
Ideally (from this perspective), the host would validate the response itself.
On Thu, 2010-08-05 at 22:23 +0200, Till Maas wrote:
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
I'm very glad to see that Fedora provides such a list. I just installed it on my computer (after filtering out hostnames not ending with fedoraproject.org, for obvious reasons).
Is it documented anywhere? For full security, every packager should install it rather than allowing ssh to add host keys on first use.
On Sun, Aug 8, 2010 at 14:04, Matt McCutchen matt@mattmccutchen.net wrote:
On Thu, 2010-08-05 at 22:23 +0200, Till Maas wrote:
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
I'm very glad to see that Fedora provides such a list. I just installed it on my computer (after filtering out hostnames not ending with fedoraproject.org, for obvious reasons).
Is it documented anywhere? For full security, every packager should install it rather than allowing ssh to add host keys on first use.
Well I am not sure that file would be all that useful as it contains lots of hosts a packager would not get to AND could conflict with other networks as it contains a lot of 10.X.X. and 192.X.X. ips. It also gets updated from time to time as we rebuild hosts.
-- Matt
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Tue, 2010-08-10 at 09:07 -0600, Stephen John Smoogen wrote:
On Sun, Aug 8, 2010 at 14:04, Matt McCutchen matt@mattmccutchen.net wrote:
On Thu, 2010-08-05 at 22:23 +0200, Till Maas wrote:
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
I'm very glad to see that Fedora provides such a list. I just installed it on my computer (after filtering out hostnames not ending with fedoraproject.org, for obvious reasons).
Is it documented anywhere? For full security, every packager should install it rather than allowing ssh to add host keys on first use.
Well I am not sure that file would be all that useful as it contains lots of hosts a packager would not get to AND could conflict with other networks as it contains a lot of 10.X.X. and 192.X.X. ips.
Then let's post an excerpt that would be useful to packagers.
It also gets updated from time to time as we rebuild hosts.
That just speaks to the need for better tooling to maintain personal known-hosts files, or for Fedora to operate an ssh certificate authority.
It appears that the ssh folks rejected X.509 out of disgust for the public CAs, found themselves left with no solution at all to authenticate hosts the first time, and are now reimplementing it incompatibly. The man page claims the ssh implementation is "much simpler" -- perhaps, but it won't integrate with X.509-based systems and will be playing catch-up on features for a while. CRLs or OCSP, anyone?
A thread from 2002 with some frank discussion that is still valid now:
http://marc.info/?t=101179752100001&r=1&w=2
On Wed, 2010-08-11 at 01:55 -0700, Matt McCutchen wrote:
On Tue, 2010-08-10 at 09:07 -0600, Stephen John Smoogen wrote:
On Sun, Aug 8, 2010 at 14:04, Matt McCutchen matt@mattmccutchen.net wrote:
On Thu, 2010-08-05 at 22:23 +0200, Till Maas wrote:
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
I'm very glad to see that Fedora provides such a list. I just installed it on my computer (after filtering out hostnames not ending with fedoraproject.org, for obvious reasons).
Is it documented anywhere? For full security, every packager should install it rather than allowing ssh to add host keys on first use.
Well I am not sure that file would be all that useful as it contains lots of hosts a packager would not get to AND could conflict with other networks as it contains a lot of 10.X.X. and 192.X.X. ips.
Then let's post an excerpt that would be useful to packagers.
It also gets updated from time to time as we rebuild hosts.
That just speaks to the need for better tooling to maintain personal known-hosts files, or for Fedora to operate an ssh certificate authority.
It appears that the ssh folks rejected X.509 out of disgust for the public CAs, found themselves left with no solution at all to authenticate hosts the first time, and are now reimplementing it incompatibly. The man page claims the ssh implementation is "much simpler" -- perhaps, but it won't integrate with X.509-based systems and will be playing catch-up on features for a while. CRLs or OCSP, anyone?
A thread from 2002 with some frank discussion that is still valid now:
The PKI is unfortunately hopelessly broken deep in its concepts.
See for example here: http://searchsecurity.techtarget.com/news/interview/0,289202,sid14_gci136014...
The DNSSSEC is much more reasonable way to go.
But we are getting off-topic here.
On Tue, Aug 10, 2010 at 09:07:21AM -0600, Stephen John Smoogen wrote:
On Sun, Aug 8, 2010 at 14:04, Matt McCutchen matt@mattmccutchen.net wrote:
On Thu, 2010-08-05 at 22:23 +0200, Till Maas wrote:
Yes ssh is secure if used properly. To get the proper known_hosts entry, one has to download https://admin.fedoraproject.org/ssh_known_hosts btw.
I'm very glad to see that Fedora provides such a list. I just installed it on my computer (after filtering out hostnames not ending with fedoraproject.org, for obvious reasons).
Is it documented anywhere? For full security, every packager should install it rather than allowing ssh to add host keys on first use.
Well I am not sure that file would be all that useful as it contains lots of hosts a packager would not get to AND could conflict with other networks as it contains a lot of 10.X.X. and 192.X.X. ips. It also gets updated from time to time as we rebuild hosts.
I cleaned up my tools to manage the fedora ssh known hosts file and copied it to fedorapeople.org: http://till.fedorapeople.org/files/fedora-ssh-known-hosts/
It allows to easily update the file and to filter out unwanted entries.
Regards Till
On Tue, 2010-08-03 at 11:29 -0400, Martin Langhoff wrote:
On Tue, Aug 3, 2010 at 11:16 AM, Matt McCutchen matt@mattmccutchen.net wrote:
don't want malware landing on my machine because someone did a MITM attack on a Fedora maintainer's unencrypted "git fetch" and inserted some extra patches to get pushed back to the real repository later.
The git protocol makes it extremely hard to inject malware successfully. It would have to match sha1, _and_ match resulting filesize _and_ be meaningful code, all without the benefits of preimaging.
Even for crypto hashes that have been "broken" for a while, doing the above is a huge challenge.
If you do consider this a real risk, here's someone who wants to want to play with you, and build a bunker, 5 miles underground... http://marc.info/?l=git&m=111375923219555&w=2
I have to say I was tickled by Linus' imagination of how five year olds behave:
"That's not engineering. That's five-year-olds discussing building their imaginary forts ("I want gun-turrets and a mechanical horse one mile high, and my command center is 5 miles under-ground and totally encased in 5 meters of lead")."
Clearly Linus stood out even in his youth =)
On Thu, Jul 29, 2010 at 11:55 PM, Jesse Keating jkeating@redhat.com wrote:
Once again I want to thank everybody who helped out and for all the (continued) patience! I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues. Look for Oxf13 on freenode. Happy gitting!
Hmm... Doesn't seem to be working at all for me.
$ fedpkg co verbiste Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in <module> args.command(args) File "/usr/bin/fedpkg", line 409, in clone pyfedpkg.clone(args.module[0], args.user, args.path, args.branch) File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 302, in clone _run_command(cmd) File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 115, in _run_command stderr=sys.stderr, shell=shell) File "/usr/lib64/python2.6/subprocess.py", line 483, in check_call retcode = call(*popenargs, **kwargs) File "/usr/lib64/python2.6/subprocess.py", line 470, in call return Popen(*popenargs, **kwargs).wait() File "/usr/lib64/python2.6/subprocess.py", line 621, in __init__ errread, errwrite) File "/usr/lib64/python2.6/subprocess.py", line 1126, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory $ rpm -q fedora-packager fedora-packager-0.5.1.0-1.fc13.noarch
Do I need to run anything before I can use fedpkg? I couldn't find anything about configuration files on the Using_Fedora_GIT wiki page.
I filed it as https://bugzilla.redhat.com/show_bug.cgi?id=619763
Regards,
On Fri, Jul 30, 2010 at 1:35 PM, Konstantin Ryabitsev icon@fedoraproject.org wrote:
Hmm... Doesn't seem to be working at all for me.
$ fedpkg co verbiste Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in <module> args.command(args) File "/usr/bin/fedpkg", line 409, in clone pyfedpkg.clone(args.module[0], args.user, args.path, args.branch) File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 302, in clone _run_command(cmd) File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 115, in _run_command stderr=sys.stderr, shell=shell) File "/usr/lib64/python2.6/subprocess.py", line 483, in check_call retcode = call(*popenargs, **kwargs) File "/usr/lib64/python2.6/subprocess.py", line 470, in call return Popen(*popenargs, **kwargs).wait() File "/usr/lib64/python2.6/subprocess.py", line 621, in __init__ errread, errwrite) File "/usr/lib64/python2.6/subprocess.py", line 1126, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory $ rpm -q fedora-packager fedora-packager-0.5.1.0-1.fc13.noarch
Ah, as silly as it was, I didn't actually check whether git was installed -- I rather assumed it would be as a dependency. Everything works after actually installing git (though it's still a bug in fedora-packager deps).
Regards,
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/30/2010 10:53 AM, Konstantin Ryabitsev wrote:
On Fri, Jul 30, 2010 at 1:35 PM, Konstantin Ryabitsev icon@fedoraproject.org wrote:
Hmm... Doesn't seem to be working at all for me.
$ fedpkg co verbiste Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in <module> args.command(args) File "/usr/bin/fedpkg", line 409, in clone pyfedpkg.clone(args.module[0], args.user, args.path, args.branch) File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 302, in clone _run_command(cmd) File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 115, in _run_command stderr=sys.stderr, shell=shell) File "/usr/lib64/python2.6/subprocess.py", line 483, in check_call retcode = call(*popenargs, **kwargs) File "/usr/lib64/python2.6/subprocess.py", line 470, in call return Popen(*popenargs, **kwargs).wait() File "/usr/lib64/python2.6/subprocess.py", line 621, in __init__ errread, errwrite) File "/usr/lib64/python2.6/subprocess.py", line 1126, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory $ rpm -q fedora-packager fedora-packager-0.5.1.0-1.fc13.noarch
Ah, as silly as it was, I didn't actually check whether git was installed -- I rather assumed it would be as a dependency. Everything works after actually installing git (though it's still a bug in fedora-packager deps).
Regards,
Hrm, this is actually a bug in GitPython. It needs to depend on /usr/bin/git.
- -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating
On 7/29/2010 10:55 PM, Jesse Keating wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This evening we opened up dist-git for business. Dist-git is our git based replacement for dist-cvs, the source control system we were using for our package specs, patches, and source files. This has been a long time coming and a massive effort. I want to take a little time here to outline what we've done and where we are going.
First a brief outline of how our CVS system worked. CVS is a daemon of sorts, and all repos typically live within a single CVSROOT. Within this CVSROOT we had an 'avail' system to make use of, that we could populate with data from Fedora Account System and dump into this file. Avail worked on path names, relative to the CVSROOT. Since we used directories for each Fedora release as pseudo branches we could set avail info on each release "branch". CVS also used some filesystem symlink tricks to create a "common" subdir for every package module, and this is where we stuffed common scripts and Makefile content. Pretty clever on one hand, we can make updates to the make system without touching every single package, but it is pretty hackish and we had constant issues where somebody would attempt an action using old common content and stuff would fall over.
Now we look at git. Git is for the most part a daemonless system. Each repository is completely stand alone and generally does not require any other infrastructure to be useful. You can interact with a repository directly on the filesystem using /usr/bin/git or you can interact with it through say ssh, again using /usr/bin/git (your local /usr/bin/git will call a remote /usr/bin/git). Generally there is no running daemon to connect to and authenticate with. Basic authentication of who can check out what and commit where with git happens at the filesystem permissions level. One twist with this is that with git, we wanted to use real branches within a package repo to reflect the different Fedora/EPEL releases. In repo branches are not reflected with path names that we could set filesystem ACLs upon, so this posed a problem for our conversion.
Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an addon system to git that provides ACL functionality including different rights for different branches within a given repository, and more! By using gitolite as a replacement for /usr/bin/git when a user connects to our git server we can again utilize the information we have within the Fedora Package Database and properly allow / restrict changes on specific branches for specific developers. The gitolite upstream ( Sitaram Chamarty, sitaram@atc.tcs.com ) has been fantastically responsive to our needs, which are admittedly a little unique. We have a very large set of repositories (over 10.5K) and a largish number of contributors (1050). The combination of the two leads to a very very large and complex ACL structure that at first broke the system quite badly. Upstream was very quick to create a "bigconfig" method of compiling the ACLs without crashing the box. Our other unique needs involve having individual accounts for each committer instead of a shared account with a large list of allowed SSH keys. Add to that some of our accounts need to be able to ssh shell into the git server for administrative duties. Throughout our trials and testing with gitolite every time we've ran into some issue that just didn't fit the mold, Sitaram has been there with a smile and a fix. At this point our production server is a whopping one line different from current gitolite upstream. This is a fantastic win for us, for our sustainability, and for the next large group that wants to make use of gitolite.
Once we had a plan for ACLs and for branches, we had to decide if we were going to replace the Make system and with what. I had never been a fan of Make, it was entirely too difficult to modify and innovate with. Since I was the one pushing this transition forward, I decided to write a new tool in my favorite language, python. The fedpkg tool was born and took off. fedpkg was born around January 4th, 2010 and has since grown into 1,523 (via sloccount) lines of code. While far from complete, it is a great start (if I do say so myself!) at replacing the make system. Because it is written in python (or maybe just !Make) it seems to be easier to contribute to, and I've already gotten a number of contributions. More will come as it starts to be more widely used. The biggest challenge with fedpkg is removing the need to update something on the end user's system every single time we added a new Fedora release and changed what happens when you build for rawhide. I'll spare you the details but I'm fairly happy with what we have. The end result should be far fewer misfires and end user confusion.
The last major piece of the puzzle was how to actually convert the existing CVS repositories, including the fun pseudo branches, into git repositories. I tried a number of options over the years (I've been working on this off and on for nearly 4 years!) ranging from the built in git cvsimport to git-svn to parsecvs and a few others. In the end, we took a page from the gnome project and used parsecvs ( http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of our repositories. There were a few that gave parsecvs fits and recent versions of git cvsimport were able to handle them. The git system is fantastic enough that we were able to merge our pseudo cvs branches into actual git branches complete with a real shared history, but again I'll spare you the details of the scripting to do this. All but the kernel repo seems to have converted successfully which is a pretty good success rate in my book. We may yet get the kernel converted, but in the interest of time we opted to start fresh with dist-git for now.
Without the help of many others, this project would never have gotten done. Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way. Things will be bumpy over the next few weeks as we really start putting distgit to the test. No amount of staging and testing can really replace production use. There will be many more updates to fedpkg as bugs are found and fixed and features are contributed. Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).
When I do fedpkg clone -B libfoo, to get the old-style layout, all the branches are identical to devel. The branches on the standard clone seem fine. I've not made any changes with the -B layout, but I thought this should be corrected or deprecated.
fedora-packager-0.5.1.0-1.fc13.noarch
-J
Once again I want to thank everybody who helped out and for all the (continued) patience! I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues. Look for Oxf13 on freenode. Happy gitting!
Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAkxSTR0ACgkQ4v2HLvE71NXEswCeOAu+EG/porsvkjMVq+4lAchy VMgAn1DNwqyYcSSC3bwX/MQ/cfwx7qjs =fBGf -----END PGP SIGNATURE----- _______________________________________________ devel-announce mailing list devel-announce@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel-announce
On 07/30/2010 03:02 PM, Jon Ciesla wrote:
On 7/29/2010 10:55 PM, Jesse Keating wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This evening we opened up dist-git for business. Dist-git is our git based replacement for dist-cvs, the source control system we were using for our package specs, patches, and source files. This has been a long time coming and a massive effort. I want to take a little time here to outline what we've done and where we are going.
First a brief outline of how our CVS system worked. CVS is a daemon of sorts, and all repos typically live within a single CVSROOT. Within this CVSROOT we had an 'avail' system to make use of, that we could populate with data from Fedora Account System and dump into this file. Avail worked on path names, relative to the CVSROOT. Since we used directories for each Fedora release as pseudo branches we could set avail info on each release "branch". CVS also used some filesystem symlink tricks to create a "common" subdir for every package module, and this is where we stuffed common scripts and Makefile content. Pretty clever on one hand, we can make updates to the make system without touching every single package, but it is pretty hackish and we had constant issues where somebody would attempt an action using old common content and stuff would fall over.
Now we look at git. Git is for the most part a daemonless system. Each repository is completely stand alone and generally does not require any other infrastructure to be useful. You can interact with a repository directly on the filesystem using /usr/bin/git or you can interact with it through say ssh, again using /usr/bin/git (your local /usr/bin/git will call a remote /usr/bin/git). Generally there is no running daemon to connect to and authenticate with. Basic authentication of who can check out what and commit where with git happens at the filesystem permissions level. One twist with this is that with git, we wanted to use real branches within a package repo to reflect the different Fedora/EPEL releases. In repo branches are not reflected with path names that we could set filesystem ACLs upon, so this posed a problem for our conversion.
Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an addon system to git that provides ACL functionality including different rights for different branches within a given repository, and more! By using gitolite as a replacement for /usr/bin/git when a user connects to our git server we can again utilize the information we have within the Fedora Package Database and properly allow / restrict changes on specific branches for specific developers. The gitolite upstream ( Sitaram Chamarty, sitaram@atc.tcs.com ) has been fantastically responsive to our needs, which are admittedly a little unique. We have a very large set of repositories (over 10.5K) and a largish number of contributors (1050). The combination of the two leads to a very very large and complex ACL structure that at first broke the system quite badly. Upstream was very quick to create a "bigconfig" method of compiling the ACLs without crashing the box. Our other unique needs involve having individual accounts for each committer instead of a shared account with a large list of allowed SSH keys. Add to that some of our accounts need to be able to ssh shell into the git server for administrative duties. Throughout our trials and testing with gitolite every time we've ran into some issue that just didn't fit the mold, Sitaram has been there with a smile and a fix. At this point our production server is a whopping one line different from current gitolite upstream. This is a fantastic win for us, for our sustainability, and for the next large group that wants to make use of gitolite.
Once we had a plan for ACLs and for branches, we had to decide if we were going to replace the Make system and with what. I had never been a fan of Make, it was entirely too difficult to modify and innovate with. Since I was the one pushing this transition forward, I decided to write a new tool in my favorite language, python. The fedpkg tool was born and took off. fedpkg was born around January 4th, 2010 and has since grown into 1,523 (via sloccount) lines of code. While far from complete, it is a great start (if I do say so myself!) at replacing the make system. Because it is written in python (or maybe just !Make) it seems to be easier to contribute to, and I've already gotten a number of contributions. More will come as it starts to be more widely used. The biggest challenge with fedpkg is removing the need to update something on the end user's system every single time we added a new Fedora release and changed what happens when you build for rawhide. I'll spare you the details but I'm fairly happy with what we have. The end result should be far fewer misfires and end user confusion.
The last major piece of the puzzle was how to actually convert the existing CVS repositories, including the fun pseudo branches, into git repositories. I tried a number of options over the years (I've been working on this off and on for nearly 4 years!) ranging from the built in git cvsimport to git-svn to parsecvs and a few others. In the end, we took a page from the gnome project and used parsecvs ( http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of our repositories. There were a few that gave parsecvs fits and recent versions of git cvsimport were able to handle them. The git system is fantastic enough that we were able to merge our pseudo cvs branches into actual git branches complete with a real shared history, but again I'll spare you the details of the scripting to do this. All but the kernel repo seems to have converted successfully which is a pretty good success rate in my book. We may yet get the kernel converted, but in the interest of time we opted to start fresh with dist-git for now.
Without the help of many others, this project would never have gotten done. Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way. Things will be bumpy over the next few weeks as we really start putting distgit to the test. No amount of staging and testing can really replace production use. There will be many more updates to fedpkg as bugs are found and fixed and features are contributed. Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).
When I do fedpkg clone -B libfoo, to get the old-style layout, all the branches are identical to devel. The branches on the standard clone seem fine. I've not made any changes with the -B layout, but I thought this should be corrected or deprecated.
fedora-packager-0.5.1.0-1.fc13.noarch
-J
Oh, and since I neglected to say so, thanks a huge heaping ton to Jesse and everyone else involved in making this happen, I've been excited to see it happen, and I have to say after using it today that the improvements in speed and usability are better than I expected. Kudos all around.
-J
Once again I want to thank everybody who helped out and for all the (continued) patience! I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues. Look for Oxf13 on freenode. Happy gitting!
Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAkxSTR0ACgkQ4v2HLvE71NXEswCeOAu+EG/porsvkjMVq+4lAchy VMgAn1DNwqyYcSSC3bwX/MQ/cfwx7qjs =fBGf -----END PGP SIGNATURE----- _______________________________________________ devel-announce mailing list devel-announce@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel-announce
Hi all, I do not know anything about GIT and I tried to follow this thread and the Using_Fedora_GIT in the wiki to make my first commit
The workflow was more or less the following: $ fedpkg clone zaz $ cd zaz/ $ cp -a ~/rpmbuild/SOURCES/zaz-0.8.0.tar.bz2 . $ cp -a ~/rpmbuild/SPECS/zaz.spec . $ fedpkg new-sources zaz-0.8.0.tar.bz2 $ git diff $ git config --global --add push.default tracking $ fedpkg clog $ fedpkg commit -F clog -p
I tried to use federa-packager available from updates. "fedpkg commit -F clog -p" did not succeed.
Then I installed the latest federa-packager from updates-testing and "fedpkg commit -F clog -p" now crashes. It even crashes with "fedpkg clog". The error is the same.
$ fedpkg clog Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in <module> args.command(args) File "/usr/bin/fedpkg", line 390, in clog mymodule = pyfedpkg.PackageModule(args.path) File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 744, in __init__ self.distval = self._findmasterbranch() File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 694, in _findmasterbranch return(int(fedoras[-1].strip('f')) + 1) IndexError: list index out of range
Help is really appreciated.
Thanks!
Andrea.
On 07/31/2010 03:19 PM, Andrea Musuruane wrote:
l return(int(fedoras[-1].strip('f')) + 1) IndexError: list index out of range
Help is really appreciated.
Thanks!
make sure you update fedora-packager to 0.5.1, remove and clone the repo again and follow the same steps. It will work.
Rahul
On Sat, Jul 31, 2010 at 11:50 AM, Rahul Sundaram metherid@gmail.com wrote:
On 07/31/2010 03:19 PM, Andrea Musuruane wrote:
l return(int(fedoras[-1].strip('f')) + 1) IndexError: list index out of range
Help is really appreciated.
Thanks!
make sure you update fedora-packager to 0.5.1, remove and clone the repo again and follow the same steps. It will work.
OK. I deleted the current repo and started again. I do have fedora-packager 0.5.1.
Now fedora-packager crashes when uploading new sources.
$ fedpkg clone zaz Cloning into zaz... [..] Resolving deltas: 100% (11/11), done. $ cd zaz/ $ cp -a ~/rpmbuild/SOURCES/zaz-0.8.0.tar.bz2 . $ cp -a ~/rpmbuild/SPECS/zaz.spec . $ fedpkg new-sources zaz-0.8.0.tar.bz2 Uploading: 6a63f986a80b4f4e1852ecf9871e9735 zaz-0.8.0.tar.bz2 Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in <module> args.command(args) File "/usr/bin/fedpkg", line 540, in new_sources mymodule.upload(args.files, replace=args.replace) File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 1278, in upload if lookaside.file_exists(self.module, file_basename, file_hash): File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 580, in file_exists curl.perform() pycurl.error: (35, 'SSL connect error') [andrea@panoramix zaz]$
Regards,
Andrea.
On Sat, Jul 31, 2010 at 11:56 AM, Andrea Musuruane musuruan@gmail.com wrote:
OK. I deleted the current repo and started again. I do have fedora-packager 0.5.1.
Now fedora-packager crashes when uploading new sources.
$ fedpkg clone zaz Cloning into zaz... [..] Resolving deltas: 100% (11/11), done. $ cd zaz/ $ cp -a ~/rpmbuild/SOURCES/zaz-0.8.0.tar.bz2 . $ cp -a ~/rpmbuild/SPECS/zaz.spec . $ fedpkg new-sources zaz-0.8.0.tar.bz2 Uploading: 6a63f986a80b4f4e1852ecf9871e9735 zaz-0.8.0.tar.bz2 Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in <module> args.command(args) File "/usr/bin/fedpkg", line 540, in new_sources mymodule.upload(args.files, replace=args.replace) File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 1278, in upload if lookaside.file_exists(self.module, file_basename, file_hash): File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 580, in file_exists curl.perform() pycurl.error: (35, 'SSL connect error') [andrea@panoramix zaz]$
I still have this error. Help is appreciated.
Thanks.
Andrea.
Am Sonntag, den 01.08.2010, 11:42 +0200 schrieb Andrea Musuruane:
On Sat, Jul 31, 2010 at 11:56 AM, Andrea Musuruane musuruan@gmail.com wrote:
OK. I deleted the current repo and started again. I do have fedora-packager 0.5.1.
Now fedora-packager crashes when uploading new sources.
$ fedpkg clone zaz Cloning into zaz... [..] Resolving deltas: 100% (11/11), done. $ cd zaz/ $ cp -a ~/rpmbuild/SOURCES/zaz-0.8.0.tar.bz2 . $ cp -a ~/rpmbuild/SPECS/zaz.spec . $ fedpkg new-sources zaz-0.8.0.tar.bz2 Uploading: 6a63f986a80b4f4e1852ecf9871e9735 zaz-0.8.0.tar.bz2 Traceback (most recent call last): File "/usr/bin/fedpkg", line 959, in <module> args.command(args) File "/usr/bin/fedpkg", line 540, in new_sources mymodule.upload(args.files, replace=args.replace) File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 1278, in upload if lookaside.file_exists(self.module, file_basename, file_hash): File "/usr/lib/python2.6/site-packages/pyfedpkg/__init__.py", line 580, in file_exists curl.perform() pycurl.error: (35, 'SSL connect error') [andrea@panoramix zaz]$
I still have this error. Help is appreciated.
I just yesterday had similar problems and then found out my ~/.fedora.cert was out of date since five days ago. Maybe it's the same for you, did you already had a look there? :)
Best Regards, Dominic
On Sun, Aug 1, 2010 at 1:05 PM, Dominic Hopf dmaphy@googlemail.com wrote:
I just yesterday had similar problems and then found out my ~/.fedora.cert was out of date since five days ago. Maybe it's the same for you, did you already had a look there? :)
Yes. It will be valid until Dec 7, 2010.
Bye,
Andrea.
On Sun, Aug 1, 2010 at 1:12 PM, Andrea Musuruane musuruan@gmail.com wrote:
On Sun, Aug 1, 2010 at 1:05 PM, Dominic Hopf dmaphy@googlemail.com wrote:
I just yesterday had similar problems and then found out my ~/.fedora.cert was out of date since five days ago. Maybe it's the same for you, did you already had a look there? :)
Yes. It will be valid until Dec 7, 2010.
It was the client side certificate. Although it was in its validity period it was no longer valid.
Regards,
Andrea.
Congratulations guys, this is awesome work. I know it has been painful and I have a rough idea about the level of effort involved. Looking over the transition, this is truly solid work, and I am confident it will carry (and exceed) the packager's needs for at least another 13 releases or so... :-)
Oh, yeah, and Jesse - thank you, for now people can finally stop mumbling blames my way. ;)
Cristian Gafton
On Thu, Jul 29, 2010 at 8:55 PM, Jesse Keating jkeating@redhat.com wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This evening we opened up dist-git for business. Dist-git is our git based replacement for dist-cvs, the source control system we were using for our package specs, patches, and source files. This has been a long time coming and a massive effort. I want to take a little time here to outline what we've done and where we are going.
First a brief outline of how our CVS system worked. CVS is a daemon of sorts, and all repos typically live within a single CVSROOT. Within this CVSROOT we had an 'avail' system to make use of, that we could populate with data from Fedora Account System and dump into this file. Avail worked on path names, relative to the CVSROOT. Since we used directories for each Fedora release as pseudo branches we could set avail info on each release "branch". CVS also used some filesystem symlink tricks to create a "common" subdir for every package module, and this is where we stuffed common scripts and Makefile content. Pretty clever on one hand, we can make updates to the make system without touching every single package, but it is pretty hackish and we had constant issues where somebody would attempt an action using old common content and stuff would fall over.
Now we look at git. Git is for the most part a daemonless system. Each repository is completely stand alone and generally does not require any other infrastructure to be useful. You can interact with a repository directly on the filesystem using /usr/bin/git or you can interact with it through say ssh, again using /usr/bin/git (your local /usr/bin/git will call a remote /usr/bin/git). Generally there is no running daemon to connect to and authenticate with. Basic authentication of who can check out what and commit where with git happens at the filesystem permissions level. One twist with this is that with git, we wanted to use real branches within a package repo to reflect the different Fedora/EPEL releases. In repo branches are not reflected with path names that we could set filesystem ACLs upon, so this posed a problem for our conversion.
Enter gitolite. Gitolite ( http://github.com/sitaramc/gitolite )is an addon system to git that provides ACL functionality including different rights for different branches within a given repository, and more! By using gitolite as a replacement for /usr/bin/git when a user connects to our git server we can again utilize the information we have within the Fedora Package Database and properly allow / restrict changes on specific branches for specific developers. The gitolite upstream ( Sitaram Chamarty, sitaram@atc.tcs.com ) has been fantastically responsive to our needs, which are admittedly a little unique. We have a very large set of repositories (over 10.5K) and a largish number of contributors (1050). The combination of the two leads to a very very large and complex ACL structure that at first broke the system quite badly. Upstream was very quick to create a "bigconfig" method of compiling the ACLs without crashing the box. Our other unique needs involve having individual accounts for each committer instead of a shared account with a large list of allowed SSH keys. Add to that some of our accounts need to be able to ssh shell into the git server for administrative duties. Throughout our trials and testing with gitolite every time we've ran into some issue that just didn't fit the mold, Sitaram has been there with a smile and a fix. At this point our production server is a whopping one line different from current gitolite upstream. This is a fantastic win for us, for our sustainability, and for the next large group that wants to make use of gitolite.
Once we had a plan for ACLs and for branches, we had to decide if we were going to replace the Make system and with what. I had never been a fan of Make, it was entirely too difficult to modify and innovate with. Since I was the one pushing this transition forward, I decided to write a new tool in my favorite language, python. The fedpkg tool was born and took off. fedpkg was born around January 4th, 2010 and has since grown into 1,523 (via sloccount) lines of code. While far from complete, it is a great start (if I do say so myself!) at replacing the make system. Because it is written in python (or maybe just !Make) it seems to be easier to contribute to, and I've already gotten a number of contributions. More will come as it starts to be more widely used. The biggest challenge with fedpkg is removing the need to update something on the end user's system every single time we added a new Fedora release and changed what happens when you build for rawhide. I'll spare you the details but I'm fairly happy with what we have. The end result should be far fewer misfires and end user confusion.
The last major piece of the puzzle was how to actually convert the existing CVS repositories, including the fun pseudo branches, into git repositories. I tried a number of options over the years (I've been working on this off and on for nearly 4 years!) ranging from the built in git cvsimport to git-svn to parsecvs and a few others. In the end, we took a page from the gnome project and used parsecvs ( http://cgit.freedesktop.org/~krh/parsecvs/ ) for the vast majority of our repositories. There were a few that gave parsecvs fits and recent versions of git cvsimport were able to handle them. The git system is fantastic enough that we were able to merge our pseudo cvs branches into actual git branches complete with a real shared history, but again I'll spare you the details of the scripting to do this. All but the kernel repo seems to have converted successfully which is a pretty good success rate in my book. We may yet get the kernel converted, but in the interest of time we opted to start fresh with dist-git for now.
Without the help of many others, this project would never have gotten done. Folks helped out with Koji modifications, with fedpkg contributions, with repeated testing of attempted conversions, with logic checking of my plans, of helping me understand more of git internals and deciphering error output, and most of all with being patient while we worked through the transition and very positive along the way. Things will be bumpy over the next few weeks as we really start putting distgit to the test. No amount of staging and testing can really replace production use. There will be many more updates to fedpkg as bugs are found and fixed and features are contributed. Wiki pages will get filled out as knowledge of how to interact with dist-git starts to spread ( https://fedoraproject.org/wiki/Using_Fedora_GIT is a good start ).
Once again I want to thank everybody who helped out and for all the (continued) patience! I'll be available via email and IRC as much as possible the next few days to help anybody with dist-git issues. Look for Oxf13 on freenode. Happy gitting!
Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAkxSTR0ACgkQ4v2HLvE71NXEswCeOAu+EG/porsvkjMVq+4lAchy VMgAn1DNwqyYcSSC3bwX/MQ/cfwx7qjs =fBGf -----END PGP SIGNATURE----- _______________________________________________ devel-announce mailing list devel-announce@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel-announce
On 08/05/2010 10:56 PM, Cristian Gafton wrote:
Congratulations guys, this is awesome work. I know it has been painful and I have a rough idea about the level of effort involved. Looking over the transition, this is truly solid work, and I am confident it will carry (and exceed) the packager's needs for at least another 13 releases or so... :-)
Oh, yeah, and Jesse - thank you, for now people can finally stop mumbling blames my way. ;)
No!!! That doesn't stop. The crazy Romanian is to blame for everything!
-
Adam Tkac -
Adam Williamson -
Andrea Musuruane -
Andreas Schwab -
Ben Boeckel -
Chen Lei -
Christof Damian -
Cristian Gafton -
Daniel J Walsh -
Dominic Hopf -
Doug Ledford -
Hans Ulrich Niedermann -
Harald Hoyer -
Horst H. von Brand -
Jef Spaleta -
Jesse Keating -
Jochen Schmitt -
Jon Ciesla -
Josh Stone -
Karel Zak -
Kevin Fenzi -
Konstantin Ryabitsev -
Martin Langhoff -
Martin Sourada -
Matej Cepl -
Matt McCutchen -
Michal Hlavinka -
Mike McGrath -
Miroslav Suchý -
Nikola Pajkovsky -
Peter Lemenkov -
Peter Robinson -
Phil Knirsch -
Rahul Sundaram -
Ralf Ertzinger -
Scott Schmit -
Stanislav Ochotnicky -
Stephen John Smoogen -
Till Maas -
Todd Zullinger -
Tomas Mraz