Dear all,
This morning Mathieu (bochecha) and I spent some time trying to get gitolite3 work on our pkgs01.stg.
This is an overview of the situation.
By design gitolite is meant to be installed with a dedicated user. Everyone uses that user. gitolite then maps the public ssh key to its user database and its configuration file to handle who is allowed to do what on which git repo.
Our setup is quite different from this. We have a single configuration file at /etc/gitolite/conf/gitolite.conf that contains all the information about who is allowed to do what on which git repos. In addition we have a single /etc/gitolite/gitolite.rc configuration file that specifies to gitolite where are the git repositories located.
Each user has then an account on the machine but by default no shell access. The access is restricted via ssh with in ~/.ssh/authorized_keys the instructions: `command="/usr/bin/gl-auth-command"` (people with shell access have a different command).
That commands combines the info from /etc/gitolite/gitolite.rc and /etc/gitolite/conf/gitolite.conf and allow or deny the access/action.
Now gitolite3 has changed quite a bit compared to our setup. Basically, gitolite3 expects the gitolite.rc file to be at ~/.gitolite.rc and the repositories to be at ~/repositories. Having symlink to these locations are fine, but they should be there.
What we can do also is play on the $HOME variable.
In pkgs01.stg we kept the configurations files (gitolite.conf and gitolite.rc) in /etc/gitolite as before, but we created a couple of symlink in /srv/git/ .gitolite -> /etc/gitolite/ repositories -> /srv/git/rpms/ The idea is to be able to set HOME=/srv/git and run the gitolite command.
This way we managed to get the genacls.sh script working. It is the script that generates the gitolite.conf file based on the pkgdb information and compile that configuration file so that gitolite can use it.
gitolite3 also changes the command in the ~/.ssh/authorized_keys file to: command="/usr/share/gitolite3/gitolite-shell user" That command is set in the sources in `src/triggers/post-compile/ssh-authkeys` and put in the file by (I believe) `gitolite compile` that is ran by genacls.sh.
In order to test, I (manually) changed Mathieu's authorized_keys to look like: command="HOME=/srv/git/ /usr/share/gitolite3/gitolite-shell user"
We then tried to get Mathieu to clone a repo and we found out: - the ~/.gitolite folder, that gitolite3 requires, needs to have a `logs/` subfolder, and that folder needs to be accessible read and write to everyone so `chmod 777 -R /srv/git/.gitolite/logs` (ie: chmod 777 /etc/gitolite/logs since it's a symlink) - the /etc/gitolite/conf/gitolite.conf-compiled.pm needs to be readable by everyone since it's the file gitolite uses to see who can do what on which repo so `chmod o+r conf/gitolite.conf-compiled.pm - Mathieu was finally able to clone and push in a repo he has access to and could not on the kernel repo (as supposed to since he does not have commit there), but the error returned is just plain ugly
So to conclude: - we managed to get it to work - its needs chmod 777 on /srv/git/.gitolite/logs - its needs chmod o+r on conf/gitolite.conf-compiled.pm (and that after each time the file is generated) - we need to find a way to fix the command in the user's ~/.ssh/authorized_keys so that they specify the HOME variable to use when running the gitolite command -> Might very well require to patch gitolite itself for this (and thus keep the patch in our package ad vitam aeternam)
That being said, I believe our options are: 1) Talk with upstream, in the past I believe he was quite reactive and willing to help us. We are the largest public deployment of gitolite maybe he'll still be willing to help us to discuss: - setting HOME in the authorized_keys - writing logs - accessing gitolite.conf-compiled.pm 2) Stick with gitolite2 -> We need to find out how easy it is to maintain it on epel7 and we need someone willing to maintain it (knowing that it's perl) -> Upstream said gitolite2 is still maintain wrt security fixes but I guess only for some time 3) Move to the model gitolite relies on (1 specific gitolite user everyone accesses the repo through). + No more need to create account on the machine for each and every packager + Inline with gitolite's design - Breaking all existing git clone around (as the push/pull urls change) --> URLs change could be handled at the fedpkg level + We are already forcing an update of fedpkg to our packager with the change from md5 to sha for the lookaside cache, so we could couple both change in the same update and preventing forcing two fedpkg updates. - Huge number of keys to handle for that user ? Performance impact of this number of users? ? Dennis mentionned logging at the meeting yesterday, but it looks like gitolite does provide logging itself 4) Move away from gitolite. I did not check what else would be available/suitable for us, so no ideas there. Suggestions welcome I guess :)
I think that covers most if not all our findings.
Questions and suggestions welcome :)
Pierre and Mathieu
On 09/05/2014 12:45 PM, Pierre-Yves Chibon wrote:
? Performance impact of this number of users?
Performance drops by 17% http://miroslav.suchy.cz/blog/archives/2014/09/05/how_the_size_of_authorized...
On 09/05/2014 02:39 PM, Miroslav Suchý wrote:
Performance drops by 17% http://miroslav.suchy.cz/blog/archives/2014/09/05/how_the_size_of_authorized...
And to emphasis: This is worst case scenario for those users at the end of file. So if we average all Fedora user the average drop will be just half - 8.5%. And this is just drop in authorization. It does not affect transfers etc.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Fri, 5 Sep 2014 12:45:38 +0200 Pierre-Yves Chibon pingou@pingoured.fr wrote:
Dear all,
This morning Mathieu (bochecha) and I spent some time trying to get gitolite3 work on our pkgs01.stg.
This is an overview of the situation.
By design gitolite is meant to be installed with a dedicated user. Everyone uses that user. gitolite then maps the public ssh key to its user database and its configuration file to handle who is allowed to do what on which git repo.
Our setup is quite different from this. We have a single configuration file at /etc/gitolite/conf/gitolite.conf that contains all the information about who is allowed to do what on which git repos. In addition we have a single /etc/gitolite/gitolite.rc configuration file that specifies to gitolite where are the git repositories located.
Each user has then an account on the machine but by default no shell access. The access is restricted via ssh with in ~/.ssh/authorized_keys the instructions: `command="/usr/bin/gl-auth-command"` (people with shell access have a different command).
That commands combines the info from /etc/gitolite/gitolite.rc and /etc/gitolite/conf/gitolite.conf and allow or deny the access/action.
Now gitolite3 has changed quite a bit compared to our setup. Basically, gitolite3 expects the gitolite.rc file to be at ~/.gitolite.rc and the repositories to be at ~/repositories. Having symlink to these locations are fine, but they should be there.
What we can do also is play on the $HOME variable.
In pkgs01.stg we kept the configurations files (gitolite.conf and gitolite.rc) in /etc/gitolite as before, but we created a couple of symlink in /srv/git/ .gitolite -> /etc/gitolite/ repositories -> /srv/git/rpms/ The idea is to be able to set HOME=/srv/git and run the gitolite command.
This way we managed to get the genacls.sh script working. It is the script that generates the gitolite.conf file based on the pkgdb information and compile that configuration file so that gitolite can use it.
gitolite3 also changes the command in the ~/.ssh/authorized_keys file to: command="/usr/share/gitolite3/gitolite-shell user" That command is set in the sources in `src/triggers/post-compile/ssh-authkeys` and put in the file by (I believe) `gitolite compile` that is ran by genacls.sh.
In order to test, I (manually) changed Mathieu's authorized_keys to look like: command="HOME=/srv/git/ /usr/share/gitolite3/gitolite-shell user"
We then tried to get Mathieu to clone a repo and we found out:
- the ~/.gitolite folder, that gitolite3 requires, needs to have a
`logs/` subfolder, and that folder needs to be accessible read and write to everyone so `chmod 777 -R /srv/git/.gitolite/logs` (ie: chmod 777 /etc/gitolite/logs since it's a symlink)
- the /etc/gitolite/conf/gitolite.conf-compiled.pm needs to be
readable by everyone since it's the file gitolite uses to see who can do what on which repo so `chmod o+r conf/gitolite.conf-compiled.pm
- Mathieu was finally able to clone and push in a repo he has access
to and could not on the kernel repo (as supposed to since he does not have commit there), but the error returned is just plain ugly
So to conclude:
- we managed to get it to work
- its needs chmod 777 on /srv/git/.gitolite/logs
- its needs chmod o+r on conf/gitolite.conf-compiled.pm (and that
after each time the file is generated)
- we need to find a way to fix the command in the user's
~/.ssh/authorized_keys so that they specify the HOME variable to use when running the gitolite command -> Might very well require to patch gitolite itself for this (and thus keep the patch in our package ad vitam aeternam)
That being said, I believe our options are:
- Talk with upstream, in the past I believe he was quite reactive
and willing to help us. We are the largest public deployment of gitolite maybe he'll still be willing to help us to discuss: - setting HOME in the authorized_keys - writing logs - accessing gitolite.conf-compiled.pm 2) Stick with gitolite2 -> We need to find out how easy it is to maintain it on epel7 and we need someone willing to maintain it (knowing that it's perl) -> Upstream said gitolite2 is still maintain wrt security fixes but I guess only for some time 3) Move to the model gitolite relies on (1 specific gitolite user everyone accesses the repo through).
- No more need to create account on the machine for each and every
packager
- Inline with gitolite's design
- Breaking all existing git clone around (as the push/pull urls
change) --> URLs change could be handled at the fedpkg level
- We are already forcing an update of fedpkg to our packager with
the change from md5 to sha for the lookaside cache, so we could couple both change in the same update and preventing forcing two fedpkg updates.
- Huge number of keys to handle for that user
? Performance impact of this number of users? ? Dennis mentionned logging at the meeting yesterday, but it looks like gitolite does provide logging itself 4) Move away from gitolite. I did not check what else would be available/suitable for us, so no ideas there. Suggestions welcome I guess :)
I think that covers most if not all our findings.
Questions and suggestions welcome :)
I think we should do number 1, followed by 4, then 2 and finally 3 if we have no other options. we have logging currently in /var/log/secure I do not know how upstreams logging works. when we setup dist-git we worked closely with upstream to get our features in. but we have not really be active with them since then. We have broken everyone's clones and required manual intervention in the past. I really would rather we do not do it again.
I would like to better understand why upstream did what they did.
Dennis
On Fri, Sep 05, 2014 at 12:45:38PM +0200, Pierre-Yves Chibon wrote:
That being said, I believe our options are:
- Talk with upstream, in the past I believe he was quite reactive and willing to help us. We are the largest public deployment of gitolite maybe he'll still be willing to help us to discuss:
- setting HOME in the authorized_keys
- writing logs
- accessing gitolite.conf-compiled.pm
Yesterday Mathieu had a great idea: all the packagers are the same group.
So we gave it a try. Give group ownership and write permission on the logs as well as group ownership and read permission on gitolite.conf-compiled.pm worked. And worked w/o requiring the 777 we tried before. So that's already a step forward.
However, the HOME question remains. Mathieu is going to contact upstream to see if we can manage to automate adding a HOME somehow.
Something new I found is that the user gen-acls that is the user running the pkgdb_sync_git_branches.py returns when run: Permission denied: '/etc/pki/fedmsg/shell-pkgs01.stg.phx2.fedoraproject.org.key
But I guess this has nothing to do with gitolite, just something to figure out down the line.
Pierre
On Tue, 2014-09-09 at 18:58 +0200, Pierre-Yves Chibon wrote:
On Fri, Sep 05, 2014 at 12:45:38PM +0200, Pierre-Yves Chibon wrote:
That being said, I believe our options are:
- Talk with upstream, in the past I believe he was quite reactive and willing to help us. We are the largest public deployment of gitolite maybe he'll still be willing to help us to discuss:
- setting HOME in the authorized_keys
- writing logs
- accessing gitolite.conf-compiled.pm
[... snip ...]
However, the HOME question remains. Mathieu is going to contact upstream to see if we can manage to automate adding a HOME somehow.
As it turns out, we figured it out: we don't use the upstream tools to generate the ~/.ssh/authorized_keys files, we have our own stuff in ansible: inventory/group_vars/pkgs-stg
We fixed this, and a couple of other things.
As of commit 13465f6a0d2ed107357b204ab9bf5ca6a4f1fef2, distgit staging is now running with Gitolite 3 on RHEL 7.
Obviously, we are now going to migrate production to it, right in the middle of the Alpha freeze. :P
More seriously, if anyone wants to try it out, here's an easy way. Save the attached file into /etc/rpkg/fedpkg-stg.conf, then:
$ alias fedpkg-stg="fedpkg --config=/etc/rpkg/fedpkg-stg.conf"
Then use the fedpkg-stg command the way you'd use fedpkg (try cloning, pushing, ...)
Things seem to work fine so far, but we need more testing, of course.
On Fri, Sep 05, 2014 at 12:45:38PM +0200, Pierre-Yves Chibon wrote:
Dear all,
This morning Mathieu (bochecha) and I spent some time trying to get gitolite3 work on our pkgs01.stg.
This is an overview of the situation.
By design gitolite is meant to be installed with a dedicated user. Everyone uses that user. gitolite then maps the public ssh key to its user database and its configuration file to handle who is allowed to do what on which git repo.
Our setup is quite different from this. We have a single configuration file at /etc/gitolite/conf/gitolite.conf that contains all the information about who is allowed to do what on which git repos. In addition we have a single /etc/gitolite/gitolite.rc configuration file that specifies to gitolite where are the git repositories located.
Each user has then an account on the machine but by default no shell access. The access is restricted via ssh with in ~/.ssh/authorized_keys the instructions: `command="/usr/bin/gl-auth-command"` (people with shell access have a different command).
That commands combines the info from /etc/gitolite/gitolite.rc and /etc/gitolite/conf/gitolite.conf and allow or deny the access/action.
Now gitolite3 has changed quite a bit compared to our setup. Basically, gitolite3 expects the gitolite.rc file to be at ~/.gitolite.rc and the repositories to be at ~/repositories. Having symlink to these locations are fine, but they should be there.
What we can do also is play on the $HOME variable.
In pkgs01.stg we kept the configurations files (gitolite.conf and gitolite.rc) in /etc/gitolite as before, but we created a couple of symlink in /srv/git/ .gitolite -> /etc/gitolite/ repositories -> /srv/git/rpms/ The idea is to be able to set HOME=/srv/git and run the gitolite command.
So Mathieu and I started to work on this again over the last couple of days.
We fix the logging issue by simply asking gitolite to rely on syslog. This means all logs from gitolite are now ending up in /var/log/messages (this has some pros and some cons)
We updated the configuration file gitolite.rc to the (completely) new one of gitolite3.
At that point we were able to clone and pull but pushing didn't work. Finally, Mathieu realized that our setup in ~/.ssh/authorized_keys was incorrect. With Gitolite2 we could specify the type of the user (user vs admin) while on gitolite3 it requires the username of the user. So we adjusted fasClient and the fas.conf configuration file to allow us to set the username in the authorized_keys file
Pull-request at: https://github.com/fedora-infra/fas/pull/97
So now, pkgs01.stg should work. We should be able to clone, push, pull changes. Email notifications are working as well, so beware which package you test against :)
Please take a stab at it and let us know if you see any problem with it. Once fasClient is updated, we should rebuild the box and do some final testing before we can consider moving this to prod.
Thanks, Pierre
On Tue, Dec 16, 2014 at 04:13:07PM +0100, Pierre-Yves Chibon wrote:
On Fri, Sep 05, 2014 at 12:45:38PM +0200, Pierre-Yves Chibon wrote:
Dear all,
This morning Mathieu (bochecha) and I spent some time trying to get gitolite3 work on our pkgs01.stg.
This is an overview of the situation.
By design gitolite is meant to be installed with a dedicated user. Everyone uses that user. gitolite then maps the public ssh key to its user database and its configuration file to handle who is allowed to do what on which git repo.
Our setup is quite different from this. We have a single configuration file at /etc/gitolite/conf/gitolite.conf that contains all the information about who is allowed to do what on which git repos. In addition we have a single /etc/gitolite/gitolite.rc configuration file that specifies to gitolite where are the git repositories located.
Each user has then an account on the machine but by default no shell access. The access is restricted via ssh with in ~/.ssh/authorized_keys the instructions: `command="/usr/bin/gl-auth-command"` (people with shell access have a different command).
That commands combines the info from /etc/gitolite/gitolite.rc and /etc/gitolite/conf/gitolite.conf and allow or deny the access/action.
Now gitolite3 has changed quite a bit compared to our setup. Basically, gitolite3 expects the gitolite.rc file to be at ~/.gitolite.rc and the repositories to be at ~/repositories. Having symlink to these locations are fine, but they should be there.
What we can do also is play on the $HOME variable.
In pkgs01.stg we kept the configurations files (gitolite.conf and gitolite.rc) in /etc/gitolite as before, but we created a couple of symlink in /srv/git/ .gitolite -> /etc/gitolite/ repositories -> /srv/git/rpms/ The idea is to be able to set HOME=/srv/git and run the gitolite command.
So Mathieu and I started to work on this again over the last couple of days.
We fix the logging issue by simply asking gitolite to rely on syslog. This means all logs from gitolite are now ending up in /var/log/messages (this has some pros and some cons)
We updated the configuration file gitolite.rc to the (completely) new one of gitolite3.
At that point we were able to clone and pull but pushing didn't work. Finally, Mathieu realized that our setup in ~/.ssh/authorized_keys was incorrect. With Gitolite2 we could specify the type of the user (user vs admin) while on gitolite3 it requires the username of the user. So we adjusted fasClient and the fas.conf configuration file to allow us to set the username in the authorized_keys file
Pull-request at: https://github.com/fedora-infra/fas/pull/97
So now, pkgs01.stg should work. We should be able to clone, push, pull changes. Email notifications are working as well, so beware which package you test against :)
Please take a stab at it and let us know if you see any problem with it. Once fasClient is updated, we should rebuild the box and do some final testing before we can consider moving this to prod.
For the known issue list:
Currently, the people that have shell access to pkgs01.stg, we currently cannot do a simple ssh pkgs01.stg...
Hopefully, we'll be able to fix this.
Pierre
"PC" == Pierre-Yves Chibon pingou@pingoured.fr writes:
PC> Currently, the people that have shell access to pkgs01.stg, we PC> currently cannot do a simple ssh pkgs01.stg...
Ouch. I guess everything is going over the console now?
PC> Hopefully, we'll be able to fix this.
sshd listening on a different port, if nothing else works?
I think the new package setup process still requires us to ssh in, so....
- J<
On Tue, 2014-12-16 at 17:31 -0600, Jason L Tibbitts III wrote:
"PC" == Pierre-Yves Chibon pingou@pingoured.fr writes:
PC> Currently, the people that have shell access to pkgs01.stg, we PC> currently cannot do a simple ssh pkgs01.stg...
Ouch. I guess everything is going over the console now?
The console? Basically, what I see at the moment is that gitolite takes all the incoming connections. So no shell, even for admins :(
PC> Hopefully, we'll be able to fix this.
sshd listening on a different port, if nothing else works?
That won't work since we are using ssh to connect to gitolite, we clone/pull/push over ssh.
I think the new package setup process still requires us to ssh in, so....
Well, maybe not. The idea is that git repos and git branches are created by the mkbranch script via a playbook triggered by pkgdb messages on fedmsg. Do we need access to the box for other things? (Knowing that sysadmin-main and maybe sysadmin-noc still have access via lockbox if needed)
Pierre
On Wed, 17 Dec 2014 08:55:54 +0100 Pierre-Yves Chibon pingou@pingoured.fr wrote:
On Tue, 2014-12-16 at 17:31 -0600, Jason L Tibbitts III wrote:
> "PC" == Pierre-Yves Chibon pingou@pingoured.fr writes:
PC> Currently, the people that have shell access to pkgs01.stg, we PC> currently cannot do a simple ssh pkgs01.stg...
Ouch. I guess everything is going over the console now?
The console? Basically, what I see at the moment is that gitolite takes all the incoming connections. So no shell, even for admins :(
PC> Hopefully, we'll be able to fix this.
sshd listening on a different port, if nothing else works?
That won't work since we are using ssh to connect to gitolite, we clone/pull/push over ssh.
I think the new package setup process still requires us to ssh in, so....
Well, maybe not. The idea is that git repos and git branches are created by the mkbranch script via a playbook triggered by pkgdb messages on fedmsg. Do we need access to the box for other things? (Knowing that sysadmin-main and maybe sysadmin-noc still have access via lockbox if needed)
I can definitely see cases we might need access to the machine: Checking logs, looking to see if acls were right, etc.
So, IMHO it would be nice to solve this if we can...
kevin
On Tue, Dec 16, 2014 at 04:13:07PM +0100, Pierre-Yves Chibon wrote:
So now, pkgs01.stg should work. We should be able to clone, push, pull changes. Email notifications are working as well, so beware which package you test against :)
Awesome work, thank you!
Please take a stab at it and let us know if you see any problem with it. Once fasClient is updated, we should rebuild the box and do some final testing before we can consider moving this to prod.
What's up with cgit there? No packages listed: http://pkgs.stg.fedoraproject.org/cgit/
Is it related?
infrastructure@lists.fedoraproject.org
-
Dennis Gilmore -
Jason L Tibbitts III -
Kevin Fenzi -
Mathieu Bridon -
Miroslav Suchý -
Pierre-Yves Chibon -
Ralph Bean