Random thoughts/crazy idea: Drop SSL certs

[Pierre-Yves Chibon]

Good morning everyone,

This week-end I had a random thought, which I quickly discussed with Dennis on
IRC on Sunday but that I thought might be interesting to discuss in a wider
audience.

The initial thought came from a text that Dennis wrote:
"""
Releng tracks this data in 2 systems, 1 of which we own: Koji and Bodhi. Koji
uses ssl certs tied to FAS and bodhi uses FAS for authentication to provide a
strong relationship between a user and the content
"""
Source: https://fedoraproject.org/wiki/ReleaseEngineering/Philosophy#Auditable

This has lead me to the question: Is this all what SSL certs are bringing us?

The following only works under the assumption that it is.
So SSL certs are basically a certain type of API token. Everyone has one,
specific to koji and the lookaside cache, time limited and gives us a way of
doing authentication and authorization server side.

So on this they behave just like any other API token, but using SSL certs has
some pros and downs:

pros:
  - Easy to find out when the token expires
  - In place and working
  - Known to the current process maintainers
 
cons:
  - One per client
  - Hard to invalidate iiuc (ie: if someone's machine is compromised/lost it is
    hard to make this user's certificate invalide)
  - Relise on the SSL pile
    - master certificate
    - self-signed vs signed by an authority
    - complex tooling
  - Makes us maintain a whole infrastructure stack around this (cf the dogtag
    discussion)
  - Relies on the master certificate that expires every X years making everyone
    generate a new client certificate

On the otherside, recently we have been more and more feeling the need for a
centralized API authentication place. Something along the line of a personalized
0Auth. This has also pros and cons.

pros
  - API token per user and per application
  - Could support multiple tokens per application
  - Central place to manage API token (ie a central place to revoke someone's
    access if a machine gets compromised/lost)
  - Simpler than dealing with the SSL stack
  - Can be re-used by multiple applications

cons:
  - It's an idea and it needs work :)
  - Impacts
    - dist-git
    - koji
    - ?

I do realize that this would be a pretty big task to undertake but we are
currently at a stage where we are planning for the future, including for the
next generation of koji but also FAS3, dogtag, our master cert expiring in a
couple of years...

So I thought I would leave this here as food for thoughts and I'm happy to
discuss pros and cons of this idea.

Thanks,
Pierre

[Colin Walters]

On Mon, Apr 27, 2015, at 09:45 AM, Pierre-Yves Chibon wrote:
>  This has lead me to the question: Is this all what SSL certs are bringing us? 
I think the ability to do a commit should be thought of as equivalent to the ability
to do a build - because anyone who can commit something to a package will
cause it to be implicitly included in the build that another person does

That implies build access should be gated by SSH key, not by API token or
SSL certificate.   (Or alternatively the commit authentication method changed
to match whatever is chosen for build)

[Miroslav Suchý]

On 04/27/2015 03:45 PM, Pierre-Yves Chibon wrote:
>  pros
>    - API token per user and per application
>    - Could support multiple tokens per application
>    - Central place to manage API token (ie a central place to revoke someone's
>      access if a machine gets compromised/lost)
>    - Simpler than dealing with the SSL stack
>    - Can be re-used by multiple applications
>  
>  cons:
>    - It's an idea and it needs work :)
>    - Impacts
>      - dist-git
>      - koji
>      - ? 
The fact, that SSL certs are used for identifying user, always seemed weird to me.
And sometimes it is painful to use it. It is definitely easier to change token than change
a ssl cert.

+1 to use normal SSL cert just for crypto and identify user using token (or even
kerberos/GSSAPI).

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Peter Robinson]

>  This week-end I had a random thought, which I quickly discussed with
> Dennis on
>  IRC on Sunday but that I thought might be interesting to discuss in a wider
>  audience.
>
>  The initial thought came from a text that Dennis wrote:
>  """
>  Releng tracks this data in 2 systems, 1 of which we own: Koji and Bodhi. Koji
>  uses ssl certs tied to FAS and bodhi uses FAS for authentication to provide a
>  strong relationship between a user and the content
>  """
>  Source: https://fedoraproject.org/wiki/ReleaseEngineering/Philosophy#Auditable
>
>  This has lead me to the question: Is this all what SSL certs are bringing us?
>
>  The following only works under the assumption that it is.
>  So SSL certs are basically a certain type of API token. Everyone has one,
>  specific to koji and the lookaside cache, time limited and gives us a way of
>  doing authentication and authorization server side. 
Are certs used for the transport interface between sigul and koji?

Peter

[Till Maas]

On Mon, Apr 27, 2015 at 03:45:00PM +0200, Pierre-Yves Chibon wrote:

>  The following only works under the assumption that it is.
>  So SSL certs are basically a certain type of API token. Everyone has one,
>  specific to koji and the lookaside cache, time limited and gives us a way of
>  doing authentication and authorization server side.
>  
>  So on this they behave just like any other API token, but using SSL certs has
>  some pros and downs: 
It is not an API token in the sense that the key is not transfered via
TLS, so if there are bugs in TLS like Poodle, they do not expose the
token.

>  cons:
>    - One per client
>    - Hard to invalidate iiuc (ie: if someone's machine is compromised/lost it is
>      hard to make this user's certificate invalide) 
This is not the case currently afaik. If one creates a new certificate,
the old one is revoked and our systems honor this.

>    - Relies on the master certificate that expires every X years
> making everyone
>      generate a new client certificate 
The expiration of the master certificate does not require the client
certificate to be regenerated as long as the same master key is used.
Also it is easily possible to move to a new master CA while keeping the
old CA valid for a migration time (e.g. the duration of normal client
certificates).

>  On the otherside, recently we have been more and more feeling the
> need for a
>  centralized API authentication place. Something along the line of a personalized
>  0Auth. This has also pros and cons.
>  
>  pros
>    - API token per user and per application 
This is something I would like very much, but also with a fine-grained
permissions system. E.g. allowing to create a token that can only be
used to retire pkgs in pkgdb could be used to automate retiring pkgs
without using credentials that can also a everything else.

Regards
Till

[Pierre-Yves Chibon]

On Mon, Apr 27, 2015 at 05:59:14PM +0200, Till Maas wrote:
>  On Mon, Apr 27, 2015 at 03:45:00PM +0200, Pierre-Yves Chibon wrote:
>  
>  > The following only works under the assumption that it is.
>  > So SSL certs are basically a certain type of API token. Everyone has one,
>  > specific to koji and the lookaside cache, time limited and gives us a way of
>  > doing authentication and authorization server side.
>  > 
>  > So on this they behave just like any other API token, but using SSL certs has
>  > some pros and downs:
>  
>  It is not an API token in the sense that the key is not transfered via
>  TLS, so if there are bugs in TLS like Poodle, they do not expose the
>  token.
>  
>  > cons:
>  >   - One per client
>  >   - Hard to invalidate iiuc (ie: if someone's machine is compromised/lost it is
>  >     hard to make this user's certificate invalide)
>  
>  This is not the case currently afaik. If one creates a new certificate,
>  the old one is revoked and our systems honor this. 
Yes but can an admin easily do this ?
Say, I'm traveling, my laptop is stolen, I do not have access to the internet
but I can text X informing of the theft. Can an admin easily invalidate my
client cert?

Don't get me wrong, I'm happy if this is the case, but I am under the impression
it isn't :)

>  >   - Relies on the master certificate that expires every X years
> making everyone
>  >     generate a new client certificate
>  
>  The expiration of the master certificate does not require the client
>  certificate to be regenerated as long as the same master key is used.
>  Also it is easily possible to move to a new master CA while keeping the
>  old CA valid for a migration time (e.g. the duration of normal client
>  certificates). 
Hm, I guess this just exposes my ignorance of the SSL workflow, but from a
discussion with Dennis, I seem to have understood that changing the master
certificate would imply asking everyone to get a new client certificate.
Could this be because koji only accepts one master cert? (Or do I just wrongly
recall? :))

>  > On the otherside, recently we have been more and more feeling
> the need for a
>  > centralized API authentication place. Something along the line of a personalized
>  > 0Auth. This has also pros and cons.
>  > 
>  > pros
>  >   - API token per user and per application
>  
>  This is something I would like very much, but also with a fine-grained
>  permissions system. E.g. allowing to create a token that can only be
>  used to retire pkgs in pkgdb could be used to automate retiring pkgs
>  without using credentials that can also a everything else. 
This is really something that would be cool to get :)


Pierre

[Dennis Gilmore]

On Monday, April 27, 2015 03:45:00 PM Pierre-Yves Chibon wrote:
>  Good morning everyone,
>  
>  This week-end I had a random thought, which I quickly discussed with Dennis
>  on IRC on Sunday but that I thought might be interesting to discuss in a
>  wider audience.
>  
>  The initial thought came from a text that Dennis wrote:
>  """
>  Releng tracks this data in 2 systems, 1 of which we own: Koji and Bodhi.
>  Koji uses ssl certs tied to FAS and bodhi uses FAS for authentication to
>  provide a strong relationship between a user and the content
>  """
>  Source:
>  https://fedoraproject.org/wiki/ReleaseEngineering/Philosophy#Auditable
>  
>  This has lead me to the question: Is this all what SSL certs are bringing
>  us? 
It does a two way authentication/authorisation. apache on the server side 
validates that the cert is signed by our CA and not revoked. while on the 
client side koji at least. I would need to double check that fedpkg does for 
lookaside cache, verifies that that server cert is signed by the appropriate 
CA and is not revoked also. 

Dennis

[Dennis Gilmore]

On Monday, April 27, 2015 04:44:37 PM Peter Robinson wrote:
>  > This week-end I had a random thought, which I quickly discussed
> with
>  > Dennis on IRC on Sunday but that I thought might be interesting to
>  > discuss in a wider audience.
>  > 
>  > The initial thought came from a text that Dennis wrote:
>  > """
>  > Releng tracks this data in 2 systems, 1 of which we own: Koji and Bodhi.
>  > Koji uses ssl certs tied to FAS and bodhi uses FAS for authentication to
>  > provide a strong relationship between a user and the content
>  > """
>  > Source:
>  > https://fedoraproject.org/wiki/ReleaseEngineering/Philosophy#Auditable
>  > 
>  > This has lead me to the question: Is this all what SSL certs are bringing
>  > us?
>  > 
>  > The following only works under the assumption that it is.
>  > So SSL certs are basically a certain type of API token. Everyone has one,
>  > specific to koji and the lookaside cache, time limited and gives us a way
>  > of doing authentication and authorization server side.
>  
>  Are certs used for the transport interface between sigul and koji? 
yes they are, there is a few certs in use in sigul and it uses them to make 
encrypted tunnels inside of encrypted tunnels for sigul's core functionality, 
additionally the bridge cert is used to upload teh signed rpm artifacts to 
koji.  

Dennis

[Dennis Gilmore]

On Monday, April 27, 2015 06:23:38 PM Pierre-Yves Chibon wrote:
>  On Mon, Apr 27, 2015 at 05:59:14PM +0200, Till Maas wrote:
>  > On Mon, Apr 27, 2015 at 03:45:00PM +0200, Pierre-Yves Chibon wrote:
>  > > The following only works under the assumption that it is.
>  > > So SSL certs are basically a certain type of API token. Everyone has
>  > > one,
>  > > specific to koji and the lookaside cache, time limited and gives us a
>  > > way of doing authentication and authorization server side.
>  > > 
>  > > So on this they behave just like any other API token, but using SSL
>  > > certs has> 
>  > > some pros and downs:
>  > It is not an API token in the sense that the key is not transfered via
>  > TLS, so if there are bugs in TLS like Poodle, they do not expose the
>  > token.
>  > 
>  > > cons:
>  > >   - One per client
>  > >   - Hard to invalidate iiuc (ie: if someone's machine is
>  > >   compromised/lost it is> >   
>  > >     hard to make this user's certificate invalide)
>  > 
>  > This is not the case currently afaik. If one creates a new certificate,
>  > the old one is revoked and our systems honor this.
>  
>  Yes but can an admin easily do this ?
>  Say, I'm traveling, my laptop is stolen, I do not have access to the
>  internet but I can text X informing of the theft. Can an admin easily
>  invalidate my client cert? 
you would need to get on fas01 today and manually revoke it. so yes it is 
doable. 

>  Don't get me wrong, I'm happy if this is the case, but I am
> under the
>  impression it isn't :)
>  
>  > >   - Relies on the master certificate that expires every X years making
>  > >   everyone> >   
>  > >     generate a new client certificate
>  > 
>  > The expiration of the master certificate does not require the client
>  > certificate to be regenerated as long as the same master key is used.
>  > Also it is easily possible to move to a new master CA while keeping the
>  > old CA valid for a migration time (e.g. the duration of normal client
>  > certificates).
>  
>  Hm, I guess this just exposes my ignorance of the SSL workflow, but from a
>  discussion with Dennis, I seem to have understood that changing the master
>  certificate would imply asking everyone to get a new client certificate.
>  Could this be because koji only accepts one master cert? (Or do I just
>  wrongly recall? :)) 
That is how I was proposing a move to dogtag or some other new robust system, 
as dogtag from previous conversations can not easily reuse the existing CA. 
but I may missunderstand that or things may have changed since I last talked 
to the dogtag folks about it.

>  > > On the otherside, recently we have been more and more
> feeling the need
>  > > for a centralized API authentication place. Something along the line of
>  > > a personalized 0Auth. This has also pros and cons.
>  > > 
>  > > pros
>  > > 
>  > >   - API token per user and per application
>  > 
>  > This is something I would like very much, but also with a fine-grained
>  > permissions system. E.g. allowing to create a token that can only be
>  > used to retire pkgs in pkgdb could be used to automate retiring pkgs
>  > without using credentials that can also a everything else.
>  
>  This is really something that would be cool to get :) 
This is not something that
can really be done with certs etc. it would require 
a fundamental change in how all the tools deal with permissions.

Dennis

[Peter Robinson]

On Mon, Apr 27, 2015 at 5:43 PM, Dennis Gilmore <dennis(a)ausil.us&gt; wrote:
>  On Monday, April 27, 2015 03:45:00 PM Pierre-Yves Chibon wrote:
> > Good morning everyone,
> >
> > This week-end I had a random thought, which I quickly discussed with Dennis
> > on IRC on Sunday but that I thought might be interesting to discuss in a
> > wider audience.
> >
> > The initial thought came from a text that Dennis wrote:
> > """
> > Releng tracks this data in 2 systems, 1 of which we own: Koji and Bodhi.
> > Koji uses ssl certs tied to FAS and bodhi uses FAS for authentication to
> > provide a strong relationship between a user and the content
> > """
> > Source:
> > https://fedoraproject.org/wiki/ReleaseEngineering/Philosophy#Auditable
> >
> > This has lead me to the question: Is this all what SSL certs are bringing
> > us?
>
>  It does a two way authentication/authorisation. apache on the server side
>  validates that the cert is signed by our CA and not revoked. while on the
>  client side koji at least. I would need to double check that fedpkg does for
>  lookaside cache, verifies that that server cert is signed by the appropriate
>  CA and is not revoked also. 
https://github.com/release-engineering/dist-git

Good overview there. To quote "The client authenticates with an ssh
certificate for git communication and with an http client certificate
for uploads to the lookaside cache." but it seems the comms is in rpkg
dep of fedpkg (or aren't we using a fedpkg with rpkg support yet?).

Peter

[Pierre-Yves Chibon]

On Mon, Apr 27, 2015 at 10:21:57AM -0400, Colin Walters wrote:
>  On Mon, Apr 27, 2015, at 09:45 AM, Pierre-Yves Chibon wrote:
>  >
>  > This has lead me to the question: Is this all what SSL certs are bringing us?
>  
>  I think the ability to do a commit should be thought of as equivalent to the ability
>  to do a build - because anyone who can commit something to a package will
>  cause it to be implicitly included in the build that another person does
>  
>  That implies build access should be gated by SSH key, not by API token or
>  SSL certificate.   (Or alternatively the commit authentication method changed
>  to match whatever is chosen for build) 
But we allow new-comers to make scratch-build on koji before they are in the
packager group. Giving them the opportunity to test their build in real
condition.
Using ssh could also become problematic for application like koschei no?

Pierre

[Colin Walters]

On Mon, Apr 27, 2015, at 01:12 PM, Pierre-Yves Chibon wrote:

>  But we allow new-comers to make scratch-build on koji before they are
> in the
>  packager group. Giving them the opportunity to test their build in real
>  condition. 
The "non-packager scratch build" case is addressed by COPR, no?

>  Using ssh could also become problematic for application like koschei
> no? 
I don't see why - it's just as easy to make a ssh key for an application
as it is for an SSL certificate.  If starting a build is a matter of ssh
restricted logins, it wouldn't be that hard to create a connection
programmatically, via subprocess or one of the ssh shared libraries.

(Also, it'd be nice if I could include X-Koji-Build-Now: yes metadata
 in the commit message and skip the secondary build invocation entirely,
 or even go to a model where builds happen on commit by default,
 with a way to opt-out)

[Pierre-Yves Chibon]

On Mon, Apr 27, 2015 at 01:39:45PM -0400, Colin Walters wrote:
>  On Mon, Apr 27, 2015, at 01:12 PM, Pierre-Yves Chibon wrote:
>  
>  > But we allow new-comers to make scratch-build on koji before they are in the
>  > packager group. Giving them the opportunity to test their build in real
>  > condition.
>  
>  The "non-packager scratch build" case is addressed by COPR, no? 
Well, not entirely, copr is not providing exactly the same environment as koji,
so some build might pass on copr and not in koji.

>  > Using ssh could also become problematic for application like
> koschei no?
>  
>  I don't see why - it's just as easy to make a ssh key for an application
>  as it is for an SSL certificate.  If starting a build is a matter of ssh
>  restricted logins, it wouldn't be that hard to create a connection
>  programmatically, via subprocess or one of the ssh shared libraries.
>  
>  (Also, it'd be nice if I could include X-Koji-Build-Now: yes metadata
>   in the commit message and skip the secondary build invocation entirely,
>   or even go to a model where builds happen on commit by default,
>   with a way to opt-out) 
I wonder how often people commit to master without building behind, a good
question for fedmsg one of these days.


Pierre

[Ryan S. Brown]

On 04/27/2015 12:50 PM, Dennis Gilmore wrote:
>  On Monday, April 27, 2015 06:23:38 PM Pierre-Yves Chibon wrote:
> > On Mon, Apr 27, 2015 at 05:59:14PM +0200, Till Maas wrote:
> >> On Mon, Apr 27, 2015 at 03:45:00PM +0200, Pierre-Yves Chibon wrote: 
[snip]

> >>> On the otherside, recently we have been more and more
> feeling the need
> >>> for a centralized API authentication place. Something along the line of
> >>> a personalized 0Auth. This has also pros and cons.
> >>>
> >>> pros
> >>>
> >>>   - API token per user and per application
> >>
> >> This is something I would like very much, but also with a fine-grained
> >> permissions system. E.g. allowing to create a token that can only be
> >> used to retire pkgs in pkgdb could be used to automate retiring pkgs
> >> without using credentials that can also a everything else.
> >
> > This is really something that would be cool to get :)
>  This is not something that can really be done with certs etc. it would require 
>  a fundamental change in how all the tools deal with permissions. 
Why isn't this possible with certs? Seems like an application/tools
authorization problem, not an authentication mechanism problem. One of
my workplaces had an internal system for distributing certs that
provided access for users and service accounts. The ou/cn/dn/groups
system has all the semantics you need to express complex permissions.

API tokens don't give delegation/permissions for free, though I do admit
that certificate expiry leaves...things to be desired.

-- 
Ryan Brown / Software Engineer, Openstack / Red Hat, Inc.

[Dennis Gilmore]

On Monday, April 27, 2015 01:48:37 PM Ryan S. Brown wrote:
>  On 04/27/2015 12:50 PM, Dennis Gilmore wrote:
>  > On Monday, April 27, 2015 06:23:38 PM Pierre-Yves Chibon wrote:
>  >> On Mon, Apr 27, 2015 at 05:59:14PM +0200, Till Maas wrote:
>  >>> On Mon, Apr 27, 2015 at 03:45:00PM +0200, Pierre-Yves Chibon wrote:
>  [snip]
>  
>  >>>> On the otherside, recently we have been more and more feeling the need
>  >>>> for a centralized API authentication place. Something along the line of
>  >>>> a personalized 0Auth. This has also pros and cons.
>  >>>> 
>  >>>> pros
>  >>>> 
>  >>>>   - API token per user and per application
>  >>> 
>  >>> This is something I would like very much, but also with a fine-grained
>  >>> permissions system. E.g. allowing to create a token that can only be
>  >>> used to retire pkgs in pkgdb could be used to automate retiring pkgs
>  >>> without using credentials that can also a everything else.
>  >> 
>  >> This is really something that would be cool to get :)
>  > 
>  > This is not something that can really be done with certs etc. it would
>  > require a fundamental change in how all the tools deal with permissions.
>  
>  Why isn't this possible with certs? Seems like an application/tools
>  authorization problem, not an authentication mechanism problem. One of
>  my workplaces had an internal system for distributing certs that
>  provided access for users and service accounts. The ou/cn/dn/groups
>  system has all the semantics you need to express complex permissions.
>  
>  API tokens don't give delegation/permissions for free, though I do admit
>  that certificate expiry leaves...things to be desired. 
Koji for instance has very limited knowledge of permissions and does not have 
a fine tuned permission setup. you could issue a cert for doing one thing, but 
koji has no way to enforce that. its also not possible with other types of 
authentication and authorization. in the above example.  there is no way that 
you can enforce the "use this token only to retire packages" its a all or 
nothing thing currently. I was trying to say that the apps need much much more 
work than just issuing different certs/tokens etc 

Dennis