Greetings.
We have a request ( https://pagure.io/fedora-infrastructure/issue/5372 ) to setup ssl cert pinning for ostree deliverables. It's also been a long wishlist item to have that for rpm deliverables too. Unfortunately there's a bunch of moving parts here that we need to sort out before we can move this forward.
First some background/info:
* kojipkgs.fedoraproject.org currently uses a valid digisign cert. It needs this because browsers download from it directly, our builders download from it directly, etc.
* pkgs/koji currently use certs signed by the Fedora Koji CA (which expires in 2024). This is currently needed by koji to do builds and the upload cgi for lookaside.
* We are hoping to deploy soon a pair of freeipa servers in production that get information from fas and allow us to issue kerberos tickets. koji can already authenticate via this method.
* There's an outstanding ticket about having a verified way to get source: https://pagure.io/fedora-infrastructure/issue/2324
Questions we need to figure out:
* Are we going to retire/replace the koji CA? My thought was yes, but I think Dennis wasn't on board with this. Can anyone who wants to save it speak up? :)
* The upload cgi would need to auth with kerberos and sigul would need to auth with kerberos for this to work.
* If we are not completely retiring the koji CA, are we replacing it?
* Is ostree going to stay distributed at kojipkgs ? Or is it going to move somewhere else? we should figure out the final place for it before we go setting up cert pinning.
* The simple way to do pinning is for the application(s) to include a hard coded list of valid certs. I guess this would require changes in librepo and somewhere in ostree?
* The complex way to do pinning would be to setup https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning For this we would need to get backup keys for our cert(s) that are used for this and setup webservers to send the right headers. This would also need (more complex) changes in librepo and/or somewhere in ostree. This would also optionally get us reports of violations.
Thoughts? Comments?
kevin
Hi,
...snip...
Questions we need to figure out:
- Are we going to retire/replace the koji CA? My thought was yes, but I think Dennis wasn't on board with this. Can anyone who wants to save it speak up? :)
I want to kill this CA. If there's anyone that sees problems with this, talk to me and I'll see how to resolve them, as I have a plan for all the issues I have so far foreseen.
- The upload cgi would need to auth with kerberos and sigul would need to auth with kerberos for this to work.
Upload CGI is no issue as it uses http auth, so is just configuration. I have a sigul patch for krb support that I'm going to merge soon.
- If we are not completely retiring the koji CA, are we replacing it?
Not if it's up to me.
Is ostree going to stay distributed at kojipkgs ? Or is it going to move somewhere else? we should figure out the final place for it before we go setting up cert pinning.
The simple way to do pinning is for the application(s) to include a hard coded list of valid certs. I guess this would require changes in librepo and somewhere in ostree?
As far as I know, yum/dnf supports setting a cafile for repos, so we can just update fedora-repos.
- The complex way to do pinning would be to setup https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning For this we would need to get backup keys for our cert(s) that are used for this and setup webservers to send the right headers. This would also need (more complex) changes in librepo and/or somewhere in ostree. This would also optionally get us reports of violations.
I would prefer this, since that means the configuration is server-side and we can phase over to a different CA or something at a later point in time way easier.
Thoughts? Comments?
kevin
Regards, Patrick Uiterwijk
On Mon, 10 Oct 2016 16:57:25 +0000 Patrick Uiterwijk puiterwijk@redhat.com wrote:
...snip...
As far as I know, yum/dnf supports setting a cafile for repos, so we can just update fedora-repos.
That doesn't help. If we are using a well known cert, it's already valid based on the system ca's, and IMHO it would be very poor to use a self signed cert for this. So, either librepo carries a static list for our base repos or we add support for HPKP.
- The complex way to do pinning would be to setup https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning For this we would need to get backup keys for our cert(s) that are used for this and setup webservers to send the right headers. This would also need (more complex) changes in librepo and/or
somewhere in ostree. This would also optionally get us reports of violations.
I would prefer this, since that means the configuration is server-side and we can phase over to a different CA or something at a later point in time way easier.
Still will need HPKP support in the clients... but yeah, it has advantages.
kevin
On Mon, Oct 10, 2016, at 01:04 PM, Kevin Fenzi wrote:
On Mon, 10 Oct 2016 16:57:25 +0000 Patrick Uiterwijk puiterwijk@redhat.com wrote:
...snip...
As far as I know, yum/dnf supports setting a cafile for repos, so we can just update fedora-repos.
That doesn't help. If we are using a well known cert, it's already valid based on the system ca's, and IMHO it would be very poor to use a self signed cert for this. So, either librepo carries a static list for our base repos or we add support for HPKP.
I would s/static list/custom root CA set/.
It's worth emphasizing that having a custom CA root set is a very standard thing to do - we're basically just doing exactly what ca-certificates does, just with a different configuration.
And it's supported today by both librepo and ostree, and many HTTP libraries in general. And both of those code paths are used today for Red Hat Enterprise Linux Atomic Host, talking to the default Akamai CDN.
HPKP is (as far as I know) mostly a browser thing[1]; it isn't implemented by libcurl for example. Though it does make sense for general outside-of-the-browser use cases, from an operating system perspective, we already have a mechanism to ship the configuration out-of-band to client systems by embedding it into the installation media and the update stream. The same as we do for GPG.
Basically, I think it's easiest if we think of the keys/configuration for CA-pinning the same as GPG.
[1] Though I did find https://github.com/tam7t/hpkp - kind of curious what Digital Ocean is doing with it if anything
On Mon, 10 Oct 2016 13:16:23 -0400 Colin Walters walters@verbum.org wrote:
On Mon, Oct 10, 2016, at 01:04 PM, Kevin Fenzi wrote:
On Mon, 10 Oct 2016 16:57:25 +0000 Patrick Uiterwijk puiterwijk@redhat.com wrote:
...snip...
As far as I know, yum/dnf supports setting a cafile for repos, so we can just update fedora-repos.
That doesn't help. If we are using a well known cert, it's already valid based on the system ca's, and IMHO it would be very poor to use a self signed cert for this. So, either librepo carries a static list for our base repos or we add support for HPKP.
I would s/static list/custom root CA set/.
It's worth emphasizing that having a custom CA root set is a very standard thing to do - we're basically just doing exactly what ca-certificates does, just with a different configuration.
And it's supported today by both librepo and ostree, and many HTTP libraries in general. And both of those code paths are used today for Red Hat Enterprise Linux Atomic Host, talking to the default Akamai CDN.
HPKP is (as far as I know) mostly a browser thing[1]; it isn't implemented by libcurl for example. Though it does make sense for general outside-of-the-browser use cases, from an operating system perspective, we already have a mechanism to ship the configuration out-of-band to client systems by embedding it into the installation media and the update stream. The same as we do for GPG.
Basically, I think it's easiest if we think of the keys/configuration for CA-pinning the same as GPG.
But does that not mean anyone going to the same place with a browser or command line downloading specific packages will get a "sorry, this cert is not trusted" ? Thats not such a big deal for ostree's, but for rpms, people do this all the time.
kevin
On Mon, Oct 10, 2016, at 01:58 PM, Kevin Fenzi wrote:
But does that not mean anyone going to the same place with a browser or command line downloading specific packages will get a "sorry, this cert is not trusted" ? Thats not such a big deal for ostree's, but for rpms, people do this all the time.
Yes, there are two things someone could do then:
1) Go to any of the many non-ca-pinned URLs I wasn't proposing switching any of the existing URLs, but adding a new one, and we should ensure that the exact same view is available with a regular ca-certificates signed cert 2) Use curl --cafile or equivalent (or hack it with curl -k etc.)
On Tue, 11 Oct 2016 14:31:55 -0400 Colin Walters walters@verbum.org wrote:
On Mon, Oct 10, 2016, at 01:58 PM, Kevin Fenzi wrote:
But does that not mean anyone going to the same place with a browser or command line downloading specific packages will get a "sorry, this cert is not trusted" ? Thats not such a big deal for ostree's, but for rpms, people do this all the time.
Yes, there are two things someone could do then:
- Go to any of the many non-ca-pinned URLs
I wasn't proposing switching any of the existing URLs, but adding a new one, and we should ensure that the exact same view is available with a regular ca-certificates signed cert 2) Use curl --cafile or equivalent (or hack it with curl -k etc.)
Sure, but they won't. They will complain that we have an invalid cert and we will need to explain to them whats going on. ;)
Instead of shipping a fedora-ca that you verify against, why not do what chrom*/firefox do and have a hardcoded list of hashes that must be in the cert?
https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning
https://src.chromium.org/viewvc/chrome/trunk/src/net/http/transport_security...
If we ever switched from using Digicert for our certs we would need to change this, but otherwise it should protect from the Rogue CA threat (unless it was Digicert I guess).
Also in the same file chom*/firefox set a list of sites to assume ssl, which would also be nice to hard code.
kevin
On Wed, Oct 12, 2016, at 03:17 PM, Kevin Fenzi wrote:
Sure, but they won't. They will complain that we have an invalid cert and we will need to explain to them whats going on. ;)
I still think this would be mostly covered if the yum repo files and the ostree remote config had a comment like:
# This URL is ca-pinned, see https://fedoraproject.org/wiki/CAPinning
But:
Instead of shipping a fedora-ca that you verify against, why not do what chrom*/firefox do and have a hardcoded list of hashes that must be in the cert?
https://wiki.mozilla.org/SecurityEngineering/Public_Key_Pinning
https://src.chromium.org/viewvc/chrome/trunk/src/net/http/transport_security...
The Chrome and Firefox implementations seem different - Firefox hashes the certs, but Chrome seems to have a CA whitelist, which is actually a lot easier to read.
Anyways, there's a higher level question here - you're arguing for pinning to Digicert rather than a custom CA. That seems good enough, but I think we need a recovery mechanism in case Digicert explodes.
So I'd propose pinning to a 3 set of CAs:
- Digicert - Some other well-regarded CA vendor - A Fedora-infra custom CA (doesn't have to be deployed, just a backup plan)
And as for a specific implementation mechanism, we'd have just those CAs in /etc/pki/tls/certs/fedora-infra.crt or so, and that file would be in the fedora-repos package. The argument for this again is that librepo and ostree already have all of the code for this, and so does curl etc.
Doing the hashes of the certs like Firefox does is certainly possible, but it requires custom logic in the HTTP layer, and there's no standard configuration file formats for the data, etc.
Also in the same file chom*/firefox set a list of sites to assume ssl, which would also be nice to hard code.
Yeah, we could follow up with this adding Fedora sites to the browser lists. Chrome's version seems saner to me.
On Fri, Oct 14, 2016, at 08:42 AM, Colin Walters wrote:
Anyways, there's a higher level question here - you're arguing for pinning to Digicert rather than a custom CA. That seems good enough, but I think we need a recovery mechanism in case Digicert explodes.
So I'd propose pinning to a 3 set of CAs:
- Digicert
- Some other well-regarded CA vendor
- A Fedora-infra custom CA (doesn't have to be deployed, just a backup plan)
Any further thoughts here?
And as for a specific implementation mechanism, we'd have just those CAs in /etc/pki/tls/certs/fedora-infra.crt or so, and that file would be in the fedora-repos package. The argument for this again is that librepo and ostree already have all of the code for this, and so does curl etc.
Doing the hashes of the certs like Firefox does is certainly possible, but it requires custom logic in the HTTP layer, and there's no standard configuration file formats for the data, etc.
Also in the same file chom*/firefox set a list of sites to assume ssl, which would also be nice to hard code.
Yeah, we could follow up with this adding Fedora sites to the browser lists. Chrome's version seems saner to me.
On Mon, 21 Nov 2016 10:16:55 -0500 Colin Walters walters@verbum.org wrote:
On Fri, Oct 14, 2016, at 08:42 AM, Colin Walters wrote:
Anyways, there's a higher level question here - you're arguing for pinning to Digicert rather than a custom CA. That seems good enough, but I think we need a recovery mechanism in case Digicert explodes.
So I'd propose pinning to a 3 set of CAs:
- Digicert
- Some other well-regarded CA vendor
- A Fedora-infra custom CA (doesn't have to be deployed, just a
backup plan)
Any further thoughts here?
And as for a specific implementation mechanism, we'd have just those CAs in /etc/pki/tls/certs/fedora-infra.crt or so, and that file would be in the fedora-repos package. The argument for this again is that librepo and ostree already have all of the code for this, and so does curl etc.
I suppose thats workable if all the stakeholders agree.
So, we would need:
1. certs in fedora-repos package 2. librepo would need changes to know to check those. 3. ostree would need changes to know to check those.
I've not heard from librepo folks, should we ask them before moving forward here? fedora-repos would need ack from Dennis (who is currently on vacation), but I don't think that should be a problem.
kevin
On Wed, Nov 23, 2016, at 12:10 PM, Kevin Fenzi wrote:
I suppose thats workable if all the stakeholders agree.
To confirm, are you agreeing with:
So I'd propose pinning to a 3 set of CAs:
- Digicert
- Some other well-regarded CA vendor
- A Fedora-infra custom CA (doesn't have to be deployed, just a backup plan)
You were arguing earlier to pin to just digicert I think (though I can't find that now).
We could probably move forward with Digicert + 1-2 other vendors as well. Maybe to be conservative 2. We can easily add a custom CA to the set as well at any point.
On Wed, 23 Nov 2016 15:45:55 -0500 Colin Walters walters@verbum.org wrote:
On Wed, Nov 23, 2016, at 12:10 PM, Kevin Fenzi wrote:
I suppose thats workable if all the stakeholders agree.
To confirm, are you agreeing with:
So I'd propose pinning to a 3 set of CAs:
- Digicert
- Some other well-regarded CA vendor
- A Fedora-infra custom CA (doesn't have to be deployed, just a backup plan)
You were arguing earlier to pin to just digicert I think (though I can't find that now).
Yeah. I am not sure the process we will need to use to get some other CA vendor. RH has a relationship with digicert, so we get our certs via that. When using another vendor we may have to go through some red-tape. So, I can't commit for a time when this would be ready.
We could probably move forward with Digicert + 1-2 other vendors as well. Maybe to be conservative 2. We can easily add a custom CA to the set as well at any point.
We should make sure that the librepo/dnf folks are on board with this plan before moving forward. :)
kevin
On Mon, Nov 28, 2016, at 11:20 AM, Kevin Fenzi wrote:
Yeah. I am not sure the process we will need to use to get some other CA vendor. RH has a relationship with digicert, so we get our certs via that. When using another vendor we may have to go through some red-tape. So, I can't commit for a time when this would be ready.
OK, can you file the issue/request and link me to it?
We could probably move forward with Digicert + 1-2 other vendors as well. Maybe to be conservative 2. We can easily add a custom CA to the set as well at any point.
We should make sure that the librepo/dnf folks are on board with this plan before moving forward. :)
Sure, I sent Honza and Igor a mail.
On Mon, 28 Nov 2016 15:32:02 -0500 Colin Walters walters@verbum.org wrote:
On Mon, Nov 28, 2016, at 11:20 AM, Kevin Fenzi wrote:
Yeah. I am not sure the process we will need to use to get some other CA vendor. RH has a relationship with digicert, so we get our certs via that. When using another vendor we may have to go through some red-tape. So, I can't commit for a time when this would be ready.
OK, can you file the issue/request and link me to it?
We could probably move forward with Digicert + 1-2 other vendors as well. Maybe to be conservative 2. We can easily add a custom CA to the set as well at any point.
We should make sure that the librepo/dnf folks are on board with this plan before moving forward. :)
Sure, I sent Honza and Igor a mail.
Hum. I was writing up an email on this, and something occurred to me.
The various browsers already have our digicert cert hard coded. So, if we ever had problems with that cert and had to switch to the secondary or tertiary certs, all browser access would be broken. ;(
So, perhaps we should be more targeted here and only do this for some particular endpoints? mirrors.fedoraproject.org and dl.fedoraproject.org ? That way if we had to fall back to another cert only those would be broken for browsers.
Or should I just not worry too much about it because anything that causes us to switch from the primary cert would likely be a massive blowup anyhow?
kevin
On Tue, Nov 29, 2016, at 02:00 PM, Kevin Fenzi wrote:
The various browsers already have our digicert cert hard coded.
Hum, really? Reference?
$ pwd /home/walters/src/github/mozilla/gecko-dev $ git rev-parse HEAD a8b5f53e7df90df655a0982e94087ee83290c22e $ git grep fedoraproject.org
Shows me what looks like a HSTS preload entry, but not a ca/cert pin. (Remember HSTS preload == use https:// by default)
On Tue, 06 Dec 2016 17:14:48 -0500 Colin Walters walters@verbum.org wrote:
On Tue, Nov 29, 2016, at 02:00 PM, Kevin Fenzi wrote:
The various browsers already have our digicert cert hard coded.
Hum, really? Reference?
$ pwd /home/walters/src/github/mozilla/gecko-dev $ git rev-parse HEAD a8b5f53e7df90df655a0982e94087ee83290c22e $ git grep fedoraproject.org
Shows me what looks like a HSTS preload entry, but not a ca/cert pin. (Remember HSTS preload == use https:// by default)
ok, I finally found some time to sit down and do some digging and... I can't find it now. ;(
I cannot even see the HSTS preload there in firefox/chromium, but that might have gotten kicked out due to a error with paste.fedoraproject.org I can investigate more.
So, sorry for that sidetrack/delay... we can move forward AFAICT.
kevin
On Tue, Nov 29, 2016, at 02:00 PM, Kevin Fenzi wrote:
The various browsers already have our digicert cert hard coded. So, if we ever had problems with that cert and had to switch to the secondary or tertiary certs, all browser access would be broken. ;(
So, perhaps we should be more targeted here and only do this for some particular endpoints? mirrors.fedoraproject.org and dl.fedoraproject.org ? That way if we had to fall back to another cert only those would be broken for browsers.
I don't understand this btw - the CA pinning we're talking about would only be for software mechanisms like dnf/rpm-ostree and possibly docker/flatpak.
I'm certainly not advocating changing any other tools right now, although one could theroetically consider things like the `bodhi` command line tools (or possibly changing the underlying shared libraries).
On 9 December 2016 at 16:51, Colin Walters walters@verbum.org wrote:
On Tue, Nov 29, 2016, at 02:00 PM, Kevin Fenzi wrote:
The various browsers already have our digicert cert hard coded. So, if we ever had problems with that cert and had to switch to the secondary or tertiary certs, all browser access would be broken. ;(
So, perhaps we should be more targeted here and only do this for some particular endpoints? mirrors.fedoraproject.org and dl.fedoraproject.org ? That way if we had to fall back to another cert only those would be broken for browsers.
I don't understand this btw - the CA pinning we're talking about would only be for software mechanisms like dnf/rpm-ostree and possibly docker/flatpak.
I'm certainly not advocating changing any other tools right now, although one could theroetically consider things like the `bodhi` command line tools (or possibly changing the underlying shared libraries).
I don't think anyone is understanding each other.. because that isn't what I was getting from this thread until now.
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org
On Fri, Dec 9, 2016, at 05:38 PM, Stephen John Smoogen wrote:
I don't think anyone is understanding each other.. because that isn't what I was getting from this thread until now.
The thread has been 95% just me and Kevin on and off over the last 6 months. I asked him for clarification. Now, aAre you going to help? If so, can you elaborate on what you are/aren't understanding?
On 13 December 2016 at 12:37, Colin Walters walters@verbum.org wrote:
On Fri, Dec 9, 2016, at 05:38 PM, Stephen John Smoogen wrote:
I don't think anyone is understanding each other.. because that isn't what I was getting from this thread until now.
The thread has been 95% just me and Kevin on and off over the last 6 months. I asked him for clarification. Now, aAre you going to help? If so, can you elaborate on what you are/aren't understanding?
So the parts I think I am seeing different answers are: 1. What are we trying to accomplish and where? 2. What infrastructure is needed to accomplish this? 3. When does this need to be done? 4. What is the budget cost for setting up these certificates, reporting systems, pinned ips etc 5. Who is paying for this (hardware, certificates, domain names, etc). 6. Who is doing the implementation work and where?
To me Kevin, Patrick and you have said similar things but that you each have different ideas about what they mean and so aren't actually communicating.
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org
On Tue, Dec 13, 2016, at 01:49 PM, Stephen John Smoogen wrote:
So the parts I think I am seeing different answers are:
- What are we trying to accomplish and where?
- What infrastructure is needed to accomplish this?
I think this stuff is pretty well covered in the thread and should be hashed out by the people who are going to do the work.
- When does this need to be done?
I don't quite understand how this process stretched to 6 months already...but soon, please - I'd like Fedora to be a reference architecture. If it takes another few months something is clearly going wrong.
- What is the budget cost for setting up these certificates,
reporting systems, pinned ips etc 5. Who is paying for this (hardware, certificates, domain names, etc).
Fedora infra.
- Who is doing the implementation work and where?
I can take point on making and testing the client side changes.
FYI, I marked this thread to reply to, but I simply have not had time lately with last week on site at the datacenter and this weekend prepping for the flag day and this week helping people with fallout from the flag day.
I'll try and get back to this this week, but please have some patience.
kevin
On Tue, Dec 13, 2016, at 10:53 PM, Kevin Fenzi wrote:
FYI, I marked this thread to reply to, but I simply have not had time lately with last week on site at the datacenter and this weekend prepping for the flag day and this week helping people with fallout from the flag day.
I'll try and get back to this this week, but please have some patience.
That's fine! This seems like something we can get done if someone has a chance to focus on it for a day or two.
To summarize then, my understanding is:
- Fedora chooses 1-2 other CA providers to use as backup, and acquires certs from those providers for at least: * mirrors.fedoraproject.org * kojipkgs.fedoraproject.org (Or maybe it's simpler to just do all of *fedoraproject.org, either way) - I will take care of prepping a patch for just the ostree portion of Atomic Host to use this configuration - We'll create a wiki page collaboratively describing this, and post to fedora-devel how to enable it with the rpm-md configuration, and have interested testers try it
- At some point later, we change the fedora-repos package to enable that configuration by default
One thing this likely will break is people who run things like `sed -i -e s,baseurl=.*,http://myinternalmirror.corp.example.com /etc/yum.repos.d/fedora.repo`, but I think we'll get past those types of minor things over time; the security win is worth it.
On Wed, 14 Dec 2016 09:16:47 -0500 Colin Walters walters@verbum.org wrote:
On Tue, Dec 13, 2016, at 10:53 PM, Kevin Fenzi wrote:
FYI, I marked this thread to reply to, but I simply have not had time lately with last week on site at the datacenter and this weekend prepping for the flag day and this week helping people with fallout from the flag day.
I'll try and get back to this this week, but please have some patience.
That's fine! This seems like something we can get done if someone has a chance to focus on it for a day or two.
yeah.
To summarize then, my understanding is:
- Fedora chooses 1-2 other CA providers to use as backup, and acquires certs from those providers for at least:
(Or maybe it's simpler to just do all of *fedoraproject.org,
- mirrors.fedoraproject.org
- kojipkgs.fedoraproject.org
either way)
Yep.
- I will take care of prepping a patch for just the ostree portion of Atomic Host to use this configuration
- We'll create a wiki page collaboratively describing this, and post to fedora-devel how to enable it with the rpm-md
configuration, and have interested testers try it
- At some point later, we change the fedora-repos package to enable that configuration by default
Did you hear back from the dnf/librepo folks? Is there a ticket or email thread I can follow along for their side of things?
I'll fire off some emails here to figure out what other CA we could use and how (our current approved process is to get them all via digicert, so we need to find out a new process), as well as run it by infosec folks to make sure we didn't forget anything.
One thing this likely will break is people who run things like `sed -i -e s,baseurl=.*,http://myinternalmirror.corp.example.com /etc/yum.repos.d/fedora.repo`, but I think we'll get past those types of minor things over time; the security win is worth it.
Yeah, I think we cant worry too much about those cases, and I think they are pretty small these days.
kevin
On Fri, 09 Dec 2016 16:51:25 -0500 Colin Walters walters@verbum.org wrote:
On Tue, Nov 29, 2016, at 02:00 PM, Kevin Fenzi wrote:
The various browsers already have our digicert cert hard coded. So, if we ever had problems with that cert and had to switch to the secondary or tertiary certs, all browser access would be broken. ;(
So, perhaps we should be more targeted here and only do this for some particular endpoints? mirrors.fedoraproject.org and dl.fedoraproject.org ? That way if we had to fall back to another cert only those would be broken for browsers.
I don't understand this btw - the CA pinning we're talking about would only be for software mechanisms like dnf/rpm-ostree and possibly docker/flatpak.
Right now for say dnf, it would hit mirrors.fedoraproject.org (for the metalink) and possibly dl.fedoraproject.org (if it happened to get it at the end of the metalink). I was saying instead of pinning our wildcard *.fedoraproject.org cert (which we use for a number of sites / places) we could just get specific non wildcard ones for these sites.
However, pondering on it more those would still have to be on the proxies, so I am not sure it would buy us in the end.
kevin
On Monday, October 10, 2016 10:27:29 AM CDT Kevin Fenzi wrote:
Greetings.
We have a request ( https://pagure.io/fedora-infrastructure/issue/5372 ) to setup ssl cert pinning for ostree deliverables. It's also been a long wishlist item to have that for rpm deliverables too. Unfortunately there's a bunch of moving parts here that we need to sort out before we can move this forward.
First some background/info:
kojipkgs.fedoraproject.org currently uses a valid digisign cert. It needs this because browsers download from it directly, our builders download from it directly, etc.
pkgs/koji currently use certs signed by the Fedora Koji CA (which expires in 2024). This is currently needed by koji to do builds and the upload cgi for lookaside.
The koji CA expires in 2018
We are hoping to deploy soon a pair of freeipa servers in production that get information from fas and allow us to issue kerberos tickets. koji can already authenticate via this method.
There's an outstanding ticket about having a verified way to get source: https://pagure.io/fedora-infrastructure/issue/2324
Questions we need to figure out:
- Are we going to retire/replace the koji CA? My thought was yes, but I think Dennis wasn't on board with this. Can anyone who wants to save it speak up? :)
I am against kerberos being the only auth mechanism. I suspect that some people either cant for reasons beyond our control or won't set up kerberos for auth
The upload cgi would need to auth with kerberos and sigul would need to auth with kerberos for this to work.
If we are not completely retiring the koji CA, are we replacing it?
If not retired it has to be replaced, could be certs from freeipa that auto renew with certmonger, which i suspect users would like better than entering their kerberos password once a day.
- Is ostree going to stay distributed at kojipkgs ? Or is it going to move somewhere else? we should figure out the final place for it before we go setting up cert pinning.
No, it needs to go on the mirrors when we sort out how to mirror it, and the client and mirrormanager support it
The simple way to do pinning is for the application(s) to include a hard coded list of valid certs. I guess this would require changes in librepo and somewhere in ostree?
The complex way to do pinning would be to setup https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning For this we would need to get backup keys for our cert(s) that are used for this and setup webservers to send the right headers. This would also need (more complex) changes in librepo and/or somewhere in ostree. This would also optionally get us reports of violations.
Thoughts? Comments?
Not against making changes, but I do not think that they totally fit into long term goals
Dennis
kevin
I meant to reply to this eariler. ;)
On Mon, 10 Oct 2016 17:20:06 -0500 Dennis Gilmore dennis@ausil.us wrote:
On Monday, October 10, 2016 10:27:29 AM CDT Kevin Fenzi wrote:
Greetings.
We have a request ( https://pagure.io/fedora-infrastructure/issue/5372 ) to setup ssl cert pinning for ostree deliverables. It's also been a long wishlist item to have that for rpm deliverables too. Unfortunately there's a bunch of moving parts here that we need to sort out before we can move this forward.
First some background/info:
- kojipkgs.fedoraproject.org currently uses a valid digisign cert.
It needs this because browsers download from it directly, our builders download from it directly, etc.
- pkgs/koji currently use certs signed by the Fedora Koji CA (which expires in 2024). This is currently needed by koji to do builds
and the upload cgi for lookaside.
The koji CA expires in 2018
ok
- We are hoping to deploy soon a pair of freeipa servers in
production that get information from fas and allow us to issue kerberos tickets. koji can already authenticate via this method.
- There's an outstanding ticket about having a verified way to get source: https://pagure.io/fedora-infrastructure/issue/2324
Questions we need to figure out:
- Are we going to retire/replace the koji CA? My thought was yes,
but I think Dennis wasn't on board with this. Can anyone who wants to save it speak up? :)
I am against kerberos being the only auth mechanism. I suspect that some people either cant for reasons beyond our control or won't set up kerberos for auth
I'm interested in what these reasons could be.
Maintaining 2 authentication stacks is not something we really want to do if we can at all avoid it. And kerberos should be a good deal more secure than certs I would think, and we still don't have a great setup to manage the certs.
- The upload cgi would need to auth with kerberos and sigul would
need to auth with kerberos for this to work.
- If we are not completely retiring the koji CA, are we replacing
it?
If not retired it has to be replaced, could be certs from freeipa that auto renew with certmonger, which i suspect users would like better than entering their kerberos password once a day.
well, we can adjust the kerberos settings. If they can renew for a week would that be sufficent?
- Is ostree going to stay distributed at kojipkgs ? Or is it going
to move somewhere else? we should figure out the final place for it before we go setting up cert pinning.
No, it needs to go on the mirrors when we sort out how to mirror it, and the client and mirrormanager support it
Right, so doing some cert pinning right now might be not a great idea if we are just going to move it.
- The simple way to do pinning is for the application(s) to include
a hard coded list of valid certs. I guess this would require changes in librepo and somewhere in ostree?
- The complex way to do pinning would be to setup https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning For this we would need to get backup keys for our cert(s) that are used for this and setup webservers to send the right headers. This would also need (more complex) changes in librepo and/or
somewhere in ostree. This would also optionally get us reports of violations.
Thoughts? Comments?
Not against making changes, but I do not think that they totally fit into long term goals
Well, I can definitely see the need for cert pinning and moving to kerberos, but perhaps you can expand on the long term goals that this doesn't address?
kevin
On jueves, 13 de octubre de 2016 1:34:42 PM CDT Kevin Fenzi wrote:
I meant to reply to this eariler. ;)
I just now saw the reply :(
On Mon, 10 Oct 2016 17:20:06 -0500
Dennis Gilmore dennis@ausil.us wrote:
On Monday, October 10, 2016 10:27:29 AM CDT Kevin Fenzi wrote:
Greetings.
We have a request ( https://pagure.io/fedora-infrastructure/issue/5372 ) to setup ssl cert pinning for ostree deliverables. It's also been a long wishlist item to have that for rpm deliverables too. Unfortunately there's a bunch of moving parts here that we need to sort out before we can move this forward.
First some background/info:
- kojipkgs.fedoraproject.org currently uses a valid digisign cert.
It needs this because browsers download from it directly, our builders download from it directly, etc.
pkgs/koji currently use certs signed by the Fedora Koji CA (which
expires in 2024). This is currently needed by koji to do builds
and the upload cgi for lookaside.
The koji CA expires in 2018
ok
- We are hoping to deploy soon a pair of freeipa servers in
production that get information from fas and allow us to issue kerberos tickets. koji can already authenticate via this method.
There's an outstanding ticket about having a verified way to get
Questions we need to figure out:
- Are we going to retire/replace the koji CA? My thought was yes,
but I think Dennis wasn't on board with this. Can anyone who wants to save it speak up? :)
I am against kerberos being the only auth mechanism. I suspect that some people either cant for reasons beyond our control or won't set up kerberos for auth
I'm interested in what these reasons could be.
The machines managed by their employers IT who forbid using any external kerberos. The users just do not like kerberos.
Maintaining 2 authentication stacks is not something we really want to do if we can at all avoid it. And kerberos should be a good deal more secure than certs I would think, and we still don't have a great setup to manage the certs.
sure. it could be IPA makes it simple for us.
- The upload cgi would need to auth with kerberos and sigul would
need to auth with kerberos for this to work.
- If we are not completely retiring the koji CA, are we replacing
it?
If not retired it has to be replaced, could be certs from freeipa that auto renew with certmonger, which i suspect users would like better than entering their kerberos password once a day.
well, we can adjust the kerberos settings. If they can renew for a week would that be sufficent?
possibly, I am not the users. though with kerberos based ssh for access to git and fedora infra resources puts another benefit to having kerberos, as well as possibly sso on all the web apps.
- Is ostree going to stay distributed at kojipkgs ? Or is it going
to move somewhere else? we should figure out the final place for it
before we go setting up cert pinning.
No, it needs to go on the mirrors when we sort out how to mirror it, and the client and mirrormanager support it
Right, so doing some cert pinning right now might be not a great idea if we are just going to move it.
There is bound to be a lot of moving coming. what is any different about any of this that we do not need to do the same for rpm repos?
- The simple way to do pinning is for the application(s) to include
a hard coded list of valid certs. I guess this would require changes in librepo and somewhere in ostree?
The complex way to do pinning would be to setup
https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning For this we would need to get backup keys for our cert(s) that are used for this and setup webservers to send the right headers. This would also need (more complex) changes in librepo and/or
somewhere in ostree. This would also optionally get us reports of violations.
Thoughts? Comments?
Not against making changes, but I do not think that they totally fit into long term goals
Well, I can definitely see the need for cert pinning and moving to kerberos, but perhaps you can expand on the long term goals that this doesn't address?
Well can we do cert pinning with mirrors? I guess we could to mirrormanager. Does it matter with signed commits or repodata? some content will never or rarely be available via https
Dennis
kevin
On 10/13/2016 09:34 PM, Kevin Fenzi wrote:
- If we are not completely retiring the koji CA, are we replacing
it?
If not retired it has to be replaced, could be certs from freeipa that auto renew with certmonger, which i suspect users would like better than entering their kerberos password once a day.
well, we can adjust the kerberos settings. If they can renew for a week would that be sufficent?
Couldn't users simply generate keytab for themselves? Koji client supports keytabs directly (via setting in koji.conf or --keytab param), for other services it should be possible to run "kinit -k" (which can even be ran from startup stripts, cron etc.)
infrastructure@lists.fedoraproject.org
-
Colin Walters -
Dennis Gilmore -
Kevin Fenzi -
Mikolaj Izdebski -
Patrick Uiterwijk -
Stephen John Smoogen