Syncing fedora packages with security patches to mirrors

[Huzaifa Sidhpurwala]

Hi All,

I work for the Red Hat Product security team, and have been a fedora
contributor for several years. I was involved with Linux security issues
like heartbleed, shellshock etc.

For some time, I have noticed that due to the way fedora mirrors work,
it takes a lot of time for the packages with security fixes (specially
ones which have critical impact like openssl) to sync to mirrors. We
have been announcing links to koji builds for our users in the meantime,
which is really not scalable for large installs etc.

Also many times, while talking in conferences and otherwise to fedora
users, it seems the main concern is the time it takes these security
fixes to hit our mirrors.

I have tried talking to several people about a possible solution,
including CentOS guys and it seems there needs to be a solution to this
problem.

One possible solution which i can think of, is to have a security repo,
which is not  mirrored but centrally location, of-course there are
several problems with this approach and needs more discussion.

Let me know if this is the wrong list, or i need to mail someone else to
get the ball rolling.

Thanks for your time.


-- 
Huzaifa Sidhpurwala / Red Hat Product Security Team

[Adrian Reber]

On Thu, Mar 19, 2015 at 12:18:41PM +0530, Huzaifa Sidhpurwala wrote:
>  I work for the Red Hat Product security team, and have been a fedora
>  contributor for several years. I was involved with Linux security issues
>  like heartbleed, shellshock etc.
>  
>  For some time, I have noticed that due to the way fedora mirrors work,
>  it takes a lot of time for the packages with security fixes (specially
>  ones which have critical impact like openssl) to sync to mirrors. We
>  have been announcing links to koji builds for our users in the meantime,
>  which is really not scalable for large installs etc.
>  
>  Also many times, while talking in conferences and otherwise to fedora
>  users, it seems the main concern is the time it takes these security
>  fixes to hit our mirrors.
>  
>  I have tried talking to several people about a possible solution,
>  including CentOS guys and it seems there needs to be a solution to this
>  problem.
>  
>  One possible solution which i can think of, is to have a security repo,
>  which is not  mirrored but centrally location, of-course there are
>  several problems with this approach and needs more discussion. 
What you are suggesting is, I think, the same what Debian does with
their security repository at security.debian.org:

https://www.debian.org/security/faq#mirror

From my mirror admin point of view the problem is not getting the
packages to the mirrors. The step in the process which takes most of the
time is building the repository. If this security repository (including
the signing) could be created faster the files would be sooner on the
mirrors. So a small repository with higher (or more intelligent) mirror
frequency would probably help a lot.

I think the bigger problem is that it needs additional tools and a
concept how the packages move from the security repository to
updates-testing/updates-released.

		Adrian

[Huzaifa Sidhpurwala]

On 03/19/2015 12:48 PM, Adrian Reber wrote:
>  On Thu, Mar 19, 2015 at 12:18:41PM +0530, Huzaifa Sidhpurwala wrote:
> > I work for the Red Hat Product security team, and have been a fedora
> > contributor for several years. I was involved with Linux security issues
> > like heartbleed, shellshock etc.
> >
> > For some time, I have noticed that due to the way fedora mirrors work,
> > it takes a lot of time for the packages with security fixes (specially
> > ones which have critical impact like openssl) to sync to mirrors. We
> > have been announcing links to koji builds for our users in the meantime,
> > which is really not scalable for large installs etc.
> >
> > Also many times, while talking in conferences and otherwise to fedora
> > users, it seems the main concern is the time it takes these security
> > fixes to hit our mirrors.
> >
> > I have tried talking to several people about a possible solution,
> > including CentOS guys and it seems there needs to be a solution to this
> > problem.
> >
> > One possible solution which i can think of, is to have a security repo,
> > which is not  mirrored but centrally location, of-course there are
> > several problems with this approach and needs more discussion.
>  
>  What you are suggesting is, I think, the same what Debian does with
>  their security repository at security.debian.org:
>  
>  https://www.debian.org/security/faq#mirror
>  
>  From my mirror admin point of view the problem is not getting the
>  packages to the mirrors. The step in the process which takes most of the
>  time is building the repository. If this security repository (including
>  the signing) could be created faster the files would be sooner on the
>  mirrors. So a small repository with higher (or more intelligent) mirror
>  frequency would probably help a lot.
>  
>  I think the bigger problem is that it needs additional tools and a
>  concept how the packages move from the security repository to
>  updates-testing/updates-released.
>   
Here is what i think:

1. We have a much smaller security-testing repo, where pkgs land as soon
as they are build and sanity testing is done.

2. At the same time, packages reach update-testing. Once these packages
are stable and are pushed to the update repo, we wait for some X days
for all the mirrors to have these pkgs. Once this is done, the pkgs are
removed from the security repo.

3. This ensures the pkgs are provided to users asap, they get some
testing and at the same time, the stable repos have these pkgs.

What does everyone think?


-- 
Huzaifa Sidhpurwala / Red Hat Product Security Team

[Kevin Fenzi]

You are all covering old ground here. ;) 

Please see: 

https://fedorahosted.org/rel-eng/ticket/5886

and
https://fedoraproject.org/wiki/Urgent_updates_policy

Basically we need to gather stakeholders and figure out how we want to
handle various things and then move on to implementing something. 

I'm not sure where the best place is to discuss, I guess the releng
list. 

kevin