developing the "critical updates repo" plan

[Matthew Miller]

See <https://fedorahosted.org/rel-eng/ticket/5886>;. In short, the time to do
an updates compose and push (plus sync times to mirrors) severely limits our
ability to put out critical updates quickly. Would anyone be interested in
filling out a plan for an alternate repository which would use a special
expedited process to make ultra-critical updates available more quickly?


-- 
Matthew Miller           mattdm(a)mattdm.org          <http://mattdm.org/>

[Eric Christensen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Thu, May 22, 2014 at 11:26:07AM -0400, Matthew Miller wrote:
>  See <https://fedorahosted.org/rel-eng/ticket/5886>;. In short,
> the time to do
>  an updates compose and push (plus sync times to mirrors) severely limits our
>  ability to put out critical updates quickly. Would anyone be interested in
>  filling out a plan for an alternate repository which would use a special
>  expedited process to make ultra-critical updates available more quickly? 
I dislike the idea of a separate repo for ultra-critical updates.  Once a fix is available
for a vulnerability it should, IMO, be shipped as soon as possible.  I know this
doesn't fit into the Microsoft model or our model of community testing but really as
soon as you go public with a fix you've also just notified all the "bad
guys" out there to the vulnerability and exactly how to exploit it.  It's a race
condition at that point.

I'd much prefer to have a mechanism in place that allows these fixes to be pushed to
the repos almost immediately (once they've been properly tested).  I'm not exactly
sure how this can work but perhaps having QE tested patches packaged and ready for the
embargo time would meet Release Engineering's criteria for testing?

- -- Eric

- --------------------------------------------------
Eric "Sparks" Christensen
Red Hat, Inc - Product Security Team

sparks(a)redhat.com - sparks(a)fedoraproject.org
097C 82C3 52DF C64A 50C2  E3A3 8076 ABDE 024B B3D1
- --------------------------------------------------
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQGcBAEBCgAGBQJTf1THAAoJEB/kgVGp2CYveoQL/2bsR6TYYGYJcK15+8dVHKc0
> OO5MZmrVunH7Uml72Qxo661wsdcgdjjfUC9W1bZHjKUQhSQJ7jPmCku5TJEvGMJk
> cWAFJ6/uhUuhW2yih/STCRhXNcbnP/8VSFhGsQsmJhQyPA/3XZw3JDJUrq5p9ozX
> QH3BuJjiXyplJolOjYQVsdjU0fuGFKKnnY6NK2FA4+RYpGWR8yvDEkn9vGINf7fs
> EEkZ9A5zrPi/qkxGVXgUeLf9QvPztwmgetpIy02iWY7vxFnFcNIIuQIdhNAO88cN
> g2Id/h1shWK+OqdtbSPD3fLGzAPLYziDPS3/GCGT0kWrKVRYvakH93Z5mGpmgA1x
> ghRkap1FoqFcFZfLQD4gBhgtSAUZkM6RYgTxI+W7XWXcs5zDitonzUHJK+MCMlDp
> bZIX8TCt/qIlIJEhcXaRu7BP6xw46WhOiMwmCXpPsxKcVUR3nqa2PABdC2TQ91rS
> tr9Jf5K2oD1shfgTUMQTA0vqHR5H00b1J4eG6G3Yew==
> =OYo7
> -----END PGP SIGNATURE-----

[Matthew Miller]

On Fri, May 23, 2014 at 10:01:46AM -0400, Eric H. Christensen wrote:
>  I dislike the idea of a separate repo for ultra-critical updates. 
> Once a
>  fix is available for a vulnerability it should, IMO, be shipped as soon as
>  possible. I know this doesn't fit into the Microsoft model or our model of
>  community testing but really as soon as you go public with a fix you've
>  also just notified all the "bad guys" out there to the vulnerability and
>  exactly how to exploit it. It's a race condition at that point. 
I'm not sure I follow here. What do you dislike? This isn't meant to be a
hidden repo -- it's the "ship as soon as possible!" repo, so it sounds like
you're agreeing.

>  I'd much prefer to have a mechanism in place that allows these
> fixes to be
>  pushed to the repos almost immediately (once they've been properly
>  tested). I'm not exactly sure how this can work but perhaps having QE
>  tested patches packaged and ready for the embargo time would meet Release
>  Engineering's criteria for testing? 
Right, exactly -- that's the mechanism I'm looking for.

-- 
Matthew Miller    --   Fedora Project    --    <mattdm(a)fedoraproject.org&gt;
                                  "Tepid change for the somewhat better!"

[Tomas Mraz]

On Pá, 2014-05-23 at 10:01 -0400, Eric H. Christensen wrote:
>  On Thu, May 22, 2014 at 11:26:07AM -0400, Matthew Miller wrote:
>  > See <https://fedorahosted.org/rel-eng/ticket/5886>;. In short, the time to do
>  > an updates compose and push (plus sync times to mirrors) severely limits our
>  > ability to put out critical updates quickly. Would anyone be interested in
>  > filling out a plan for an alternate repository which would use a special
>  > expedited process to make ultra-critical updates available more quickly?
>  
>  I dislike the idea of a separate repo for ultra-critical updates.
>  Once a fix is available for a vulnerability it should, IMO, be shipped
>  as soon as possible.  I know this doesn't fit into the Microsoft model
>  or our model of community testing but really as soon as you go public
>  with a fix you've also just notified all the "bad guys" out there to
>  the vulnerability and exactly how to exploit it.  It's a race
>  condition at that point. 
Except the severity of various vulnerabilities is very different. And
the number of the updates that fix serious vulnerabilities is much lower
than the number of the remaining updates (including non-security ones).

>  I'd much prefer to have a mechanism in place that allows these
> fixes
>  to be pushed to the repos almost immediately (once they've been
>  properly tested).  I'm not exactly sure how this can work but perhaps
>  having QE tested patches packaged and ready for the embargo time would
>  meet Release Engineering's criteria for testing? 
You have to understand that the repoclosure process takes some time and
the metadata on the updates repository is much larger. This affects work
of release engineers and it basically prevents the mirrors to sync such
large repository too often.

-- 
Tomas Mraz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb
(You'll never know whether the road is wrong though.)

[Eric Christensen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Fri, May 23, 2014 at 10:16:41AM -0400, Matthew Miller wrote:
>  On Fri, May 23, 2014 at 10:01:46AM -0400, Eric H. Christensen wrote:
>  > I dislike the idea of a separate repo for ultra-critical updates.  Once a
>  > fix is available for a vulnerability it should, IMO, be shipped as soon as
>  > possible. I know this doesn't fit into the Microsoft model or our model of
>  > community testing but really as soon as you go public with a fix you've
>  > also just notified all the "bad guys" out there to the vulnerability and
>  > exactly how to exploit it. It's a race condition at that point.
>  
>  I'm not sure I follow here. What do you dislike? This isn't meant to be a
>  hidden repo -- it's the "ship as soon as possible!" repo, so it sounds
> like
>  you're agreeing. 
I guess I don't understand the need for the extra repo.  Why not just push it to
fedora-updates?

- -- 

- --------------------------------------------------
Eric "Sparks" Christensen
Fedora Project

sparks(a)fedoraproject.org - sparks(a)redhat.com
097C 82C3 52DF C64A 50C2  E3A3 8076 ABDE 024B B3D1
- --------------------------------------------------
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQGcBAEBCgAGBQJTf2V/AAoJEB/kgVGp2CYvEHoMAI2aV5YFzi29DxN0Hzsw8m/v
> EBWmze1DDb6yvatSpuLxmhMbaGPXbvy3dtKSZOf7o7fcYBomEbAtymLlzYOEggH0
> P2iccuKC5L41xCYlbTjDH9sAfP1/I5rH2fXnvRq6s/Pj5uygIUoWuEPRBxyvCkBt
> HWBCS/BXQ6D3zaO3IEiATuyyfGSOfqED3whYS8ShJnQAPpcXIz5fEqv0m3EHa+s1
> YS7SJtmMqrB4EjggS1MCOZaNOHxBBAP4ETHxCTopKx4qdDBIwv65BcL1OOeTi8I9
> h+/5J6CJ0308HjQphm+LKfX09IN4UjeZmfNmYE1ZQPV24K4J4I8O/NaIhA8P9qvE
> XBD8TWCNtjiSL/ra6UHYDUXg7vXNVFIYZS1NoC2MGkwb0cUISVjXfSQYbEOQE+yd
> Z4SHzHLh7Opjw8eOL60Bw5SbdfG2zZJyJJXY74WNTf8Z3LmCVa6inpNdQtdcfNcY
> d+r5AwPnFZQT9Unq3/6eHbHQiEA8a/ulB3N8Ouzb8w==
> =4zpT
> -----END PGP SIGNATURE-----

[Eric Christensen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Fri, May 23, 2014 at 04:21:43PM +0200, Tomas Mraz wrote:
>  You have to understand that the repoclosure process takes some time
> and
>  the metadata on the updates repository is much larger. This affects work
>  of release engineers and it basically prevents the mirrors to sync such
>  large repository too often. 
Okay, so would this new repo, only for [critical?] security updates be turned on by
default so everyone would be checking this repo?  As long as it's not an
optional/non-enabled repo I think it's a fine idea.

- -- Eric

- --------------------------------------------------
Eric "Sparks" Christensen
Red Hat, Inc - Product Security Team

sparks(a)redhat.com - sparks(a)fedoraproject.org
097C 82C3 52DF C64A 50C2  E3A3 8076 ABDE 024B B3D1
- --------------------------------------------------
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQGcBAEBCgAGBQJTf2ZoAAoJEB/kgVGp2CYvRzoMAJiHfOyXZ9W5JBpZLi0Bk6xi
> /E8GdmfFUcqN8UzW1MIFMOhbqriKrgo7XnVEgmw617adUEwsb/ah8o3FrAGH+EN9
> +p1Si94I+ZFSuFBpXCSPSmgHGfuL5hVP49XVGfnG2Hafli9cx/BVYPqoiReb+qss
> 1RgJdsacC4zaHM7rrTWZfbt5ELhd713nvu56nQlTdUVif4dPwxdzv7yoOUpYIczg
> R0wmGMoDAVzRoZ1T+7MPya/5llxi6StYAf1T9g9NBKadjPtUDkCRDSXPT3SE4iJv
> rQhMuEPAsrIEuiJ1PieNS6NQ8oX3Ecgaqsh8jRMOX9awIkLamq/cgcYLV+K/CiLO
> L79c7i3QcwIU6HABjtLhUg/k+ZVwudYZrFoMFn3/7vpSyl8Q3f+HB2Cp1kQB8ku6
> 6jCGUKHb6HmF/0GxrEjUMUCqrf2HloA578Yf8+reoofqirSUkBgeCvujaI89aYXV
> 0QC4In12ESlCbbvcAzEq9NYS2YW4dYvUXL2jyZnh6g==
> =iw/6
> -----END PGP SIGNATURE-----

[Matthew Miller]

On Fri, May 23, 2014 at 11:13:05AM -0400, Eric H. Christensen wrote:
>  > I'm not sure I follow here. What do you dislike? This
> isn't meant to be
>  > a hidden repo -- it's the "ship as soon as possible!" repo, so it
> sounds
>  > like you're agreeing.
>  I guess I don't understand the need for the extra repo.  Why not just push
>  it to fedora-updates? 
That takes a really long time. Something like 7 hours, best case. Improving
that would help, but even then having a separate repo might introduce the
possibilities of mirror sites refreshing just that small set more often.

I expect that usually it will be size _zero_ -- my idea is that the same
packages would also be pushed to the regular updates repo, and then
eventually removed from the critical updates one.

-- 
Matthew Miller    --   Fedora Project    --    <mattdm(a)fedoraproject.org&gt;
                                  "Tepid change for the somewhat better!"

[Matthew Miller]

On Fri, May 23, 2014 at 11:16:58AM -0400, Eric H. Christensen wrote:
>  Okay, so would this new repo, only for [critical?] security updates
> be
>  turned on by default so everyone would be checking this repo? As long as
>  it's not an optional/non-enabled repo I think it's a fine idea. 
Oh yes. Definitely.

-- 
Matthew Miller    --   Fedora Project    --    <mattdm(a)fedoraproject.org&gt;
                                  "Tepid change for the somewhat better!"

[Eric Christensen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Fri, May 23, 2014 at 11:37:21AM -0400, Matthew Miller wrote:
>  On Fri, May 23, 2014 at 11:16:58AM -0400, Eric H. Christensen wrote:
>  > Okay, so would this new repo, only for [critical?] security updates be
>  > turned on by default so everyone would be checking this repo? As long as
>  > it's not an optional/non-enabled repo I think it's a fine idea.
>  
>  Oh yes. Definitely. 
Oh, then I think it's a fine idea; well thought out.  :)  I'll pass along the
original to the Product Security folks just to make sure they don't have any further
input.

I really need to wake up before opening email in the morning.

- -- Eric

- --------------------------------------------------
Eric "Sparks" Christensen
Fedora Project

sparks(a)fedoraproject.org - sparks(a)redhat.com
097C 82C3 52DF C64A 50C2  E3A3 8076 ABDE 024B B3D1
- --------------------------------------------------
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQGcBAEBCgAGBQJTf4CAAAoJEB/kgVGp2CYvEPsL/0I+ZS/ZNyf6ABUv0fO/6ouJ
> DLRcT4pR/9YcLuaNCRcM8zCTtOtaLzZgRcjPznfh16CKSGmP5CDlMJElvoZfdKUc
> 4m6+shTmhK9vzK2iXSALT0DSiboP8CBckUsiaKPkuGFXZonyUTxYui5Ofg3rglY5
> lB+KfEKa31Q9MP2Zb73S2CmCZUWacatwYfzKPR5P5b9bNMHAU1Xbqb3/k++jjgJ9
> nZ5TfqIPYJIiJ/dvAXb7eXm6G97El9SKD56Hc1xfn5TTc7G6iIg96phX6R52a6Al
> IOyHMkwASjL1iYOGWWDb8z/8U63y6/kbZlp8yQl9MJfPyxHUeNqrooP7ZqeYDi41
> MP0ecD9LlkLRVafUCX012kNtpxeyPpUz6VbEU/vUlFwNTMQE9225o6BWgAnRyaas
> cTcoDjqGcNSj9/UO9vHuyFVS1IeqU7whXDzG2FQ109KUTk9KOeHver4K/LX5lLnD
> 2hMdn5E5ldAdBQ1ycmpSn8D8v2zDMRXihSnIFdVBbA==
> =Sr2k
> -----END PGP SIGNATURE-----

[Matthew Miller]

On Fri, May 23, 2014 at 01:08:22PM -0400, Eric H. Christensen wrote:
>  Oh, then I think it's a fine idea; well thought out.  :) 
> I'll pass along
>  the original to the Product Security folks just to make sure they don't
>  have any further input. 
Cool.

>  I really need to wake up before opening email in the morning. 
Once you explained what you were thinking, I see where you got that
interpretation from.  But nope. :)


-- 
Matthew Miller    --   Fedora Project    --    <mattdm(a)fedoraproject.org&gt;
                                  "Tepid change for the somewhat better!"

[Dennis Gilmore]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 23 May 2014 16:21:43 +0200
Tomas Mraz <tmraz(a)redhat.com&gt; wrote:

>  On Pá, 2014-05-23 at 10:01 -0400, Eric H. Christensen wrote:
>  > On Thu, May 22, 2014 at 11:26:07AM -0400, Matthew Miller wrote:
>  > > See <https://fedorahosted.org/rel-eng/ticket/5886>;. In short, the
>  > > time to do an updates compose and push (plus sync times to
>  > > mirrors) severely limits our ability to put out critical updates
>  > > quickly. Would anyone be interested in filling out a plan for an
>  > > alternate repository which would use a special expedited process
>  > > to make ultra-critical updates available more quickly?
>  > 
>  > I dislike the idea of a separate repo for ultra-critical updates.
>  > Once a fix is available for a vulnerability it should, IMO, be
>  > shipped as soon as possible.  I know this doesn't fit into the
>  > Microsoft model or our model of community testing but really as
>  > soon as you go public with a fix you've also just notified all the
>  > "bad guys" out there to the vulnerability and exactly how to
>  > exploit it.  It's a race condition at that point.
>  
>  Except the severity of various vulnerabilities is very different. And
>  the number of the updates that fix serious vulnerabilities is much
>  lower than the number of the remaining updates (including
>  non-security ones).
>  
Really we have only needed this once in 10 years. Doesn't mean we
should not try to fix the issues.

>  > I'd much prefer to have a mechanism in place that allows
> these fixes
>  > to be pushed to the repos almost immediately (once they've been
>  > properly tested).  I'm not exactly sure how this can work but
>  > perhaps having QE tested patches packaged and ready for the embargo
>  > time would meet Release Engineering's criteria for testing?
>  
>  You have to understand that the repoclosure process takes some time
>  and the metadata on the updates repository is much larger. This
>  affects work of release engineers and it basically prevents the
>  mirrors to sync such large repository too often.
>  
I am not trying to be rude here but your statement clearly shows you
have no idea how the push process works. There is no use of repoclosure
in the push process. how it works is as follows:

* we tell bodhi we want to push, it gives us a list of builds and says
  do you want to . to which we say no.
* We sign all updates in the push process.
* we tell bodhi we want to push, but say yes do it.
* Resign the updates to make sure we catch newer updates.
* bodhi updates each update adding comments that update is being pushed
* bodhi tags the packages in koji as needed.
* bodhi mashes the tags for updates and updates-testing for each release
  - mashing takes the longest period of time. creating deltarpms is one
    of the most resource intensive tasks.
* bodhi updates the updateinfo.xml files
* bodhi updates some symlinks so that the script that syncs the updates
  to the master mirror can sync
* once bodhi detects that the updates have hit the master mirror it
  then updates bugzilla for each update



Matt's idea was to have a repo enabled by default that we can tag builds
into a special tag, mash and sync out so that we could get an update
like the openssl one out in a matter of minutes to the mirrors rather
than hours.  we had a netapp upgrade that has sped up the updates
process but its still time consuming and quite fragile.  Things
frequently go wrong on the bag end of bodhi

Dennis
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.22 (GNU/Linux)
>
> iQIcBAEBAgAGBQJTgKOPAAoJEH7ltONmPFDRBboQANDpZogDz2EsZ/3gRw4F0dEY
> yMuFzy08kf3qOQYw6DZ+iDAg93XGneBClm2W6fOCvI5Ms8xaKo8qyinYc6fPrn+x
> m3VEFCBIekJAEmM0qHpODnGI79BQ/gAjtckymTk6CjoRu9755ZaFxEHdML9fEUJL
> 7kDnrobRUU3v1Pz0WZc/2rafyrHQohinpL7YqnoSwpFyoNjWYEKaGnIxg2cWopUg
> nKsplBh/dchNiMkDsct891K15Yfu48uMuuWGAi9TY7qiW/bDjFpwQm3ilX6LoaMo
> oZzz8M8PeVPJ5EBiIEQYwUGXbK5FYSW43R2622vXkbS26nPCI6TZpDeaPQwtMM2e
> JzpMd2vsoGD/GQMuv4Ax4tzsyKS1B4wT4v0bT10K8O6ZAmSK/hZBKmjvqrkAinmf
> 1QRpsn8PTLXOFoNEZyvJHPoHJtK4Yeo+8CAzPnEgJ0+opjHndrAn8p9KvnMP18yn
> 93KXWE28JdXcFc7DCoB0X+Sy3bJ6XQurIpC+eVrJgPuXKLdzHyjjyqBIBUWmFGjV
> etcGw8oVx8STQ8D9i8DLSd9Yer8Yzns+w1c/VH6+6FVx+UuZ0KQ6UkVC/JOg10Cj
> hjeN5huTYaB7D4buufD7oIrsKZ6GkXYTitIsPtoTn/7IAlcXw+d85dia7VgBPDVy
> v1G5If4A8Rl/Snd57aG1
> =O5eS
> -----END PGP SIGNATURE-----
>

[Kevin Fenzi]

On Fri, 23 May 2014 11:36:40 -0400
Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:

>  On Fri, May 23, 2014 at 11:13:05AM -0400, Eric H. Christensen wrote:
>  > > I'm not sure I follow here. What do you dislike? This isn't meant
>  > > to be a hidden repo -- it's the "ship as soon as possible!"
> repo,
>  > > so it sounds like you're agreeing.
>  > I guess I don't understand the need for the extra repo.  Why not
>  > just push it to fedora-updates?
>  
>  That takes a really long time. Something like 7 hours, best case.
>  Improving that would help, but even then having a separate repo might
>  introduce the possibilities of mirror sites refreshing just that
>  small set more often. 
I'll note that I checked the last updates push yesterday... it was
about 6.5 hours from me starting signing to all pushed for
f19/f20/updates/updates-testing. 

I expect parallel drpm creation in createrepo will help that some. 

If there was an urgent update we wanted to go stable, we could just
push stable and that could be faster. 

Just some more datapoints. 

kevin

[Pavel Kankovsky]

On Fri, 23 May 2014, Matthew Miller wrote:

>  > Why not just push it to fedora-updates?
>  That takes a really long time. Something like 7 hours, best case. 
Perhaps some kind of interim hot patching that'd obviate the need to
develop a clean fix, rebuild the whole package, test it, distribute it,
install it, and restart affected services (if not the whole system) would
be a better way to address such 0day-in-the-wild-the-sky-is-falling-
patch-now-or-die situations... until a regular update is ready.

-- 
Pavel Kankovsky aka Peak                      "Que sais-je?"