3:38 p.m.
Hi all,
within the whole Meltdown and Spectre story I realized that Koji pushes
even security updates to batched only, not directly to stable. In
concrete case we have the firefox update [1], which already received 10
positive karma and many users complaining that it takes so long to get
it out as a stable update. Shouldn't we change that behaviour to get
such security updates out fast when maintainer relies on autopush when
karma threshold is reached?
Greetings,
Christian
[1] https://bodhi.fedoraproject.org/updates/FEDORA-2018-276558ff6f
6:36 p.m.
On 01/07/2018 01:38 PM, Christian Dersch wrote:
Hi all,
within the whole Meltdown and Spectre story I realized that Koji pushes
even security updates to batched only, not directly to stable. In
The critera for bypassing batched is if the update is marked "urgent".
concrete case we have the firefox update [1], which already received
10
positive karma and many users complaining that it takes so long to get
it out as a stable update. Shouldn't we change that behaviour to get
such security updates out fast when maintainer relies on autopush when
karma threshold is reached?
If they are urgent sure... perhaps this should have been so marked?
Or the maintainer/submittor can request stable anytime after it has
enough karma.
Anyhow, I have pushed it to request stable and will do another f27 push
after the current ones finish to get it out today.
kevin
10:01 p.m.
Kevin Fenzi wrote:
The critera for bypassing batched is if the update is marked
"urgent".
The problem is, this appears to be insufficient.
I really don't understand why we do this "batched" thing to begin with.
Users who want to batch updates have always been able to do it, GNOME
Software will even do it for them. Now, those users who want to batch their
updates are forced to follow Fedora rel-eng's schedule for the batches
rather than being able to pick their own weekday, so how does the server-
side batching help them? And those users (like me) who want to get their
updates, including security fixes (!) as we see here, as soon as they passed
testing are now screwed.
If people insist on that "batched" misfeature, can we please at least get a
fast track repository that contains all the batched updates (but no updates
that are still in testing and have not been batched yet!)?
Kevin Kofler
11:39 a.m.
On 01/07/2018 08:01 PM, Kevin Kofler wrote:
Kevin Fenzi wrote:
> The critera for bypassing batched is if the update is marked "urgent".
The problem is, this appears to be insufficient.
Well, if this firefox update was
urgent, shouldn't it have been marked
urgent?
I really don't understand why we do this "batched"
thing to begin with.
To reduce the constant flow of updates that are very minor or affect
very few mixed in with the major updates that affect lots of people and
are urgent. To save users downloads of repodata.
Users who want to batch updates have always been able to do it, GNOME
Software will even do it for theNow, those users who want to batch their
updates are forced to follow Fedora rel-eng's schedule for the batches
rather than being able to pick their own weekday, so how does the server-
side batching help them? And those users (like me) who want to get their
updates, including security fixes (!) as we see here, as soon as they passed
testing are now screwed.
rel-eng's schedule is a cron job at 03:00 on tuesday (so the batch
appears on wed's pushes).
There was some discussion about changing the gnome batching based on the
Fedora batching, but I don't know whats happened there.
I haven't seen a bunch of urgent updates get blocked by this process.
Do you have more data for updates that hit this?
If people insist on that "batched" misfeature, can we
please at least get a
fast track repository that contains all the batched updates (but no updates
that are still in testing and have not been batched yet!)?
I would be very much against additional repos like this.
kevin
12:53 p.m.
Kevin Fenzi wrote:
Well, if this firefox update was urgent, shouldn't it have been
marked
urgent?
Urgency is always in the eye of the beholder. I as a user consider all
security updates "urgent", and in addition, I want ALL updates as soon as
they passed testing no matter whether they actually are urgent.
> I really don't understand why we do this "batched"
thing to begin with.
To reduce the constant flow of updates that are very minor or affect
very few mixed in with the major updates that affect lots of people and
are urgent.
But the users were already able to opt to update only weekly. So why force a
fixed schedule on them?
To save users downloads of repodata.
This does not work in practice because there is almost always at least one
urgent update that requires downloading the whole repodata. (And also
because maintainers are free to skip batched without giving a reason. I
always do this because I consider batching a disservice to my users.)
rel-eng's schedule is a cron job at 03:00 on tuesday (so the
batch
appears on wed's pushes).
There was some discussion about changing the gnome batching based on the
Fedora batching, but I don't know whats happened there.
But what if this is a company that wants to do weekly updates on Monday
morning in order to be free from disruptions for the working week? Then they
will get the updates up to almost 2 weeks late rather than up to 1 week as
both you and they intend.
Batching really needs to be left to the client side!
I haven't seen a bunch of urgent updates get blocked by this
process.
Do you have more data for updates that hit this?
I have empirically seen security updates end up in the batches, but I have
not checked each of them to see whether it actually went through "batched"
or just happened to go out on that day.
But again, I think it is essential for users to be able to opt to getting
updates without arbitrary delays.
> If people insist on that "batched" misfeature, can we
please at least get
> a fast track repository that contains all the batched updates (but no
> updates that are still in testing and have not been batched yet!)?
I would be very much against additional repos like this.
Why? It would allow you to keep the server-side batching while still
allowing those users like me to opt out of it. And the repodata download
size for fast track would be minimal if updates that went out to stable get
removed from fast track.
Even RHEL has a fast track repository.
Kevin Kofler
7:46 p.m.
On Mon, 08 Jan 2018 19:53:01 +0100
Kevin Kofler <kevin.kofler(a)chello.at> wrote:
Batching really needs to be left to the client side!
Right.
I did wonder what was going on with the updates...
dnf-cron, gnome-software etc could all *default* to weekly.
But if I'm doing a dnf update, I want the updates *now*.
11:59 a.m.
On 01/08/2018 10:53 AM, Kevin Kofler wrote:
Kevin Fenzi wrote:
> Well, if this firefox update was urgent, shouldn't it have been marked
> urgent?
Urgency is always in the eye of the beholder. I as a user consider all
security updates "urgent", and in addition, I want ALL updates as soon as
they passed testing no matter whether they actually are urgent.
You also don't want updates-testing to even exist right?
>> I really don't understand why we do this "batched" thing to begin
with.
>
> To reduce the constant flow of updates that are very minor or affect
> very few mixed in with the major updates that affect lots of people and
> are urgent.
But the users were already able to opt to update only weekly. So why force a
fixed schedule on them?
To save all the Fedora users in the world from having to update metadata
for minor changes. Since there's a hourly dnf makecache every user in
the world pulls down new metadata ever time we update a repo. If we
update a repo for some minor enhancements it means everyone in the world
has to pay for that. If we just push all those out every tuesday and
don't update those unless there's something urgent we save everyone a
lot of bandwith and us computing time/resources.
> To save users downloads of repodata.
This does not work in practice because there is almost always at least one
urgent update that requires downloading the whole repodata. (And also
because maintainers are free to skip batched without giving a reason. I
always do this because I consider batching a disservice to my users.)
There are definitely more days when there are no updates for a
particular repo now. Of course there would be even more if you (or those
who do likewise) wouldn't skip batched, but probibly we need to explain
why more clearly.
...snip...
> I would be very much against additional repos like this.
Why? It would allow you to keep the server-side batching while still
allowing those users like me to opt out of it. And the repodata download
size for fast track would be minimal if updates that went out to stable get
removed from fast track.
because it would be a ton more infrastructure and resources.
kevin
2:57 p.m.
Kevin Fenzi wrote:
You also don't want updates-testing to even exist right?
That is not true. I want to leave the decision whether and for how long an
update needs to be tested to the package maintainer instead of enforcing
minimum testing requirements in the software, because the software can never
understand the exact context. Removing updates-testing entirely is not what
I want! But this is unrelated to the current issue of artificially delaying
updates that satisfy all the criteria for being pushed to stable.
To save all the Fedora users in the world from having to update
metadata
for minor changes. Since there's a hourly dnf makecache every user in
the world pulls down new metadata ever time we update a repo.
So to save people the download, you make a change that totally defeats the
point of dnf checking for updates every hour to begin with?
If we update a repo for some minor enhancements it means everyone in
the
world has to pay for that. If we just push all those out every tuesday and
don't update those unless there's something urgent we save everyone a
lot of bandwith and us computing time/resources.
This does not work in practice because there are always updates that are not
batched.
There are definitely more days when there are no updates for a
particular repo now. Of course there would be even more if you (or those
who do likewise) wouldn't skip batched, but probibly we need to explain
why more clearly.
Are there really? The last couple days, there were basically daily pushes
with around 2 updates each.
The batching only makes the daily pushes smaller and not empty, which does
not help at all for reducing repodata download size, because there are still
no repodata deltas implemented.
because it would be a ton more infrastructure and resources.
Really? Bodhi composes (or triggers the compose of, let's please not discuss
the technical details down to that level) 2 repositories per release at each
push, of which one (updates-testing) already works pretty much the way the
fast track repository would work (updates transit there, but leave again
when they reach the next level). How would adding a third level require a
ton more resources? It would just need some small changes to the Bodhi
implementation ("pushes to batched" would have to be actually pushed, to the
fast track repository) and could otherwise use all the existing mirroring
infrastructure. And users would be able to opt in to getting stable updates
without batching. And even if those users keep the regular repodata
downloads enabled, the downloads for fast track would still be much smaller
than the repodata downloads for all of stable. And having fast track
available would also reduce the proportion of updates that skip batched. (I
know I would think twice about skipping batched for my updates if I knew
that the users have a way to opt out of the batching. Right now, I don't
even consider using batched.)
I see only advantages of such an approach, for minimal infrastructure costs.
Almost everything you need is already there!
Kevin Kofler
6:01 a.m.
January 9, 2018 9:59 PM, "Kevin Kofler" <kevin.kofler(a)chello.at> wrote:
Kevin Fenzi wrote:
> To save all the Fedora users in the world from having to update metadata
> for minor changes. Since there's a hourly dnf makecache every user in
> the world pulls down new metadata ever time we update a repo.
So to save people the download, you make a change that totally defeats the
point of dnf checking for updates every hour to begin with?
> If we update a repo for some minor enhancements it means everyone in the
> world has to pay for that. If we just push all those out every tuesday and
> don't update those unless there's something urgent we save everyone a
> lot of bandwith and us computing time/resources.
This does not work in practice because there are always updates that are not
batched.
> There are definitely more days when there are no updates for a
> particular repo now. Of course there would be even more if you (or those
> who do likewise) wouldn't skip batched, but probibly we need to explain
> why more clearly.
Are there really? The last couple days, there were basically daily pushes
with around 2 updates each.
So, if there ARE updates to be pushed (marked urgent?) and we update metadata
ANYWAY, this is a good point to flush everything from batched in this push.
10:12 a.m.
On 01/10/2018 04:01 AM, Tomasz Torcz wrote:
So, if there ARE updates to be pushed (marked urgent?) and we update metadata
ANYWAY, this is a good point to flush everything from batched in this push.
We could yeah. We could also only ever push when there are urgent updates...
kevin
10:05 a.m.
On 01/09/2018 12:57 PM, Kevin Kofler wrote:
Kevin Fenzi wrote:
> You also don't want updates-testing to even exist right?
That is not true. I want to leave the decision whether and for how long an
update needs to be tested to the package maintainer instead of enforcing
minimum testing requirements in the software, because the software can never
understand the exact context. Removing updates-testing entirely is not what
I want! But this is unrelated to the current issue of artificially delaying
updates that satisfy all the criteria for being pushed to stable.
Agreed. Thanks for clarifying.
> To save all the Fedora users in the world from having to update
metadata
> for minor changes. Since there's a hourly dnf makecache every user in
> the world pulls down new metadata ever time we update a repo.
So to save people the download, you make a change that totally defeats the
point of dnf checking for updates every hour to begin with?
It doesn't do that though. dnf wants the latest metadata so it can let
users use that cache for things like searching for packages or listing
them or the like.
> If we update a repo for some minor enhancements it means everyone in the
> world has to pay for that. If we just push all those out every tuesday and
> don't update those unless there's something urgent we save everyone a
> lot of bandwith and us computing time/resources.
This does not work in practice because there are always updates that are not
batched.
I... have seen updates pushes that do not take place when I have been
pushing updates, so I assert you are incorrect. True, it doesn't happen
as often as I was hoping, but it does and has happened.
> There are definitely more days when there are no updates for a
> particular repo now. Of course there would be even more if you (or those
> who do likewise) wouldn't skip batched, but probibly we need to explain
> why more clearly.
Are there really? The last couple days, there were basically daily pushes
with around 2 updates each.
At least one of those cases was me pushing firefox, right after a
f27-updates push just finished, so yeah, it only had 2 updates in it.
The batching only makes the daily pushes smaller and not empty, which does
not help at all for reducing repodata download size, because there are still
no repodata deltas implemented.
> because it would be a ton more infrastructure and resources.
Really? Bodhi composes (or triggers the compose of, let's please not discuss
the technical details down to that level)
...snip...
but the technical details are what matters here. ;) Making another repo,
building drpms, and doing all the compose will take time, disk space and
cpu cycles and slow all other updates pushes down.
Anyhow, I don't want this to be a back and forth between just us, I'd
like to hear some other opinions and proposals and have FESCo decide on
some adjustment ehre.
I agree the current setup is not ideal and personally I'm open to
adjusting/changing it, but I don't know that I think we should just drop
batched entirely. We should come up with something that balances all the
various needs (you want updates as soon as they are tested, I want not
to update metadata on people all the time, etc).
kevin
7:30 a.m.
Kevin Fenzi wrote:
On 01/09/2018 12:57 PM, Kevin Kofler wrote:
> Kevin Fenzi wrote:
>> If we update a repo for some minor enhancements it means everyone in the
>> world has to pay for that. If we just push all those out every tuesday
>> and don't update those unless there's something urgent we save everyone
>> a lot of bandwith and us computing time/resources.
>
> This does not work in practice because there are always updates that are
> not batched.
I... have seen updates pushes that do not take place when I have been
pushing updates, so I assert you are incorrect. True, it doesn't happen
as often as I was hoping, but it does and has happened.
This week, there have been almost daily nonempty update pushes (listing only
the SRPMs here, and only the updates that affected me):
* Jan 10 (previous batch)
* Jan 11 (not batched, gtk3 and microcode-ctl)
* Jan 12 (not batched, kernel, dhcp, dnfdragora, hplip, webkitgtk4)
* none on Sat Jan 13 (some updates from RPM Fusion, but those have separate
repodata, so they don't count)
* Jan 14/15 (night, not batched, bluez, python-nbconvert, python-qtconsole)
* Jan 15/16 (night, not batched, llvm with all its revdeps, hplip again)
and now today (Jan 17)'s batch includes (for me) 135 (!) binary packages
(less if you count the SRPMs, but in the end, the binary packages are what
really matter).
I don't see how this is helpful.
Kevin Kofler
7:47 a.m.
Thanks, Kevin. Knowing when the updates are actually going out adds
important context to this discussion.
On Wed, Jan 17, 2018 at 7:30 AM, Kevin Kofler <kevin.kofler(a)chello.at>
wrote:
I don't see how this is helpful.
Kevin has a point here. Clearly what we have now is, in practice, not
working as intended.
Michael
8:37 a.m.
On 01/17/2018 08:47 AM, mcatanzaro(a)gnome.org wrote:
Clearly what we have now is, in practice, not working as intended.
The original intent as I understood it from the thread long ago[0] was
to reduce the number of updates that go out on non-Tuesdays, and make
most updates happen on Tuesdays. The data that Kevin cited seems to be
accomplishing that purpose.
[0]
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
9:13 a.m.
Randy Barlow wrote:
The original intent as I understood it from the thread long ago[0]
was
to reduce the number of updates that go out on non-Tuesdays, and make
most updates happen on Tuesdays. The data that Kevin cited seems to be
accomplishing that purpose.
But whom does this help? There are still updates going out daily, the
repodata download cost is still there, the notifications too if you aren't
doing client-side batching (and if you are, you don't need server-side
batching to begin with).
The only difference I see is that some fixes take longer to reach the users
and that the huge Tuesday (Wednesday morning for most users) pushes are a
huge pain to go through if you want to actually check what you are updating.
(Am I the only one who bothers reading update notes?)
Kevin Kofler
2:39 p.m.
On 01/18/2018 10:13 AM, Kevin Kofler wrote:
But whom does this help? There are still updates going out daily, the
repodata download cost is still there, the notifications too if you aren't
doing client-side batching (and if you are, you don't need server-side
batching to begin with).
It would help people with more minimal package sets more often. On days
with just a handful of updates, it's possible that some users don't use
any of them and thus don't have to get notified.
There is some discussion around possibly forming a formal policy around
batching updates[0]. If we did batch updates more then users would more
often get days when they don't need to download metadata, which would be
nice.
[0] https://pagure.io/fesco/issue/1820
6:24 a.m.
Randy Barlow wrote:
It would help people with more minimal package sets more often. On
days
with just a handful of updates, it's possible that some users don't use
any of them and thus don't have to get notified.
Even if they don't get notified, they still have to download the whole
metadata in the background.
Kevin Kofler
9:09 a.m.
I wrote:
This week, there have been almost daily nonempty update pushes
(listing
only the SRPMs here, and only the updates that affected me):
* Jan 10 (previous batch)
* Jan 11 (not batched, gtk3 and microcode-ctl)
* Jan 12 (not batched, kernel, dhcp, dnfdragora, hplip, webkitgtk4)
* none on Sat Jan 13 (some updates from RPM Fusion, but those have
separate repodata, so they don't count)
* Jan 14/15 (night, not batched, bluez, python-nbconvert,
python-qtconsole)
* Jan 15/16 (night, not batched, llvm with all its revdeps, hplip again)
and now today (Jan 17)'s batch includes (for me) 135 (!) binary packages
(less if you count the SRPMs, but in the end, the binary packages are what
really matter).
And today (Jan 18), the day after the batch, there's yet another nonempty
push: redhat-rpm-config/nim-srpm-macros, cmake, jnr-*, python-sphinx,
twolame moved from RPM Fusion to Fedora.
Kevin Kofler
4:09 p.m.
Kevin Fenzi <kevin(a)scrye.com> wrote:
[…]
>>> I really don't understand why we do this
"batched" thing to begin with.
>> To reduce the constant flow of updates that are very minor or
affect
>> very few mixed in with the major updates that affect lots of people and
>> are urgent.
> But the users were already able to opt to update only weekly. So
why force a
> fixed schedule on them?
To save all the Fedora users in the world from having to update
metadata
for minor changes. Since there's a hourly dnf makecache every user in
the world pulls down new metadata ever time we update a repo. If we
update a repo for some minor enhancements it means everyone in the world
has to pay for that. If we just push all those out every tuesday and
don't update those unless there's something urgent we save everyone a
lot of bandwith and us computing time/resources.
[…]
BTW, are there technical reasons why the metadata is updated
en bloc and not incrementally like for example delta RPMs,
or is it just that nobody bothered to implement something
like that yet for metadata?
Tim
10:06 a.m.
On 01/09/2018 02:09 PM, Tim Landscheidt wrote:
Kevin Fenzi <kevin(a)scrye.com> wrote:
> […]
>>>> I really don't understand why we do this "batched" thing to
begin with.
>>> To reduce the constant flow of updates that are very minor or affect
>>> very few mixed in with the major updates that affect lots of people and
>>> are urgent.
>> But the users were already able to opt to update only weekly. So why force a
>> fixed schedule on them?
> To save all the Fedora users in the world from having to update metadata
> for minor changes. Since there's a hourly dnf makecache every user in
> the world pulls down new metadata ever time we update a repo. If we
> update a repo for some minor enhancements it means everyone in the world
> has to pay for that. If we just push all those out every tuesday and
> don't update those unless there's something urgent we save everyone a
> lot of bandwith and us computing time/resources.
> […]
BTW, are there technical reasons why the metadata is updated
en bloc and not incrementally like for example delta RPMs,
or is it just that nobody bothered to implement something
like that yet for metadata?
There is no implementation that I know of. It's been talked about for
many years, but not been implemented.
kevin
9:54 a.m.
On Wed, 2018-01-10 at 08:06 -0800, Kevin Fenzi wrote:
On 01/09/2018 02:09 PM, Tim Landscheidt wrote:
> BTW, are there technical reasons why the metadata is updated
> en bloc and not incrementally like for example delta RPMs,
> or is it just that nobody bothered to implement something
> like that yet for metadata?
There is no implementation that I know of. It's been talked about for
many years, but not been implemented.
We talked a bit about using casync for delta metadata at Flock last
summer, and I even did a small proof of concept to see what kind of
savings we'd see, but there were a couple of issues that came up:
* casync creates a large number of very small files (roughly 4k files
for yesterday's metadata) that mirrors would need to sync
* Users would need to download a significant portion of those files to
delta against (anywhere from 0 to all of them, depending on how
similar their metadata is)
Downloading any significant number of tiny files over http/1.x is
really, really slow as the downloads are serial.
On reflection, one idea that might fix both problems would be if casync
could concatenate the cacnk files into one large block (cablk,
perhaps), and have the caidx file contain the location of each cacnk in
the block file. Using http ranges, you would then be able to download
just the parts of the file you wanted (though I have not actually
tested whether specifying a large list of http ranges will download
faster than serially downloading each individual file).
FWIW, casyncing Jan 4th's metadata over Dec 15th's downloads 9.6MB (in
just over 1k cacnk files), while downloading the same metadata directly
would total 16MB. More frequent updates would be smaller, but I'm not
convinced the difference from an average updates push to another would
be less than 2MB.
Jonathan
2:37 p.m.
On 01/09/2018 02:09 PM, Tim Landscheidt wrote:
BTW, are there technical reasons why the metadata is updated
en bloc and not incrementally like for example delta RPMs,
or is it just that nobody bothered to implement something
like that yet for metadata?
This has been discussed for a very long time:
https://bugzilla.redhat.com/show_bug.cgi?id=850896
1:23 p.m.
On Jan 9, 2018, at 9:59 AM, Kevin Fenzi <kevin(a)scrye.com>
wrote:
> On 01/08/2018 10:53 AM, Kevin Kofler wrote:
> Kevin Fenzi wrote:
>> Well, if this firefox update was urgent, shouldn't it have been marked
>> urgent?
>
> Urgency is always in the eye of the beholder. I as a user consider all
> security updates "urgent", and in addition, I want ALL updates as soon as
> they passed testing no matter whether they actually are urgent.
You also don't want updates-testing to even exist right?
>
>>> I really don't understand why we do this "batched" thing to
begin with.
>>
>> To reduce the constant flow of updates that are very minor or affect
>> very few mixed in with the major updates that affect lots of people and
>> are urgent.
>
> But the users were already able to opt to update only weekly. So why force a
> fixed schedule on them?
To save all the Fedora users in the world from having to update metadata
for minor changes. Since there's a hourly dnf makecache every user in
the world pulls down new metadata ever time we update a repo.
Could Fedora, perhaps, come up with a way to make incremental metadata
updates fast? This shouldn't be particularly hard -- a tool like
casync or even svn should work pretty well. Or it could be a simple
ad-hoc thing. Have the mirrors serve up both the whole metadata blob
and a list of metadata changes. The client could grab the list of
changes from last time, compute a bitwise-identical blob, and verify
the signature on that, deltarpm style. (But on the decompressed data,
of course -- no need to repeat the mistakes of the past.)
It seems that all this batched stuff is a rather weak hack to work
around the extreme inefficiency of Fedora's metadata distribution
model.
1:38 p.m.
On 10 January 2018 at 14:23, Andrew Lutomirski <luto(a)mit.edu> wrote:
> On Jan 9, 2018, at 9:59 AM, Kevin Fenzi <kevin(a)scrye.com>
wrote:
>
>> On 01/08/2018 10:53 AM, Kevin Kofler wrote:
>> Kevin Fenzi wrote:
>>> Well, if this firefox update was urgent, shouldn't it have been marked
>>> urgent?
>>
>> Urgency is always in the eye of the beholder. I as a user consider all
>> security updates "urgent", and in addition, I want ALL updates as soon
as
>> they passed testing no matter whether they actually are urgent.
>
> You also don't want updates-testing to even exist right?
>
>>
>>>> I really don't understand why we do this "batched" thing to
begin with.
>>>
>>> To reduce the constant flow of updates that are very minor or affect
>>> very few mixed in with the major updates that affect lots of people and
>>> are urgent.
>>
>> But the users were already able to opt to update only weekly. So why force a
>> fixed schedule on them?
>
> To save all the Fedora users in the world from having to update metadata
> for minor changes. Since there's a hourly dnf makecache every user in
> the world pulls down new metadata ever time we update a repo.
Could Fedora, perhaps, come up with a way to make incremental metadata
updates fast? This shouldn't be particularly hard -- a tool like
casync or even svn should work pretty well. Or it could be a simple
This sounds a lot like the Atomic project and how it does things...
ad-hoc thing. Have the mirrors serve up both the whole metadata
blob
and a list of metadata changes. The client could grab the list of
changes from last time, compute a bitwise-identical blob, and verify
the signature on that, deltarpm style. (But on the decompressed data,
of course -- no need to repeat the mistakes of the past.)
It seems that all this batched stuff is a rather weak hack to work
around the extreme inefficiency of Fedora's metadata distribution
model.
All things are a weak hack to work around some inefficiency and an
extremely important process to deal with needs. Trying to label
something without taking into consideration of the other is where most
arguments go pear shaped.
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
--
Stephen J Smoogen.
1:46 p.m.
On Wed, Jan 10, 2018 at 11:38 AM, Stephen John Smoogen <smooge(a)gmail.com>
wrote:
On 10 January 2018 at 14:23, Andrew Lutomirski <luto(a)mit.edu>
wrote:
>> On Jan 9, 2018, at 9:59 AM, Kevin Fenzi <kevin(a)scrye.com> wrote:
>>
>>> On 01/08/2018 10:53 AM, Kevin Kofler wrote:
>>> Kevin Fenzi wrote:
>>>> Well, if this firefox update was urgent, shouldn't it have been
marked
>>>> urgent?
>>>
>>> Urgency is always in the eye of the beholder. I as a user consider all
>>> security updates "urgent", and in addition, I want ALL updates as
soon
as
>>> they passed testing no matter whether they actually are urgent.
>>
>> You also don't want updates-testing to even exist right?
>>
>>>
>>>>> I really don't understand why we do this "batched"
thing to begin
with.
>>>>
>>>> To reduce the constant flow of updates that are very minor or affect
>>>> very few mixed in with the major updates that affect lots of people
and
>>>> are urgent.
>>>
>>> But the users were already able to opt to update only weekly. So why
force a
>>> fixed schedule on them?
>>
>> To save all the Fedora users in the world from having to update metadata
>> for minor changes. Since there's a hourly dnf makecache every user in
>> the world pulls down new metadata ever time we update a repo.
>
> Could Fedora, perhaps, come up with a way to make incremental metadata
> updates fast? This shouldn't be particularly hard -- a tool like
> casync or even svn should work pretty well. Or it could be a simple
This sounds a lot like the Atomic project and how it does things...
Maybe some of Atomic's infrastructure could be used to distribute metadata
for regular old Fedora.
--Andy
2:01 p.m.
On 10 January 2018 at 14:46, Andrew Lutomirski <luto(a)mit.edu> wrote:
On Wed, Jan 10, 2018 at 11:38 AM, Stephen John Smoogen <smooge(a)gmail.com>
wrote:
>
> On 10 January 2018 at 14:23, Andrew Lutomirski <luto(a)mit.edu> wrote:
> >> On Jan 9, 2018, at 9:59 AM, Kevin Fenzi <kevin(a)scrye.com> wrote:
> >>
> >>> On 01/08/2018 10:53 AM, Kevin Kofler wrote:
> >>> Kevin Fenzi wrote:
> >>>> Well, if this firefox update was urgent, shouldn't it have been
> >>>> marked
> >>>> urgent?
> >>>
> >>> Urgency is always in the eye of the beholder. I as a user consider all
> >>> security updates "urgent", and in addition, I want ALL updates
as soon
> >>> as
> >>> they passed testing no matter whether they actually are urgent.
> >>
> >> You also don't want updates-testing to even exist right?
> >>
> >>>
> >>>>> I really don't understand why we do this "batched"
thing to begin
> >>>>> with.
> >>>>
> >>>> To reduce the constant flow of updates that are very minor or
affect
> >>>> very few mixed in with the major updates that affect lots of people
> >>>> and
> >>>> are urgent.
> >>>
> >>> But the users were already able to opt to update only weekly. So why
> >>> force a
> >>> fixed schedule on them?
> >>
> >> To save all the Fedora users in the world from having to update
> >> metadata
> >> for minor changes. Since there's a hourly dnf makecache every user in
> >> the world pulls down new metadata ever time we update a repo.
> >
> > Could Fedora, perhaps, come up with a way to make incremental metadata
> > updates fast? This shouldn't be particularly hard -- a tool like
> > casync or even svn should work pretty well. Or it could be a simple
>
> This sounds a lot like the Atomic project and how it does things...
>
Maybe some of Atomic's infrastructure could be used to distribute metadata
for regular old Fedora.
OK clearly I should not try to help with a single sentence email as it
didn't help anyone. I should have asked more questions on what you
meant by metadata and what you meant by distribution and how svn would
have been used instead. I should have also asked if you knew enough
about how atomic did things to give an informed opinion on how that
was similar to the request. Without that we ended up talking past each
other.
--
Stephen J Smoogen.
2:19 p.m.
On Wed, Jan 10, 2018 at 12:01 PM, Stephen John Smoogen <smooge(a)gmail.com> wrote:
On 10 January 2018 at 14:46, Andrew Lutomirski <luto(a)mit.edu> wrote:
>
>
> On Wed, Jan 10, 2018 at 11:38 AM, Stephen John Smoogen <smooge(a)gmail.com>
> wrote:
>>
>> On 10 January 2018 at 14:23, Andrew Lutomirski <luto(a)mit.edu> wrote:
>> >> On Jan 9, 2018, at 9:59 AM, Kevin Fenzi <kevin(a)scrye.com> wrote:
>> >>
>> >>> On 01/08/2018 10:53 AM, Kevin Kofler wrote:
>> >>> Kevin Fenzi wrote:
>> >>>> Well, if this firefox update was urgent, shouldn't it have
been
>> >>>> marked
>> >>>> urgent?
>> >>>
>> >>> Urgency is always in the eye of the beholder. I as a user consider
all
>> >>> security updates "urgent", and in addition, I want ALL
updates as soon
>> >>> as
>> >>> they passed testing no matter whether they actually are urgent.
>> >>
>> >> You also don't want updates-testing to even exist right?
>> >>
>> >>>
>> >>>>> I really don't understand why we do this
"batched" thing to begin
>> >>>>> with.
>> >>>>
>> >>>> To reduce the constant flow of updates that are very minor or
affect
>> >>>> very few mixed in with the major updates that affect lots of
people
>> >>>> and
>> >>>> are urgent.
>> >>>
>> >>> But the users were already able to opt to update only weekly. So
why
>> >>> force a
>> >>> fixed schedule on them?
>> >>
>> >> To save all the Fedora users in the world from having to update
>> >> metadata
>> >> for minor changes. Since there's a hourly dnf makecache every user
in
>> >> the world pulls down new metadata ever time we update a repo.
>> >
>> > Could Fedora, perhaps, come up with a way to make incremental metadata
>> > updates fast? This shouldn't be particularly hard -- a tool like
>> > casync or even svn should work pretty well. Or it could be a simple
>>
>> This sounds a lot like the Atomic project and how it does things...
>>
>
> Maybe some of Atomic's infrastructure could be used to distribute metadata
> for regular old Fedora.
>
OK clearly I should not try to help with a single sentence email as it
didn't help anyone. I should have asked more questions on what you
meant by metadata and what you meant by distribution and how svn would
have been used instead.
SVN is probably a poor model, but imagine that the various metadata
files (repodata, I think) were check in, uncompressed, to an SVN repo.
Then the client could do 'svn up' and take advantage of delta
compression. I suspect the server load would be too high, though.
Basically, what I'm getting at is that Fedora seems to distribute
comps, prestodelta, primary, filelists, and updateinfo, and they add
up to 20MB compressed for the updates repo. AFAICT it gets
re-downlaoded from scratch every time, even though the actual list of
changes is probably minimal from day to day.
I should have also asked if you knew enough
about how atomic did things to give an informed opinion on how that
was similar to the request. Without that we ended up talking past each
other.
I don't know too much about the details. I suspect that, if the
.xml.gz files were, instead, one file per package, then they could be
served up from an ostree repo.
7:41 a.m.
On Wed, Jan 10, 2018, at 2:38 PM, Stephen John Smoogen wrote:
This sounds a lot like the Atomic project and how it does things...
Atomic could mean either (rpm)-ostree or Docker/OCI. In the
Docker/OCI world search isn't standardized yet AIUI but there's
progress on that upstream. I am not aware of the state of things
overall there - it'd be really interesting to have someone who is
clued in comment.
I do ostree and rpm, which is used for Atomic Host (and
the Atomic Workstation). Now we're in
the midst of a deeply fundamental rework:
https://github.com/projectatomic/rpm-ostree/issues/1081
And very specifically on this topic:
https://github.com/projectatomic/rpm-ostree/issues/1127
(That issue includes my thoughts on e.g. ostree vs git for metadata)
That said, if someone is actually planning to *work* on this,
the best place to discuss is http://lists.rpm.org/pipermail/rpm-ecosystem/
as you're more likely to have the relevant people involved.
I do plan to take some of my ideas from the above rpm-ostree
issue to that list, but it's going to come after implementing jigdo.
7:58 a.m.
One followup that should help people understand things:
When someone pushes an update to a package that isn't
in Atomic Host (or Workstation), *and* one is using rpm-ostree
in "pure ostree" mode (i.e. you never ran `rpm-ostree install`),
then checking for updates just uses libostree, which like any
sane image system, makes this a very cheap operation.
In the case of libostree, checking for "no changes" is just a few
HTTP requests for tiny files.
Say for example you've got a bunch of CentOS containers
on your FAH system (IMO this is not just a valid use case but
one I'd encourage) - you don't see an available update whenever
a new Firefox or whatever appears.
We were also doing ~two week batching for FAH updates before
Bodhi started doing batching...the interaction between those
really could be better...that's a whole other topic.
Anyways though to finish the explanation:
the second you do `rpm-ostree install libvirt` for example
(as I have done on my home server), you're doing *both* systems;
now libdnf comes into play, and it's the same RPM metadata from Fedora
with all the same performance penalty. Smoothing out this dramatic cliff of
a transition is part of the reason I'm working on rpm-ostree jigdo.
7:58 a.m.
On Wed, Jan 10, 2018 at 2:23 PM, Andrew Lutomirski <luto(a)mit.edu> wrote:
> On Jan 9, 2018, at 9:59 AM, Kevin Fenzi <kevin(a)scrye.com>
wrote:
>
>> On 01/08/2018 10:53 AM, Kevin Kofler wrote:
>> Kevin Fenzi wrote:
>>> Well, if this firefox update was urgent, shouldn't it have been marked
>>> urgent?
>>
>> Urgency is always in the eye of the beholder. I as a user consider all
>> security updates "urgent", and in addition, I want ALL updates as soon
as
>> they passed testing no matter whether they actually are urgent.
>
> You also don't want updates-testing to even exist right?
>
>>
>>>> I really don't understand why we do this "batched" thing to
begin with.
>>>
>>> To reduce the constant flow of updates that are very minor or affect
>>> very few mixed in with the major updates that affect lots of people and
>>> are urgent.
>>
>> But the users were already able to opt to update only weekly. So why force a
>> fixed schedule on them?
>
> To save all the Fedora users in the world from having to update metadata
> for minor changes. Since there's a hourly dnf makecache every user in
> the world pulls down new metadata ever time we update a repo.
Could Fedora, perhaps, come up with a way to make incremental metadata
updates fast? This shouldn't be particularly hard -- a tool like
casync or even svn should work pretty well. Or it could be a simple
ad-hoc thing. Have the mirrors serve up both the whole metadata blob
and a list of metadata changes. The client could grab the list of
changes from last time, compute a bitwise-identical blob, and verify
the signature on that, deltarpm style. (But on the decompressed data,
of course -- no need to repeat the mistakes of the past.)
It seems that all this batched stuff is a rather weak hack to work
around the extreme inefficiency of Fedora's metadata distribution
model.
This was actually prototyped two years ago with zsync:
https://github.com/rh-lab-q/deltametadata-prototype
It worked, but it was slow since it wasn't implemented in librepo and
of course we never got it implemented in the infrastructure.
Details:
https://github.com/rh-lab-q/deltametadata-prototype/wiki/Deltametadata-of...
The Plan of 2017 that never got done:
https://github.com/rh-lab-q/deltametadata-prototype/wiki/Plan-2017
--
真実はいつも一つ！/ Always, there's only one truth!
8:17 p.m.
On Mon, Jan 8, 2018 at 5:39 PM, Kevin Fenzi <kevin(a)scrye.com> wrote:
On 01/07/2018 08:01 PM, Kevin Kofler wrote:
> Kevin Fenzi wrote:
>> The critera for bypassing batched is if the update is marked "urgent".
>
> The problem is, this appears to be insufficient.
Well, if this firefox update was urgent, shouldn't it have been marked
urgent?
I thought for some reason that all updates marked as security were
automatically urgent, maybe I'm misremembering, but if not it might be
good to do that as a RFE that way all security updates go out non
batched.
>> I really don't understand why we do this "batched" thing to begin
with.
>
> To reduce the constant flow of updates that are very minor or affect
> very few mixed in with the major updates that affect lots of people and
> are urgent. To save users downloads of repodata.
>
>> Users who want to batch updates have always been able to do it, GNOME
>> Software will even do it for theNow, those users who want to batch their
>> updates are forced to follow Fedora rel-eng's schedule for the batches
>> rather than being able to pick their own weekday, so how does the server-
>> side batching help them? And those users (like me) who want to get their
>> updates, including security fixes (!) as we see here, as soon as they passed
>> testing are now screwed.
>
> rel-eng's schedule is a cron job at 03:00 on tuesday (so the batch
> appears on wed's pushes).
>
> There was some discussion about changing the gnome batching based on the
> Fedora batching, but I don't know whats happened there.
>
> I haven't seen a bunch of urgent updates get blocked by this process.
> Do you have more data for updates that hit this?
>
>> If people insist on that "batched" misfeature, can we please at least
get a
>> fast track repository that contains all the batched updates (but no updates
>> that are still in testing and have not been batched yet!)?
>
> I would be very much against additional repos like this.
>
>
> kevin
>
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>
10 p.m.
On Mon, Jan 8, 2018 at 8:17 PM, Peter Robinson <pbrobinson(a)gmail.com>
wrote:
I thought for some reason that all updates marked as security were
automatically urgent, maybe I'm misremembering, but if not it might
be good to do that as a RFE that way all security updates go out non
batched.
Most security updates are simply not urgent, and we really don't want
to pester users with these on a daily basis. So it would be nice if
security updates went through batched by default.
Still, this update was marked as a high-severity security update. It
probably makes sense that those should skip batched, in addition to
updates marked as urgent.
Michael
7:43 a.m.
On Tue, Jan 09, 2018 at 02:17:43AM +0000, Peter Robinson wrote:
I thought for some reason that all updates marked as security were
automatically urgent, maybe I'm misremembering, but if not it might be
good to do that as a RFE that way all security updates go out non
batched.
There was some discussion. There are plenty of updates which have some
security implication but which aren't really urgent. I'd rather them
stay described as non-urgent security updates than people start not
calling them security updates. When they _are_ important to get to
users quickly, they should be marked that way. But that's always a
tradeoff -- more time in testing gives more of a chance to find
regressions or other probems.
Also, I think everyone in this discussion on _this_ list who would like
updates faster should probably be using updates-testing. Or at least
_looking_ at updates-testing. You can always pull individual updates
from there on a per-package basis, and doing this helps everyone else.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader
3:06 p.m.
Matthew Miller wrote:
Also, I think everyone in this discussion on _this_ list who would
like
updates faster should probably be using updates-testing. Or at least
_looking_ at updates-testing. You can always pull individual updates
from there on a per-package basis, and doing this helps everyone else.
I want tested updates faster. I don't want untested updates.
I also don't think it would be a productive use of my time to go through all
testing updates every day to see which ones are batched for stable. This is
something that could easily be automated by software (i.e., by having Bodhi
push the updates to a fast track repository), why should I be doing this by
hand?
What I do normally do is read the update notes of every update that I am
about to apply. This is also a reason why I hate the batching, because the
batches are huge and painful to check through. I prefer more frequent
smaller pushes that I can easily read through. But what I can definitely
tell you is that most updates do not contain enough information in the
update notes to really know their impact, so using that as a basis for
applying or not applying some update from updates-testing is not going to
scale either.
By the way, the reason I did not voice these complaints in the thread
initially announcing this change is that I was both too angry and too busy
at that time to come up with a polite reply, and besides, the change was
already implemented when announced anyway.
Kevin Kofler
12:59 a.m.
> within the whole Meltdown and Spectre story I realized that Koji
pushes
> even security updates to batched only, not directly to stable. In
The critera for bypassing batched is if the update is marked "urgent".
Or once you push it stable and it's queued for batched the maintainer
then has an option to push stable in the same spot as usual, just
needs the maintainer to push the button again.
>> concrete case we have the firefox update [1], which already received 10
>> positive karma and many users complaining that it takes so long to get
>> it out as a stable update. Shouldn't we change that behaviour to get
>> such security updates out fast when maintainer relies on autopush when
>> karma threshold is reached?
>
> If they are urgent sure... perhaps this should have been so marked?
> Or the maintainer/submittor can request stable anytime after it has
> enough karma.
>
> Anyhow, I have pushed it to request stable and will do another f27 push
> after the current ones finish to get it out today.
>
> kevin
>
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>
6:19 a.m.
On 01/08/2018 01:36 AM, Kevin Fenzi wrote:
On 01/07/2018 01:38 PM, Christian Dersch wrote:
> Hi all,
>
> within the whole Meltdown and Spectre story I realized that Koji pushes
> even security updates to batched only, not directly to stable. In
The critera for bypassing batched is if the update is marked "urgent".
> concrete case we have the firefox update [1], which already received 10
> positive karma and many users complaining that it takes so long to get
> it out as a stable update. Shouldn't we change that behaviour to get
> such security updates out fast when maintainer relies on autopush when
> karma threshold is reached?
If they are urgent sure... perhaps this should have been so marked?
Or the maintainer/submittor can request stable anytime after it has
enough karma.
Anyhow, I have pushed it to request stable and will do another f27 push
after the current ones finish to get it out today.
Seems to be my fault then, didn't know that the urgent/high state has
the meaning there.
ma.
2285
days inactive
2311
days old
36 comments
18 participants
participants (18)
-
Andrew Clayton -
Andrew Lutomirski -
Benjamin Kircher -
Christian Dersch -
Colin Walters -
Jonathan Dieter -
Kevin Fenzi -
Kevin Kofler -
Martin Stransky -
Matthew Miller -
mcatanzaro＠gnome.org -
Neal Gompa -
Peter Robinson -
Randy Barlow -
Samuel Sieb -
Stephen John Smoogen -
Tim Landscheidt -
Tomasz Torcz