8:24 p.m.
Hi,
I had my computer hang during a major yum upgrade. Now, package-cleanup
mentions like 50 duplicate packages installed. I had it clean them up. But
now my system does not feel "clean". I am afraid if I reboot it won't come
up. Is there a smart way to verify all system files are intact. rpm -Va
outputs lots of crappy changes that are "normal". Any way to only get a
warning, when something really important has changed ?
Thanks
10:50 p.m.
On Thu, 2008-09-11 at 03:24 +0200, Ahmed Kamal wrote:
Hi,
I had my computer hang during a major yum upgrade.
When this happens you should run:
yum-complete-transaction
at the next possible time to see if it can finish the aborted
transaction.
Now, package-cleanup mentions like 50 duplicate packages installed.
I
had it clean them up. But now my system does not feel "clean". I am
afraid if I reboot it won't come up. Is there a smart way to verify
all system files are intact. rpm -Va outputs lots of crappy changes
that are "normal". Any way to only get a warning, when something
really important has changed ?
Really important is a bit subjective.
I'd recommend package-cleanup --cleandupes
then run:
rpm -Va and look for problems in files in /usr/lib /lib/ /boot, and all
of the bin dirs.
If you don't see any problems in those places then odds are you'll be
okay with a reboot.
-sv
12:42 a.m.
On Thu, Sep 11, 2008 at 3:50 PM, Seth Vidal <skvidal(a)fedoraproject.org> wrote:
When this happens you should run:
yum-complete-transaction
Interesting toy! I think you mentioned it at Fudcon Boston and I
hadn't been able to recall the right name.
Thinking of using it in the use case of the school server (very
unreliable power, no sysadmins available, 100% unattended updates) -
- Is it safe to run at boot time via an init script?
- Is there an easy way to check for pending transactions?
- Does it have useful exit codes indicating whether it's done anything?
I'd recommend package-cleanup --cleandupes
Another good tool to add to the arsenal.
then run:
rpm -Va and look for problems in files in /usr/lib /lib/ /boot, and all
of the bin dirs.
Is that different from `package-cleanup --problems` ?
cheers,
m
--
martin.langhoff(a)gmail.com
martin(a)laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
2:53 a.m.
Trying rpm -Va, I am getting lots of these lines
S.?..... /usr/bin/kblankscrn.kss
S.?..... /usr/bin/kcminit
S.?..... /usr/bin/kcminit_startup
Basically, the size has changed, and the md5 check cannot be performed ?! I
understand this is due to "prelink", but that sux ! This effectively kills
the rpm -V functionality. Is it not possible to prelink binaries on the
server before wrapping them into rpms ? Any suggested solution around this ?
On Thu, Sep 11, 2008 at 7:42 AM, Martin Langhoff
<martin.langhoff(a)gmail.com>wrote:
On Thu, Sep 11, 2008 at 3:50 PM, Seth Vidal
<skvidal(a)fedoraproject.org>
wrote:
> When this happens you should run:
> yum-complete-transaction
Interesting toy! I think you mentioned it at Fudcon Boston and I
hadn't been able to recall the right name.
Thinking of using it in the use case of the school server (very
unreliable power, no sysadmins available, 100% unattended updates) -
- Is it safe to run at boot time via an init script?
- Is there an easy way to check for pending transactions?
- Does it have useful exit codes indicating whether it's done anything?
> I'd recommend package-cleanup --cleandupes
Another good tool to add to the arsenal.
> then run:
> rpm -Va and look for problems in files in /usr/lib /lib/ /boot, and all
> of the bin dirs.
Is that different from `package-cleanup --problems` ?
cheers,
m
--
martin.langhoff(a)gmail.com
martin(a)laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
--
fedora-devel-list mailing list
fedora-devel-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
4:03 a.m.
On Thu, 11 Sep 2008 09:53:02 +0200, Ahmed Kamal wrote:
Trying rpm -Va, I am getting lots of these lines
S.?..... /usr/bin/kblankscrn.kss
S.?..... /usr/bin/kcminit
S.?..... /usr/bin/kcminit_startup
Then run the test as "root".
Basically, the size has changed, and the md5 check cannot be
performed ?! I
understand this is due to "prelink", but that sux ! This effectively kills
the rpm -V functionality. Is it not possible to prelink binaries on the
server before wrapping them into rpms ? Any suggested solution around this ?
rpm -Va is aware of prelinking under normal circumstances.
4:19 a.m.
I am running as root
On Thu, Sep 11, 2008 at 11:03 AM, Michael Schwendt <mschwendt(a)gmail.com>wrote:
On Thu, 11 Sep 2008 09:53:02 +0200, Ahmed Kamal wrote:
> Trying rpm -Va, I am getting lots of these lines
>
> S.?..... /usr/bin/kblankscrn.kss
> S.?..... /usr/bin/kcminit
> S.?..... /usr/bin/kcminit_startup
Then run the test as "root".
> Basically, the size has changed, and the md5 check cannot be performed ?!
I
> understand this is due to "prelink", but that sux ! This effectively
kills
> the rpm -V functionality. Is it not possible to prelink binaries on the
> server before wrapping them into rpms ? Any suggested solution around
this ?
rpm -Va is aware of prelinking under normal circumstances.
--
fedora-devel-list mailing list
fedora-devel-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
10:02 a.m.
On Thu, 2008-09-11 at 09:53 +0200, Ahmed Kamal wrote:
Trying rpm -Va, I am getting lots of these lines
S.?..... /usr/bin/kblankscrn.kss
S.?..... /usr/bin/kcminit
S.?..... /usr/bin/kcminit_startup
Basically, the size has changed, and the md5 check cannot be
performed ?! I understand this is due to "prelink", but that sux !
This effectively kills the rpm -V functionality. Is it not possible to
prelink binaries on the server before wrapping them into rpms ? Any
suggested solution around this ?
You might consider reinstalling kdebase-workspace
yum reinstall kdebase-workspace
see if that works for you.
-sv
10:01 a.m.
On Thu, 2008-09-11 at 17:42 +1200, Martin Langhoff wrote:
On Thu, Sep 11, 2008 at 3:50 PM, Seth Vidal
<skvidal(a)fedoraproject.org> wrote:
> When this happens you should run:
> yum-complete-transaction
Interesting toy! I think you mentioned it at Fudcon Boston and I
hadn't been able to recall the right name.
Thinking of using it in the use case of the school server (very
unreliable power, no sysadmins available, 100% unattended updates) -
- Is it safe to run at boot time via an init script?
as long as the network is up, it should be.
- Is there an easy way to check for pending transactions?
yum-complete-transaction checks for them itself
- Does it have useful exit codes indicating whether it's done
anything?
If the results codes are not good enough we can fix that easily enough.
Is that different from `package-cleanup --problems` ?
--problems reports a different set of things - package conflicts and
unresolved deps. A dupe is not, implicitly, a problem. it CAN be, but it
is not always.
-sv
4:54 p.m.
On Fri, Sep 12, 2008 at 3:01 AM, Seth Vidal <skvidal(a)fedoraproject.org> wrote:
> - Is it safe to run at boot time via an init script?
as long as the network is up, it should be.
That is not very useful for us. I don't worry so much about the
machine being killed in the 'download stuff' part, bit AFAIK that's
not part of the 'transaction'. When yum-complete-transaction is
called, I am expecting it to work with the RPMs it has, and ensure
anything it was trying to do gets done...
WRT networking, in places like Peru my rule of thumb is that 50% of
the servers will _not_ have internet. Package installations / upgrades
are very likely to happen by sending out USB keys with a bunch of
RPMs, and a GPG-signed script that triggers the yum process.
yum supports -C... can yum-complete-transaction support -C as well?
> - Is there an easy way to check for pending transactions?
yum-complete-transaction checks for them itself
Yes, though it's useful to be able to check that for logging/reporting...
> - Does it have useful exit codes indicating whether it's
done anything?
If the results codes are not good enough we can fix that easily enough.
I am asking because checking the manpage and reading the source did
not give me any hints as to exit codes. Neither does 'man yum' talk
about exit codes except for the "check-update" command.
So I wonder what's the expected behaviour WRT exit codes for yum?
> Is that different from `package-cleanup --problems` ?
--problems reports a different set of things - package conflicts and
unresolved deps. A dupe is not, implicitly, a problem. it CAN be, but it
is not always.
Right - that's lower priority for my use cases. Thanks for the hint.
cheers,
m
--
martin.langhoff(a)gmail.com
martin(a)laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
9:20 p.m.
On Fri, 2008-09-12 at 09:54 +1200, Martin Langhoff wrote:
That is not very useful for us. I don't worry so much about the
machine being killed in the 'download stuff' part, bit AFAIK that's
not part of the 'transaction'. When yum-complete-transaction is
called, I am expecting it to work with the RPMs it has, and ensure
anything it was trying to do gets done...
The transaction file is written out after the pkgs are downloaded, so
the transaction should complete just fine if the system hasn't changed
state dramatically.
WRT networking, in places like Peru my rule of thumb is that 50% of
the servers will _not_ have internet. Package installations / upgrades
are very likely to happen by sending out USB keys with a bunch of
RPMs, and a GPG-signed script that triggers the yum process.
Then that won't be an issue at all, you'll have a local repo on the usb
key which yum can access. It won't even need -C b/c it will have
connectivity to the local file:/// based repo.
yum supports -C... can yum-complete-transaction support -C as well?
it already does. - try it: yum-complete-transaction --help
>> - Is there an easy way to check for pending transactions?
>
> yum-complete-transaction checks for them itself
Yes, though it's useful to be able to check that for logging/reporting...
You can check with one call:
import yum.misc
yum.misc.find_unfinished_transactions()
if it returns a non-empty list then you have unfinished transactions.
>> - Does it have useful exit codes indicating whether
it's done anything?
>
> If the results codes are not good enough we can fix that easily enough.
I am asking because checking the manpage and reading the source did
not give me any hints as to exit codes. Neither does 'man yum' talk
about exit codes except for the "check-update" command.
at this point yum should be returning 0 for any non-erroring exit and 1
for any error condition. Including rpm scriptlet failures.
That should also be true for yum-complete-transaction
-sv
5:05 a.m.
On Sep 11, 2008, Seth Vidal <skvidal(a)fedoraproject.org> wrote:
On Thu, 2008-09-11 at 03:24 +0200, Ahmed Kamal wrote:
> Hi,
> I had my computer hang during a major yum upgrade.
When this happens you should run:
yum-complete-transaction
Neat!
In a slightly different scenario: I often find that, if the ssh
connection from which I start 'yum update' is broken for whatever
reason (say the machine from which I started reboots or so), yum gets
an error posting its progress reports, and then it starts removing
lots and lots of packages to keep dependencies from being unmet.
Wouldn't it be much nicer if, like, it ran to completion without
regarding the stdout errors; recorded a recovery transaction to revert
the removals, or at least recorded a transaction to complete the
interrupted update?
I realize the latter is risky, for you may end up unable to start yum
in the first place, but this unfortunately is also the case of the
current code. Last time I got this error, an elfutils update was part
of the transaction, and it stopped at such an unfortunate time that
elfutils got removed, and rpm would no longer run. Oops :-)
Is this bug report material?
--
Alexandre Oliva http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist oliva(a){lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer aoliva(a){redhat.com, gcc.gnu.org}
7:59 a.m.
On Sun, Sep 21, 2008 at 07:05:37 -0300,
Alexandre Oliva <aoliva(a)redhat.com> wrote:
In a slightly different scenario: I often find that, if the ssh
connection from which I start 'yum update' is broken for whatever
reason (say the machine from which I started reboots or so), yum gets
an error posting its progress reports, and then it starts removing
lots and lots of packages to keep dependencies from being unmet.
Use screen. If the connection gets broken you can reconnect to the screen
terminal from a new one. It is also a good idea to do this when running
yum inside an xterm as upgrades occassionally toast the x server.
3:55 p.m.
On Sep 21, 2008, Bruno Wolff III <bruno(a)wolff.to> wrote:
On Sun, Sep 21, 2008 at 07:05:37 -0300,
Alexandre Oliva <aoliva(a)redhat.com> wrote:
> In a slightly different scenario: I often find that, if the ssh
> connection from which I start 'yum update' is broken for whatever
> reason (say the machine from which I started reboots or so), yum gets
> an error posting its progress reports, and then it starts removing
> lots and lots of packages to keep dependencies from being unmet.
Use screen.
I do. But how do you propose to script that over 10+ (or 100+ :-)
boxes?
I use screen on the server from which I initiate the updates, but that
doesn't cut it when it's the server that is rebooted, or gets
disconnected from the client.
And then, using screen is just a work around for a serious problem.
Wouldn't we be better off solving the problem in the first place?
--
Alexandre Oliva http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist oliva(a){lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer aoliva(a){redhat.com, gcc.gnu.org}
5:42 p.m.
On Sun, Sep 21, 2008 at 17:55:40 -0300,
Alexandre Oliva <aoliva(a)redhat.com> wrote:
> On Sun, Sep 21, 2008 at 07:05:37 -0300,
> Alexandre Oliva <aoliva(a)redhat.com> wrote:
I do. But how do you propose to script that over 10+ (or 100+ :-)
boxes?
If you are scripting it, shouldn't you be able to run yum not attached to
any terminal? For example you might run it in the background with the -y
option. The clients could even be polling a local mirror so that you could
use changes to the mirror to start the update process. You'd still have to
make sure that the updates actually occured, but it should be rare for
them to get killed in the middle of updating.
7:06 p.m.
On Sep 21, 2008, Bruno Wolff III <bruno(a)wolff.to> wrote:
On Sun, Sep 21, 2008 at 17:55:40 -0300,
Alexandre Oliva <aoliva(a)redhat.com> wrote:
> > On Sun, Sep 21, 2008 at 07:05:37 -0300,
> > Alexandre Oliva <aoliva(a)redhat.com> wrote:
>
> I do. But how do you propose to script that over 10+ (or 100+ :-)
> boxes?
If you are scripting it, shouldn't you be able to run yum not
attached to
any terminal?
That's exactly how I run it. It knows it doesn't have a terminal, and
it doesn't print as much in terms of progress report as it does when
it's run in a terminal. But it still prints messages once it installs
each package, and I redirect that to a file on the server. Still, if
the server fails, yum stops installing packages, and then starts
removing packages that it had scheduled to remove in the transaction,
or something like that.
And then, again, you seem to be proposing work-arounds for what I
perceive as an actual problem.
Do you really not think there is a problem, or are you just trying to
be helpful to me?
My interest is not in work arounds, it's in addressing the actual
problem, if there's agreement that there is one, so that it doesn't
bite people who didn't know about the need for work arounds in the
first place.
--
Alexandre Oliva http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist oliva(a){lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer aoliva(a){redhat.com, gcc.gnu.org}
7:51 p.m.
On Sun, Sep 21, 2008 at 21:06:03 -0300,
Alexandre Oliva <aoliva(a)redhat.com> wrote:
My interest is not in work arounds, it's in addressing the actual
problem, if there's agreement that there is one, so that it doesn't
bite people who didn't know about the need for work arounds in the
first place.
There seems to be two problems you are referring to. One is convenient
remote management and the other is transactions not being atomic.
The grid people probably have some remote management tools to help
them out. If these aren't available in Fedora, then maybe someone can
get them into Fedora. There have been a few messages from people trying to
get a SIG off the ground that would be a good place to address this.
Doing atomic yum transactions is going to be hard. Probably your effort would
be better spent keeping the system from crashing during an update then
trying to implement atomic updates.
10:29 p.m.
On Mon, Sep 22, 2008 at 12:51 PM, Bruno Wolff III <bruno(a)wolff.to> wrote:
Doing atomic yum transactions is going to be hard.
No doubt about that. RPM/Yum atomicity without FS snapshot support is
impossible to achieve. But I am sure a lot of things can be improved.
Probably your effort would
be better spent keeping the system from crashing during an update then
trying to implement atomic updates.
I cannot speak for Alexandre - but in my case that is just not
possible. I will soon be dealing with thousands of servers running in
rural schools in many countries. Very unreliable power, spotty
internet connectivity. Once yum has started an operation, I hope it
can work hard at completing it, even if that's across several reboots.
cheers,
martin
--
martin.langhoff(a)gmail.com
martin(a)laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
11:07 p.m.
On Mon, 2008-09-22 at 15:29 +1200, Martin Langhoff wrote:
On Mon, Sep 22, 2008 at 12:51 PM, Bruno Wolff III
<bruno(a)wolff.to> wrote:
> Doing atomic yum transactions is going to be hard.
No doubt about that. RPM/Yum atomicity without FS snapshot support is
impossible to achieve. But I am sure a lot of things can be improved.
> Probably your effort would
> be better spent keeping the system from crashing during an update then
> trying to implement atomic updates.
I cannot speak for Alexandre - but in my case that is just not
possible. I will soon be dealing with thousands of servers running in
rural schools in many countries. Very unreliable power, spotty
internet connectivity. Once yum has started an operation, I hope it
can work hard at completing it, even if that's across several reboots.
Rpm updates should behave as expected. When updating packages rpm
installs all the new packages first, then cleans up the old packages.
So, at no time is the system without at least 1 copy of the package. The
most common problem we've run into with yum is getting halfway through a
transaction and power or network or something fails such that the second
part, cleaning up the older packages, cannot occur.
this is where yum-complete-transaction comes in. It will finish up the
remaining steps since it has enumerated them.
Hope this helps, explain.
-sv
11:31 p.m.
On Mon, Sep 22, 2008 at 12:07:45AM -0400, Seth Vidal wrote:
So, at no time is the system without at least 1 copy of the package.
The
most common problem we've run into with yum is getting halfway through a
transaction and power or network or something fails such that the second
part, cleaning up the older packages, cannot occur.
this is where yum-complete-transaction comes in. It will finish up the
remaining steps since it has enumerated them.
Seth,
Due to memory pressure and constraints on the XO, we regularly see some
RPM %pre- and %post-scripts fail due to OOM. Is there anything which
could reasonably be done to mitigate this problem for our yum-users?
Thanks,
Michael
(Background: we believe that yum is used on our systems primarily by
hackers and 'technically minded' G1G1 donors for customizing OLPC
software releases; to date, general software updates have been made
available through olpc-update, which is atomic, bandwidth-efficient, and
which permits 'emergency rollback' to the previous (known-good) OS
tree.)
11:42 p.m.
On Mon, 2008-09-22 at 00:31 -0400, Michael Stone wrote:
On Mon, Sep 22, 2008 at 12:07:45AM -0400, Seth Vidal wrote:
>So, at no time is the system without at least 1 copy of the package. The
>most common problem we've run into with yum is getting halfway through a
>transaction and power or network or something fails such that the second
>part, cleaning up the older packages, cannot occur.
>
>this is where yum-complete-transaction comes in. It will finish up the
>remaining steps since it has enumerated them.
Seth,
Due to memory pressure and constraints on the XO, we regularly see some
RPM %pre- and %post-scripts fail due to OOM. Is there anything which
could reasonably be done to mitigate this problem for our yum-users?
Not much - if it's OOM'ing the only real option is:
1. get more memory
2. have it use less memory
if you want to pursue 2 that's fine - but to be very clear - everything
after:
Running RPM Transaction Test
is in rpm-land as to trimming out memory/cpu usage. There's very little
yum can do there.
-sv
11:53 p.m.
On Mon, Sep 22, 2008 at 4:42 PM, Seth Vidal <skvidal(a)fedoraproject.org> wrote:
Not much - if it's OOM'ing the only real option is:
1. get more memory
2. have it use less memory
if you want to pursue 2 that's fine - but to be very clear - everything
after:
Running RPM Transaction Test
is in rpm-land as to trimming out memory/cpu usage. There's very little
yum can do there.
I *think* that what Michael is talking about is that when you get to
that point yum is still running and can be a substantial part of the
memory pressure.
Perhaps yum's memory footprint could be trimmed? If by then it's
loaded hard-to-shed things (like python libs) perhaps it can exec a
minimalistic "yum stage 2" that drives rpm?
cheers,
m
--
martin.langhoff(a)gmail.com
martin(a)laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
midnight
Martin,
I'm sure there are easier things to prune than yum itself. I was simply
asking whether yum's transacation-completion machinery was applicable
here.
In the meantime, I suspect that we will recommend the use of a temporary
off-NAND swap partition or swap-file by the people who need it.
Michael
12:19 a.m.
On Mon, Sep 22, 2008 at 5:00 PM, Michael Stone <michael(a)laptop.org> wrote:
I'm sure there are easier things to prune than yum itself.
Perhaps you are right. I was probably expressing my own worries about
yum memory usage on the XO :-)
In a 30s unscientific test yum does seem to consume a fair bit of RAM
while RPM runs. The F9 machine I use for dev/build purposes had 9
updates to apply.
I just did `yum -yt update` while running repeatedly `ps_mem.py |
grep yum ` (in hindsight, I should have grepped for rpm as well). It
quickly ramped up to 44.9 MiB private + 1.5 MiB shared during the
Resolving Dependencies stage, and stayed there for the duration. While
rpm was running, yum was _still_ sitting on thos 46MiB.
It looks like it has the whole package metadata in a memory structure.
Perhaps it's needed for the plugins, though I hope the internal APIs
don't require that all that stuff is in memory.
whether yum's transacation-completion machinery was applicable
here.
From what I read in this thread the transaction-completion machinery
doesn't try to parse the repo data into memory. It just reads the plan
from the transaction log and tries to execute it. If that's true, then
it's got a good chance as a workaround.
cheers,
m
--
martin.langhoff(a)gmail.com
martin(a)laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
12:38 a.m.
On Mon, 2008-09-22 at 17:19 +1200, Martin Langhoff wrote:
On Mon, Sep 22, 2008 at 5:00 PM, Michael Stone
<michael(a)laptop.org> wrote:
> I'm sure there are easier things to prune than yum itself.
Perhaps you are right. I was probably expressing my own worries about
yum memory usage on the XO :-)
In a 30s unscientific test yum does seem to consume a fair bit of RAM
while RPM runs. The F9 machine I use for dev/build purposes had 9
updates to apply.
I just did `yum -yt update` while running repeatedly `ps_mem.py |
grep yum ` (in hindsight, I should have grepped for rpm as well). It
quickly ramped up to 44.9 MiB private + 1.5 MiB shared during the
Resolving Dependencies stage, and stayed there for the duration. While
rpm was running, yum was _still_ sitting on thos 46MiB.
What arch? How many repos. did you have enabled?
In my tests the latest yum does a "large" update in about 40MB RSS, a
few packages is much more likely to be in the 25-30MB range (VSZ is
higher, but is just unpaged stuff, not actual swap).
It looks like it has the whole package metadata in a memory
structure.
Perhaps it's needed for the plugins, though I hope the internal APIs
don't require that all that stuff is in memory.
We drop all the cached data from within yum when the depsolving is
finished, however we have little control of how much of that is then
usable by rpm to do the transaction (that'd be mostly upto python and
libc).
We also have plans to split transactions, but I'm not sure you should
rely on that to significantly drop the memory usage requirements
(although it will make the failure case much nicer).
--
James Antill <james.antill(a)redhat.com>
Red Hat
12:56 a.m.
2008/9/22 James Antill <james.antill(a)redhat.com>:
What arch? How many repos. did you have enabled?
i386 - and the very minimum repos
$ yum repolist
Loaded plugins: refresh-packagekit
repo id repo name status
fedora Fedora 9 - i386 enabled : 9,897
updates Fedora 9 - i386 - Updates enabled : 10
updates-newkey Fedora 9 - i386 - Updates Newkey enabled : 3,951
repolist: 13,858
In my tests the latest yum does a "large" update in about
40MB RSS, a
few packages is much more likely to be in the 25-30MB range (VSZ is
higher, but is just unpaged stuff, not actual swap).
I'm looking at the output of ps_mem.py which uses the kernel smap data
-- which I understand is regarded as much more reliable than what top
reports these days - see
http://www.pixelbeat.org/scripts/ps_mem.py
We drop all the cached data from within yum when the depsolving is
finished, however we have little control of how much of that is then
There was no observable drop in memory footprint from depsolving to
rpm stages, even if we were just updating 9 packages.
If my rough test is reasonably accurate then it's easy to see why
people would get OOM with yum on the XO -- just reading the release
repo, yum is sitting on a good chunk of our available memory, and we
have _no_ swap available.
One workaround would be to say --disablerepo=release but that means
any install that requires packages from the main repo will fail.
We also have plans to split transactions, but I'm not sure you
should
rely on that to significantly drop the memory usage requirements
(although it will make the failure case much nicer).
I'm *all* for making the failure case better as the XS installations
will fail... plenty :-)
cheers,
m
--
martin.langhoff(a)gmail.com
martin(a)laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
2:29 a.m.
On Mon, 2008-09-22 at 16:53 +1200, Martin Langhoff wrote:
On Mon, Sep 22, 2008 at 4:42 PM, Seth Vidal
<skvidal(a)fedoraproject.org> wrote:
> Not much - if it's OOM'ing the only real option is:
> 1. get more memory
> 2. have it use less memory
>
> if you want to pursue 2 that's fine - but to be very clear - everything
> after:
> Running RPM Transaction Test
>
> is in rpm-land as to trimming out memory/cpu usage. There's very little
> yum can do there.
I *think* that what Michael is talking about is that when you get to
that point yum is still running and can be a substantial part of the
memory pressure.
Perhaps yum's memory footprint could be trimmed?
How much memory do you have?
I am able to run yum (FC9) on an i586 w/ 64MB RAM and 256MB swap.
In the past, yum occasionally had OOM'ed, but this hasn't happened for
quite a while. The key to me had been to slim down the kernel memory
requirements (Switching off SELinux worked wonders! This particular
machine OOMs when booting with SELinux enabled)
Ralf
2:32 a.m.
Quoth Ralf Corsepius:
On Mon, 2008-09-22 at 16:53 +1200, Martin Langhoff wrote:
> On Mon, Sep 22, 2008 at 4:42 PM, Seth Vidal <skvidal(a)fedoraproject.org>
wrote:
> > Not much - if it's OOM'ing the only real option
is:
> > 1. get more memory
> > 2. have it use less memory
> >
> > if you want to pursue 2 that's fine - but to be very clear - everything
> > after:
> > Running RPM Transaction Test
> >
> > is in rpm-land as to trimming out memory/cpu usage. There's very little
> > yum can do there.
>
> I *think* that what Michael is talking about is that when you get to
> that point yum is still running and can be a substantial part of the
> memory pressure.
>
> Perhaps yum's memory footprint could be trimmed?
How much memory do you have?
I am able to run yum (FC9) on an i586 w/ 64MB RAM and 256MB swap.
In the past, yum occasionally had OOM'ed, but this hasn't happened for
quite a while. The key to me had been to slim down the kernel memory
requirements (Switching off SELinux worked wonders! This particular
machine OOMs when booting with SELinux enabled)
Ralf
Right, I run yum on a similar (i586, 128M of ram, plenty of swap) machine and
it works great nowadays.
Regards,
--
Conrad Meyer <konrad(a)tylerc.org>
4:07 a.m.
On Mon, Sep 22, 2008 at 7:32 PM, Conrad Meyer <konrad(a)tylerc.org> wrote:
Quoth Ralf Corsepius:
> I am able to run yum (FC9) on an i586 w/ 64MB RAM and 256MB swap.
Right, I run yum on a similar (i586, 128M of ram, plenty of swap) machine and
it works great nowadays.
We're talking about OLPC's XOs - with 256 RAM and no swap. Our 'base
working set' is about half of that - so 128MB is all there is to play
with. We're trying hard to trim out base working set, but that's where
we are now (and some dev builds have been much more hoggish).
The kernel is a champ as long as there's memory. When memory pressure
kicks in it all turns nasty pretty quickly...
Anyway - I'm happy to test again in a more controlled test-case if
Seth wants it, but at first blush it looks like yum is not releasing
memory at a time when it could.
cheers,
m
--
martin.langhoff(a)gmail.com
martin(a)laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
4:28 a.m.
On Mon, 2008-09-22 at 21:07 +1200, Martin Langhoff wrote:
at first blush it looks like yum is not releasing
memory at a time when it could.
I guess its always possible that the unused memory hasn't been garbage
collected by python yet. Happens a lot for anyone playing with pygtk
pixbufs for example. Maybe experiment with whacking a great big
gc.collect() at the end of depsolving to see if it makes a blind bit of
difference.
C.
4:46 a.m.
Quoth Martin Langhoff:
On Mon, Sep 22, 2008 at 7:32 PM, Conrad Meyer
<konrad(a)tylerc.org> wrote:
> Quoth Ralf Corsepius:
>> I am able to run yum (FC9) on an i586 w/ 64MB RAM and 256MB swap.
> Right, I run yum on a similar (i586, 128M of ram, plenty of swap) machine
and
> it works great nowadays.
We're talking about OLPC's XOs - with 256 RAM and no swap. Our 'base
working set' is about half of that - so 128MB is all there is to play
with. We're trying hard to trim out base working set, but that's where
we are now (and some dev builds have been much more hoggish).
The kernel is a champ as long as there's memory. When memory pressure
kicks in it all turns nasty pretty quickly...
Anyway - I'm happy to test again in a more controlled test-case if
Seth wants it, but at first blush it looks like yum is not releasing
memory at a time when it could.
cheers,
m
--
martin.langhoff(a)gmail.com
martin(a)laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
Any reason for not having swap? Or if you don't want to use the disk all the
time, perhaps create a swap file before doing updates with yum? But as you've
said before, most users won't be using yum anyways (and even fewer users will
actually be getting updates at all). Power users can easily add swap.
Regards,
--
Conrad Meyer <konrad(a)tylerc.org>
5:47 a.m.
On Mon, Sep 22, 2008 at 9:46 PM, Conrad Meyer <konrad(a)tylerc.org> wrote:
Any reason for not having swap? Or if you don't want to use the
disk all the
time, perhaps create a swap file before doing updates with yum?
Lots. We are running jffs2 on bare NAND. A NAND partition dedicated to
a conventional swap would wear out badly. And jffs2 is a compressed fs
(no per-file flag to disable compression) with very strange
performance patterns -- a swapfile in there is bad news.
But as you've
said before, most users won't be using yum anyways (and even fewer users will
actually be getting updates at all). Power users can easily add swap.
Well, we are using a homegrown update mechanism. It has some
advantages over yum -- for starters, it works on our hw :-) . Let's
say that the fact that yum+rpm do _not_ work on our hardware for large
upgrades - due to memory issues - closes many avenues.
cheers,
m
--
martin.langhoff(a)gmail.com
martin(a)laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
6:06 a.m.
On Mon, 2008-09-22 at 22:47 +1200, Martin Langhoff wrote:
On Mon, Sep 22, 2008 at 9:46 PM, Conrad Meyer
<konrad(a)tylerc.org> wrote:
> Any reason for not having swap? Or if you don't want to use the disk all the
> time, perhaps create a swap file before doing updates with yum?
Lots. We are running jffs2 on bare NAND. A NAND partition dedicated to
a conventional swap would wear out badly. And jffs2 is a compressed fs
(no per-file flag to disable compression) with very strange
performance patterns -- a swapfile in there is bad news.
Why? You could create a
temporary swapfile when launching yum and delete
it afterwards. It would not cause more wear-out than any other dynamic
file somewhere on the file system.
> But as you've
> said before, most users won't be using yum anyways (and even fewer users will
> actually be getting updates at all). Power users can easily add swap.
Well, we are using a homegrown update mechanism. It has some
advantages over yum -- for starters, it works on our hw :-) . Let's
say that the fact that yum+rpm do _not_ work on our hardware for large
upgrades - due to memory issues - closes many avenues.
Openly said, based on my
experience with my i586, I don't buy this.
Usually, the memory requirements introduced other applications are
magnitudes above that of yum.
However I am not sufficiently familiar with OPLC to be able to further
comment on this.
Ralf
8:08 p.m.
On Mon, Sep 22, 2008 at 01:06:24PM +0200, Ralf Corsepius wrote:
Openly said, based on my experience with my i586, I don't buy
this.
Usually, the memory requirements introduced other applications are
magnitudes above that of yum.
My impression is that our lack of swap really hurts us here; on most
systems, it's possible to swap some pages (dirty or clean) at relatively
fixed cost and keep going. We _suspect_ (but have not yet proven) that
on XOs, under memory pressure, we start thrashing code pages (which can
be unmapped since they are clean but which must be de-compressed in
order to be remapped) and eventually OOM when we finally receive an
allocation that cannot possibly be satisfied.
However I am not sufficiently familiar with OPLC to be able to
further
comment on this.
If you would like to become sufficiently familiar with XOs to comment
further, it can be easily arranged, either through our Developer's
Program [1] or through a renewed G1G1 effort scheduled for November [2].
Please consider it -- we could use your help and it's a great way to
help OLPC get better at doing things the Fedora way.
Michael
[1]: http://wiki.laptop.org/go/Contributors_program
[2]: sneak preview; more information to follow in coming weeks
11:05 p.m.
On Sep 21, 2008, Bruno Wolff III <bruno(a)wolff.to> wrote:
There seems to be two problems you are referring to. One is
convenient
remote management and the other is transactions not being atomic.
I'm actually talking about a third problem: that yum starts removing
packages just because of an error in reporting progress to
stdout/stderr. Everything else was just context to trigger the
problem.
--
Alexandre Oliva http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist oliva(a){lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer aoliva(a){redhat.com, gcc.gnu.org}
6:55 a.m.
On Mon, 2008-09-22 at 01:05 -0300, Alexandre Oliva wrote:
On Sep 21, 2008, Bruno Wolff III <bruno(a)wolff.to> wrote:
> There seems to be two problems you are referring to. One is convenient
> remote management and the other is transactions not being atomic.
I'm actually talking about a third problem: that yum starts removing
packages just because of an error in reporting progress to
stdout/stderr. Everything else was just context to trigger the
problem.
So let's clear a few things up:
1. Once the transaction starts 'yum' isn't doing anything. It is
reporting what rpm is doing.
2. The only things it is possible for rpm to do is what is in the
transaction set and the user agrees to perform by hitting 'y'
So, if you have a case where pkgs are being removed that you have not
agreed to remove, please file a bug with details.
-sv
1:47 p.m.
On Sep 22, 2008, Seth Vidal <skvidal(a)fedoraproject.org> wrote:
On Mon, 2008-09-22 at 01:05 -0300, Alexandre Oliva wrote:
> On Sep 21, 2008, Bruno Wolff III <bruno(a)wolff.to> wrote:
>
> > There seems to be two problems you are referring to. One is convenient
> > remote management and the other is transactions not being atomic.
>
> I'm actually talking about a third problem: that yum starts removing
> packages just because of an error in reporting progress to
> stdout/stderr. Everything else was just context to trigger the
> problem.
So let's clear a few things up:
1. Once the transaction starts 'yum' isn't doing
anything. It is
reporting what rpm is doing.
It's precisely when yum's reporting fails that the situation arises.
2. The only things it is possible for rpm to do is what is in the
transaction set and the user agrees to perform by hitting 'y'
Right. The problem is when you agree to "update x", i.e., "install
x-N+1; remove x-N', and it fails (say, in case sys.stdout.write or
sys.stdout.flush raise an exception within callback) before it gets to
install x-N+1. Then RPM still ends up removing x-N, although
sometimes what I saw looked more like a --justdb removal (files and
libraries left behind, but package gone from the rpmdb), but in
others, as in the most recent case, elfutils libraries were really
gone.
So, if you have a case where pkgs are being removed that you have
not
agreed to remove, please file a bug with details.
In the scenario above, would you say you agreed to remove x-N or not?
I.e., does it pass the condition you set for a bug report to be filed?
Could it be the solution is as simple as guarding (exception-wise)
sys.stdout calls in callbacks, to avoid exception leakage that could
get the rpm transaction aborted in this way? (I'm not sufficiently
versed in Python to know whether write or flush will raise exceptions).
--
Alexandre Oliva http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist oliva(a){lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer aoliva(a){redhat.com, gcc.gnu.org}
1:52 p.m.
On Mon, 2008-09-22 at 15:47 -0300, Alexandre Oliva wrote:
Right. The problem is when you agree to "update x", i.e.,
"install
x-N+1; remove x-N', and it fails (say, in case sys.stdout.write or
sys.stdout.flush raise an exception within callback) before it gets to
install x-N+1. Then RPM still ends up removing x-N, although
sometimes what I saw looked more like a --justdb removal (files and
libraries left behind, but package gone from the rpmdb), but in
others, as in the most recent case, elfutils libraries were really
gone.
The above really isn't possible. If you can recreate this then please
file a bug. File the bug against rpm but please cc me on it.
> So, if you have a case where pkgs are being removed that you
have not
> agreed to remove, please file a bug with details.
In the scenario above, would you say you agreed to remove x-N or not?
I.e., does it pass the condition you set for a bug report to be filed?
If you have a case where x-N is being removed w/o x-N+1 being installed
then yes, file a bug.
-sv
4:03 a.m.
On Mon, 22 Sep 2008, seth vidal wrote:
On Mon, 2008-09-22 at 15:47 -0300, Alexandre Oliva wrote:
> Right. The problem is when you agree to "update x", i.e., "install
> x-N+1; remove x-N', and it fails (say, in case sys.stdout.write or
> sys.stdout.flush raise an exception within callback) before it gets to
> install x-N+1. Then RPM still ends up removing x-N, although
> sometimes what I saw looked more like a --justdb removal (files and
> libraries left behind, but package gone from the rpmdb), but in
> others, as in the most recent case, elfutils libraries were really
> gone.
The above really isn't possible. If you can recreate this then please
file a bug. File the bug against rpm but please cc me on it.
I wouldn't call it impossible, in fact I just managed to reproduce it with
this:
--- a/yum/rpmtrans.py
+++ b/yum/rpmtrans.py
@@ -374,6 +374,7 @@ class RPMTransaction:
self.total_installed += 1
self.complete_actions += 1
self.installed_pkg_names.append(hdr['name'])
+ raise IOError
return fd
else:
self.display.errorlog("Error: No Header to INST_OPEN_FILE")
Without having yet looked deeper into it, it probably comes down to this
in rpmtsRun():
/*
* XXX This has always been a hack, now mostly broken.
* If install failed, then we shouldn't erase.
*/
The hack in question is easily fooled, and so rpm is ultimately
responsible for the damage that results from yum code tracebacking in the
transaction callback. Rpm needs fixing (I'll go look into it right now),
but I'd suggest you go and comb through the ts callback code in yum - you
do NOT want it tracebacking, especially on "trivial" things like
sys.stdout operations failing.
- Panu -
7:10 a.m.
On Tue, 2008-09-23 at 12:03 +0300, Panu Matilainen wrote:
> The above really isn't possible. If you can recreate this
then please
> file a bug. File the bug against rpm but please cc me on it.
I wouldn't call it impossible, in fact I just managed to reproduce it with
this:
You're right - I should have said 'shouldn't be possible'. Not isn't
possible.
Without having yet looked deeper into it, it probably comes down to
this
in rpmtsRun():
/*
* XXX This has always been a hack, now mostly broken.
* If install failed, then we shouldn't erase.
*/
That's a nice fixme to have. :)
The hack in question is easily fooled, and so rpm is ultimately
responsible for the damage that results from yum code tracebacking in the
transaction callback. Rpm needs fixing (I'll go look into it right now),
but I'd suggest you go and comb through the ts callback code in yum - you
do NOT want it tracebacking, especially on "trivial" things like
sys.stdout operations failing.
I'll look into it, thanks,
-sv
9:41 a.m.
On Tue, 2008-09-23 at 12:03 +0300, Panu Matilainen wrote:
On Mon, 22 Sep 2008, seth vidal wrote:
> On Mon, 2008-09-22 at 15:47 -0300, Alexandre Oliva wrote:
>
>> Right. The problem is when you agree to "update x", i.e.,
"install
>> x-N+1; remove x-N', and it fails (say, in case sys.stdout.write or
>> sys.stdout.flush raise an exception within callback) before it gets to
>> install x-N+1. Then RPM still ends up removing x-N, although
>> sometimes what I saw looked more like a --justdb removal (files and
>> libraries left behind, but package gone from the rpmdb), but in
>> others, as in the most recent case, elfutils libraries were really
>> gone.
>
> The above really isn't possible. If you can recreate this then please
> file a bug. File the bug against rpm but please cc me on it.
I wouldn't call it impossible, in fact I just managed to reproduce it with
this:
--- a/yum/rpmtrans.py
+++ b/yum/rpmtrans.py
@@ -374,6 +374,7 @@ class RPMTransaction:
self.total_installed += 1
self.complete_actions += 1
self.installed_pkg_names.append(hdr['name'])
+ raise IOError
return fd
else:
self.display.errorlog("Error: No Header to INST_OPEN_FILE")
Without having yet looked deeper into it, it probably comes down to this
in rpmtsRun():
/*
* XXX This has always been a hack, now mostly broken.
* If install failed, then we shouldn't erase.
*/
The hack in question is easily fooled, and so rpm is ultimately
responsible for the damage that results from yum code tracebacking in the
transaction callback.
I think Seth meant "I find it impossible that rpm has a bug/feature
which would allow this to happen" ... but he's just way too much of a
raving optimist :).
Rpm needs fixing (I'll go look into it right now),
but I'd suggest you go and comb through the ts callback code in yum - you
do NOT want it tracebacking, especially on "trivial" things like
sys.stdout operations failing.
I think I've worked around the "normal" cases on the yum side with:
http://devel.linux.duke.edu/gitweb/?p=yum.git;a=commitdiff;h=cf896fc7ccc1...
and
http://devel.linux.duke.edu/gitweb/?p=yum.git;a=commitdiff;h=0aed92573666...
...although it doesn't "fix" the patch above, I think it'll work around
the ssh case ... Alexandre, if you can reproduce it can you try it with
the latest from the yum-3_2_X branch?
--
James Antill <james.antill(a)redhat.com>
Red Hat
4:27 p.m.
On Sun, 2008-09-21 at 07:05 -0300, Alexandre Oliva wrote:
On Sep 11, 2008, Seth Vidal <skvidal(a)fedoraproject.org> wrote:
> On Thu, 2008-09-11 at 03:24 +0200, Ahmed Kamal wrote:
>> Hi,
>> I had my computer hang during a major yum upgrade.
> When this happens you should run:
> yum-complete-transaction
Also note that the latest yum will inform you to run the above, when
you need to.
In a slightly different scenario: I often find that, if the ssh
connection from which I start 'yum update' is broken for whatever
reason (say the machine from which I started reboots or so), yum gets
an error posting its progress reports, and then it starts removing
lots and lots of packages to keep dependencies from being unmet.
I haven't heard this before, and I don't see how yum could do that. I
assume you mean when the connection dies after the "Running transaction"
stage? At this point yum is really rpm.
Although I'd still be surprised if rpm altered the running transaction
to remove a bunch of applications, it seems more plausible than yum
doing the same.
The error I usually see is when the connection dies and yum/rpm
instantly die with it ... at which point rpm is some way through the
transaction and so for all the updates which have been done without the
corresponding cleanup you actually have two versions of those packages
installed. Due to version deps. this sometimes means that things don't
work.
Wouldn't it be much nicer if, like, it ran to completion without
regarding the stdout errors; recorded a recovery transaction to revert
the removals, or at least recorded a transaction to complete the
interrupted update?
That's what yum-complete-transaction does. "running to completion"
would only really be possible if we compartmentalized the UI from the
backend (in different processes), which would be nice ... but is a lot
of work.
I realize the latter is risky, for you may end up unable to start
yum
in the first place, but this unfortunately is also the case of the
current code. Last time I got this error, an elfutils update was part
of the transaction, and it stopped at such an unfortunate time that
elfutils got removed, and rpm would no longer run. Oops :-)
Is this bug report material?
If you have a reproducer feel free to report it and we'll investigate.
--
James Antill <james(a)fedoraproject.org>
Fedora
5730
days inactive
5742
days old
40 comments
12 participants
participants (12)
-
Ahmed Kamal -
Alexandre Oliva -
Bruno Wolff III -
Caolán McNamara -
Conrad Meyer -
James Antill -
Martin Langhoff -
Michael Schwendt -
Michael Stone -
Panu Matilainen -
Ralf Corsepius -
Seth Vidal