tip: automated way to delete duplicate rpms

[Kim Lux]

I recently updated a server from fc3 to fc6.  All seemed to go well
except the first yum attempt to update the machine failed and when I
entered synaptic I received a ton of errors about duplicate rpms
installed on the machine.  I've run into this issue before.
Unfortunately neither yum nor synaptic has a way to deal with it, as far
as I can tell.   

Rather than manually go through the list of offending rpms and delete
them, I did things semi automatically.  There were 120 duplicate rpms on
my machine, so manually typing the full package names would have taken a
while.

Here is what I did instead.

1) Generate a list of all the rpm packages installed.

$rpm -qa --queryformat '%{name}\n' > pkglist

This generates an unsorted list that looks like this.
libavc1394
libmusicbrainz
ethtool
desktop-file-utils
less
time
libXrandr
rmt

2) Sort the list and find the duplicates

$sort -g pkglist | uniq -d > duplicates

This list will look like 

a2ps
anacron
at
audit
...

3) Generate a list of the packages with versions to look at and edit

I created a small script file called dupWithVersions with this in it

for pkg in `cat duplicates`; do
  rpm -q $pkg
done

Redirect the output to another file like this.  $dupWithVersions
> removeList 
The remove list will look like this:

a2ps-4.13b-57
a2ps-4.13b-57.fc6.3
anacron-2.3-41.fc6
anacron-2.3-44.fc6
at-3.1.8-82.fc6
at-3.1.8-84.fc6
audit-1.2.8-1.fc6
audit-1.3-2.fc6

Go through the remove list and manually remove the versions you want to
KEEP !  Remember this is a list of rpms you want to remove !

4) Remove the rpms in the remove list.

To do this I generated a little script as follows:

for pkg in `cat removelist`; do
  rpm -e $pkg
done

This process quickly and correctly removed the duplicate rpms from my
machine. 

I hope this helps someone. 












 

-- 
Kim Lux,  Diesel Research Inc.

[Jacques B.]

>  I created a small script file called dupWithVersions with this in it
>
>  for pkg in `cat duplicates`; do
>    rpm -q $pkg
>  done
>
>  Redirect the output to another file like this.  $dupWithVersions
>  >removeList 
Great tip!  Might I add something.  Once you have a list of unique
files (unedited - full list of unique files) run a for loop using
those files for input (as above) but do a grep against the original
list of rpms, using the count option.  Then with a simple "if"
command, if the count is greater than 1, remove it.  Continue the for
loop.

That way the script automatically determines if there is more than one
version of an rpm installed and removes it.

Jacques B.

[Development discussions related to Fedora]

Kim Lux wrote:
>  I recently updated a server from fc3 to fc6.  All seemed to go well
>  except the first yum attempt to update the machine failed and when I
>  entered synaptic I received a ton of errors about duplicate rpms
>  installed on the machine.  I've run into this issue before.
>  Unfortunately neither yum nor synaptic has a way to deal with it, as far
>  as I can tell.   
>  
>  Rather than manually go through the list of offending rpms and delete
>  them, I did things semi automatically.  There were 120 duplicate rpms on
>  my machine, so manually typing the full package names would have taken a 
...
>  This process quickly and correctly removed the duplicate rpms from
> my
>  machine. 
>  
>  I hope this helps someone.  
There is a tool available to solve duplicate package problems: see
https://lists.dulug.duke.edu/pipermail/yum/2006-December/009451.html

It seems to work fine, no heroics {like Kim used above} were required.

DaveT.

[Oldman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kim Lux wrote:
>  I recently updated a server from fc3 to fc6.  All seemed to go well
>  except the first yum attempt to update the machine failed and when I
>  entered synaptic I received a ton of errors about duplicate rpms
>  installed on the machine.  I've run into this issue before.
>  Unfortunately neither yum nor synaptic has a way to deal with it, as far
>  as I can tell.   
>  
>  Rather than manually go through the list of offending rpms and delete
>  them, I did things semi automatically.  There were 120 duplicate rpms on
>  my machine, so manually typing the full package names would have taken a
>  while.
>  
>  Here is what I did instead.
>  
>  1) Generate a list of all the rpm packages installed.
>  
>  $rpm -qa --queryformat '%{name}\n' > pkglist
>  
>  This generates an unsorted list that looks like this.
>  libavc1394
>  libmusicbrainz
>  ethtool
>  desktop-file-utils
>  less
>  time
>  libXrandr
>  rmt
>  
>  2) Sort the list and find the duplicates
>  
>  $sort -g pkglist | uniq -d > duplicates
>  
>  This list will look like 
>  
>  a2ps
>  anacron
>  at
>  audit
>  ...
>  
>  3) Generate a list of the packages with versions to look at and edit
>  
>  I created a small script file called dupWithVersions with this in it
>  
>  for pkg in `cat duplicates`; do
>    rpm -q $pkg
>  done
>  
>  Redirect the output to another file like this.  $dupWithVersions
> > removeList
>  
>  The remove list will look like this:
>  
>  a2ps-4.13b-57
>  a2ps-4.13b-57.fc6.3
>  anacron-2.3-41.fc6
>  anacron-2.3-44.fc6
>  at-3.1.8-82.fc6
>  at-3.1.8-84.fc6
>  audit-1.2.8-1.fc6
>  audit-1.3-2.fc6
>  
>  Go through the remove list and manually remove the versions you want to
>  KEEP !  Remember this is a list of rpms you want to remove !
>  
>  4) Remove the rpms in the remove list.
>  
>  To do this I generated a little script as follows:
>  
>  for pkg in `cat removelist`; do
>    rpm -e $pkg
>  done
>  
>  This process quickly and correctly removed the duplicate rpms from my
>  machine. 
>  
>  I hope this helps someone. 
>   
	This worries me a little bit.  If you remove a package aren't you
likely to remove identically named files from the newer package as well?

	I ask because I usually do a rpm -e --justdb <pkgname> as I once found
myself without some services after deleting an old package.

Scott

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
>
> iD8DBQFFlsSm5mBKdb7VQEcRAnDTAKCxEod2+NmlvLXAofs0VYwCe0YM2ACgl3j9
> +fFHeQpUi5ZzrDFpbraXH5E=
> =vGO8
> -----END PGP SIGNATURE-----

[Jim Cornette]

oldman wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>  
>  Kim Lux wrote:
> > I recently updated a server from fc3 to fc6.  All seemed to go well
> > except the first yum attempt to update the machine failed and when I
> > entered synaptic I received a ton of errors about duplicate rpms
> > installed on the machine.  I've run into this issue before.
> > Unfortunately neither yum nor synaptic has a way to deal with it, as far
> > as I can tell.   
> >
> > Rather than manually go through the list of offending rpms and delete
> > them, I did things semi automatically.  There were 120 duplicate rpms on
> > my machine, so manually typing the full package names would have taken a
> > while.
> >
> > Here is what I did instead.
> >
> > 1) Generate a list of all the rpm packages installed.
> >
> > $rpm -qa --queryformat '%{name}\n' > pkglist
> >
> > This generates an unsorted list that looks like this.
> > libavc1394
> > libmusicbrainz
> > ethtool
> > desktop-file-utils
> > less
> > time
> > libXrandr
> > rmt
> >
> > 2) Sort the list and find the duplicates
> >
> > $sort -g pkglist | uniq -d > duplicates
> >
> > This list will look like 
> >
> > a2ps
> > anacron
> > at
> > audit
> > ...
> >
> > 3) Generate a list of the packages with versions to look at and edit
> >
> > I created a small script file called dupWithVersions with this in it
> >
> > for pkg in `cat duplicates`; do
> >   rpm -q $pkg
> > done
> >
> > Redirect the output to another file like this.  $dupWithVersions
> >> removeList
> > The remove list will look like this:
> >
> > a2ps-4.13b-57
> > a2ps-4.13b-57.fc6.3
> > anacron-2.3-41.fc6
> > anacron-2.3-44.fc6
> > at-3.1.8-82.fc6
> > at-3.1.8-84.fc6
> > audit-1.2.8-1.fc6
> > audit-1.3-2.fc6
> >
> > Go through the remove list and manually remove the versions you want to
> > KEEP !  Remember this is a list of rpms you want to remove !
> >
> > 4) Remove the rpms in the remove list.
> >
> > To do this I generated a little script as follows:
> >
> > for pkg in `cat removelist`; do
> >   rpm -e $pkg
> > done
> >
> > This process quickly and correctly removed the duplicate rpms from my
> > machine. 
> >
> > I hope this helps someone. 
> >
>  	This worries me a little bit.  If you remove a package aren't you
>  likely to remove identically named files from the newer package as well?
>  
>  	I ask because I usually do a rpm -e --justdb <pkgname> as I once found
>  myself without some services after deleting an old package.
>  
>  Scott 
The safest way to remove the older package remnants is to download the 
newer packages and run
'rpm -Uvh --replacepkgs --replacefiles packagename(s).arch.rpm'
because of the reasons that you pointed to. If the older package has 
files that are the same name as used in the newer package, erasure of 
the files is probable when you erase the older version of the rpm.

Running 'rpm -Uvh --justdb --nodeps packagename(s).arch.rpm' will safely 
only remove the database entry but will leave the documentation (has a 
version number) and any files that are not contained within the new 
package lingering on your system. But for most occasions that I seen, 
each version of an rpm usually has files placed in the same location and 
will differ only in date and other file characteristics. That is why 
when you do an 'rpm -qV packagename' on the older package you usually 
see a lot of output with file details being different and when checking 
the later version you get no output because all is well with the newer 
version rpm.

I think so anyway! :-)
Jim

-- 
People of privilege will always risk their complete destruction
rather than surrender any material part of their advantage.
		-- John Kenneth Galbraith

[Oldman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jim Cornette wrote:

>  
>  The safest way to remove the older package remnants is to download the
>  newer packages and run
>  'rpm -Uvh --replacepkgs --replacefiles packagename(s).arch.rpm'
>  because of the reasons that you pointed to. If the older package has
>  files that are the same name as used in the newer package, erasure of
>  the files is probable when you erase the older version of the rpm.
>  
>  Running 'rpm -Uvh --justdb --nodeps packagename(s).arch.rpm' will safely
>  only remove the database entry but will leave the documentation (has a
>  version number) and any files that are not contained within the new
>  package lingering on your system. But for most occasions that I seen,
>  each version of an rpm usually has files placed in the same location and
>  will differ only in date and other file characteristics. That is why
>  when you do an 'rpm -qV packagename' on the older package you usually
>  see a lot of output with file details being different and when checking
>  the later version you get no output because all is well with the newer
>  version rpm.
>  
>  I think so anyway! :-)
>  Jim
>   
	Thanks Jim
	When this happened previously, I found out when some new program being
updated would not update the dependency or the program due to the older
package. removing the older package from the db allowed the upgrade to
progress.
	I might have an extra file or two running around on my machine, but it
beats the hell out of not having a file or two! :-)
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
>
> iD8DBQFFmASR5mBKdb7VQEcRAiIRAKCOmZKnaCHjxU/7X0wbkCsInV8B3QCfV0Sw
> kgjVNGiJPxZOYaH3tPj0Bdc=
> =Wtj3
> -----END PGP SIGNATURE-----

[Jim Cornette]

oldman wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA1
>  
>  Jim Cornette wrote:
>  
> > The safest way to remove the older package remnants is to download the
> > newer packages and run
> > 'rpm -Uvh --replacepkgs --replacefiles packagename(s).arch.rpm'
> > because of the reasons that you pointed to. If the older package has
> > files that are the same name as used in the newer package, erasure of
> > the files is probable when you erase the older version of the rpm.
> >
> > Running 'rpm -Uvh --justdb --nodeps packagename(s).arch.rpm' will safely
> > only remove the database entry but will leave the documentation (has a
> > version number) and any files that are not contained within the new
> > package lingering on your system. But for most occasions that I seen,
> > each version of an rpm usually has files placed in the same location and
> > will differ only in date and other file characteristics. That is why
> > when you do an 'rpm -qV packagename' on the older package you usually
> > see a lot of output with file details being different and when checking
> > the later version you get no output because all is well with the newer
> > version rpm.
> >
> > I think so anyway! :-)
> > Jim
> >
>  	Thanks Jim
>  	When this happened previously, I found out when some new program being
>  updated would not update the dependency or the program due to the older
>  package. removing the older package from the db allowed the upgrade to
>  progress.
>  	I might have an extra file or two running around on my machine, but it
>  beats the hell out of not having a file or two! :-) 
I usually do the rpm -e --nodep --justdb oldpackage-version myself. I 
only recently realized that there was most likely doc files left behind. 
Previously I assumed that all files would be of the new version and the 
old files would not be present.

rpm -Uvh --replacefiles --replacepkgs latestpackage

is the best bet though. You are right that having lingering 
documentation from the earlier version is better than removing needed 
files which are most likely from the latest version that was installed.

Jim

-- 
When in doubt, follow your heart.

[Kim Lux]

On Sat, 2006-12-30 at 11:57 -0800, oldman wrote:
>   This worries me a little bit.  If you remove a package aren't
> you
>  likely to remove identically named files from the newer package as
>  well?
>  
>          I ask because I usually do a rpm -e --justdb <pkgname> as I
>  once found
>  myself without some services after deleting an old package. 
I agree its not good to have multiple versions installed.

You could always do a refresh on the packages that remain behind.  One
could put a "yumdownloader $pkg rpm --refresh $pkg in a  loop for the
list of packages that are left installed. 



 
-- 
Kim Lux,  Diesel Research Inc.