Replicating an installation

[Geoffrey Leach]

My internet service comes over a satellite, and with it a relatively
small monthly download allowance. Which motivates the following
question.

Once I have installed a new disto and downloaded the RPMs that I use, is
there a procedure by which I could gather together everything that I
have added, so that I could transfer the files (RPMs, or whatever) to
a local system, without resorting to the internet? (Or, at least, to a
significant amount!)

Thanks.

[George N. White III]

On Sun, 5 Apr 2020 at 17:35, Geoffrey Leach  wrote:
> My internet service comes over a satellite, and with it a relatively
> small monthly download allowance. Which motivates the following
> question.
>
> Once I have installed a new disto and downloaded the RPMs that I use, is
> there a procedure by which I could gather together everything that I
> have added, so that I could transfer the files (RPMs, or whatever) to
> a local system, without resorting to the internet? (Or, at least, to a
> significant amount!)

Is the "local system" identical to the one where you did the initial
installation?

In a former life, with RHEL versions 4--6, I used a local mirror for the
packages needed to support our use cases.    When there were updates,
they appeared in the mirror and systems updated from there.   I don't
recall there being a lot of guidance for creating a local mirror, more a
matter of trial and error to get a local mirror working with a small get of
packages.

https://fedoraproject.org/wiki/Docs/Drafts/MockSetupUsingLocalMirror
seems to be roughly the same vintage.    The example uses a .xml
list of packages which you could adjust to what is needed at your site.

I also had a mirror for Ubuntu.   I recall using rsync excludes in
a mirror script, but that may have been used for Ubuntu.
> --

George N. White III

[stan]

On Sun, 5 Apr 2020 13:34:34 -0700
Geoffrey Leach  wrote:
> My internet service comes over a satellite, and with it a relatively
> small monthly download allowance. Which motivates the following
> question.
>
> Once I have installed a new disto and downloaded the RPMs that I use,
> is there a procedure by which I could gather together everything that
> I have added, so that I could transfer the files (RPMs, or whatever)
> to a local system, without resorting to the internet? (Or, at least,
> to a significant amount!)

If the two systems are the same version of fedora (or even different
versions if they use the same dnf layout), you can put 
keepcache=1
in /etc/dnf/dnf.conf to keep the rpms after they are installed.  See
the man page for dnf.conf for an explanation.  Those rpms are kept in
directories like /var/cache/dnf/fedora-[hash]/packages/, with a
directory for each file in /etc/yum.repos.d.  You can then copy them to
the other machine's similar directory, and they will reside there until
you decide to remove them.

If the cache gets too full with the keepcache option, you can go into
the directory and remove the packages individually, or you can run
dnf clean packages
to remove them from every cache.

But, I have to wonder why you are doing this.  Unless you are updating
another machine, and want to save the bandwidth from happening twice,
once the packages are installed, you should never need them again on
the same machine.  That's why the default for keepcache is 0, so that
cleanup occurs whenever a successful update happens.

This is a lot simpler than setting up a local repository.

[stan]

On Sun, 5 Apr 2020 18:10:00 -0700
stan  wrote:
> On Sun, 5 Apr 2020 13:34:34 -0700
> Geoffrey Leach  wrote:
> > My internet service comes over a satellite, and with it a relatively
> > small monthly download allowance. Which motivates the following
> > question.
> >
> > Once I have installed a new disto and downloaded the RPMs that I
> > use, is there a procedure by which I could gather together
> > everything that I have added, so that I could transfer the files
> > (RPMs, or whatever) to a local system, without resorting to the
> > internet? (Or, at least, to a significant amount!)
>
> If the two systems are the same version of fedora (or even different
> versions if they use the same dnf layout), you can put 
> keepcache=1
> in /etc/dnf/dnf.conf to keep the rpms after they are installed.  See
> the man page for dnf.conf for an explanation.  Those rpms are kept in
> directories like /var/cache/dnf/fedora-[hash]/packages/, with a
> directory for each file in /etc/yum.repos.d.  You can then copy them
> to the other machine's similar directory, and they will reside there
> until you decide to remove them.
>
> If the cache gets too full with the keepcache option, you can go into
> the directory and remove the packages individually, or you can run
> dnf clean packages
> to remove them from every cache.
>
> But, I have to wonder why you are doing this.  Unless you are updating
> another machine, and want to save the bandwidth from happening twice,
> once the packages are installed, you should never need them again on
> the same machine.  That's why the default for keepcache is 0, so that
> cleanup occurs whenever a successful update happens.
>
> This is a lot simpler than setting up a local repository.

I forgot to say that the rpms are signed, so if you want to use them on
an older system, that system will need to have its fedora-gpg-keys
package, and probably its fedora-repos package, updated to at least the
version of the system that installed them.  Usually simple dnf update
will work for version n-1, but beyond that you will have to use --force
with rpm, or turn off gpg checking in dnf, because the system dnf won't
have the keys to check the updates.

[Geoffrey Leach]

No, not identical.

On Sun, 5 Apr 2020 18:50:49 -0300
"George N. White III"  wrote:
> On Sun, 5 Apr 2020 at 17:35, Geoffrey Leach  wrote:
> > My internet service comes over a satellite, and with it a relatively
> > small monthly download allowance. Which motivates the following
> > question.
> >
> > Once I have installed a new disto and downloaded the RPMs that I
> > use, is there a procedure by which I could gather together
> > everything that I have added, so that I could transfer the files
> > (RPMs, or whatever) to a local system, without resorting to the
> > internet? (Or, at least, to a significant amount!)
>
> Is the "local system" identical to the one where you did the initial
> installation?
>
> In a former life, with RHEL versions 4--6, I used a local mirror for
> the packages needed to support our use cases.    When there were
> updates, they appeared in the mirror and systems updated from there.
>  I don't recall there being a lot of guidance for creating a local
> mirror, more a matter of trial and error to get a local mirror
> working with a small get of packages.
>
> https://fedoraproject.org/wiki/Docs/Drafts/MockSetupUsingLocalMirror
> seems to be roughly the same vintage.    The example uses a .xml
> list of packages which you could adjust to what is needed at your
> site.
>
> I also had a mirror for Ubuntu.   I recall using rsync excludes in
> a mirror script, but that may have been used for Ubuntu.
> > --
>
> George N. White III

[Geoffrey Leach]

On Sun, 5 Apr 2020 18:18:29 -0700
stan via users  wrote:
> On Sun, 5 Apr 2020 18:10:00 -0700
> stan  wrote:
> > On Sun, 5 Apr 2020 13:34:34 -0700
> > Geoffrey Leach  wrote:
> > > My internet service comes over a satellite, and with it a
> > > relatively small monthly download allowance. Which motivates the
> > > following question.
> > >
> > > Once I have installed a new disto and downloaded the RPMs that I
> > > use, is there a procedure by which I could gather together
> > > everything that I have added, so that I could transfer the files
> > > (RPMs, or whatever) to a local system, without resorting to the
> > > internet? (Or, at least, to a significant amount!)
> >
> > If the two systems are the same version of fedora (or even different
> > versions if they use the same dnf layout), you can put 
> > keepcache=1
> > in /etc/dnf/dnf.conf to keep the rpms after they are installed.  See
> > the man page for dnf.conf for an explanation.  Those rpms are kept
> > in directories like /var/cache/dnf/fedora-[hash]/packages/, with a
> > directory for each file in /etc/yum.repos.d.  You can then copy them
> > to the other machine's similar directory, and they will reside there
> > until you decide to remove them.
> >
> > If the cache gets too full with the keepcache option, you can go
> > into the directory and remove the packages individually, or you can
> > run dnf clean packages
> > to remove them from every cache.
> >
> > But, I have to wonder why you are doing this.  Unless you are
> > updating another machine, and want to save the bandwidth from
> > happening twice, once the packages are installed, you should never
> > need them again on the same machine.  That's why the default for
> > keepcache is 0, so that cleanup occurs whenever a successful update
> > happens.
> >
> > This is a lot simpler than setting up a local repository.
>
> I forgot to say that the rpms are signed, so if you want to use them
> on an older system, that system will need to have its fedora-gpg-keys
> package, and probably its fedora-repos package, updated to at least
> the version of the system that installed them.  Usually simple dnf
> update will work for version n-1, but beyond that you will have to
> use --force with rpm, or turn off gpg checking in dnf, because the
> system dnf won't have the keys to check the updates.

Saving bandwidth is the goal.

I'll be updating three systems with different hardware to the same
Fedora version. Sounds like keepcache is exactly what I need.

Thanks.

[Fabrice BAUZAC-STEHLY]

Hi there,

Geoffrey Leach writes:
> My internet service comes over a satellite, and with it a relatively
> small monthly download allowance. Which motivates the following
> question.
>
> Once I have installed a new disto and downloaded the RPMs that I use, is
> there a procedure by which I could gather together everything that I
> have added, so that I could transfer the files (RPMs, or whatever) to
> a local system, without resorting to the internet? (Or, at least, to a
> significant amount!)

In your situation I think I would look into setting up a caching proxy
such as Squid.  It should be able to cache many kinds of files you may
need on several of your machines, not only rpms downloaded by dnf but
maybe also some common webpages, javascript or images you download via a
browser from several machines.

This is a need I had for a set of 2-3 Debian systems, where I simply
installed a pre-made configuration dedicated to .deb packages [1].  I'm
not knowledgeable enough to update the configuration for caching HTML
and images though.

Does anybody have enough Squid knowledge to setup such a proxy for rpm
and more?

[1] https://packages.debian.org/en/jessie/squid-deb-proxy

Thanks!

Best regards

--
Fabrice BAUZAC-STEHLY
PGP 015AE9B25DCB0511D200A75DE5674DEA514C891D

[ja]

On Mon, 2020-04-06 at 07:20 +0200, Fabrice BAUZAC-STEHLY wrote:
> Hi there,
>
> Geoffrey Leach writes:
> > My internet service comes over a satellite, and with it a relatively
> > small monthly download allowance. Which motivates the following
> > question.
> >
> > Once I have installed a new disto and downloaded the RPMs that I use, is
> > there a procedure by which I could gather together everything that I
> > have added, so that I could transfer the files (RPMs, or whatever) to
> > a local system, without resorting to the internet? (Or, at least, to a
> > significant amount!)
>
> In your situation I think I would look into setting up a caching proxy
> such as Squid.  It should be able to cache many kinds of files you may
> need on several of your machines, not only rpms downloaded by dnf but
> maybe also some common webpages, javascript or images you download via a
> browser from several machines.
>
> This is a need I had for a set of 2-3 Debian systems, where I simply
> installed a pre-made configuration dedicated to .deb packages [1].  I'm
> not knowledgeable enough to update the configuration for caching HTML
> and images though.
>
> Does anybody have enough Squid knowledge to setup such a proxy for rpm
> and more?
>
> [1] https://packages.debian.org/en/jessie/squid-deb-proxy
>
> Thanks!
>
> Best regards

Not absolutely certain this is relevant to your case, ...

The approach I have been using recently is to  down load a live
image (XFCE in my case) and save on a fast USB stick.

Select something like "Install to Hard drive" when the Live image is booted.

The install from a live XFCE image to a "clean" SSD only takes 2 or 3 mins.

Boot the new installation.
To avoid immediate dnf update, disable ... (can't remember what)!!

//--------------------------------
Then create a local repo.
I use a network NFS repo site but I guess an external USB stick
could be used.

From my notes:

dnf install createrepo_c	#createrepo_c already present 	F31
To install a plugin
dnf install python3-dnf-plugins*  #Only installs python3-dnf-plugins-extras-common F31
dnf install python3-dnf-plugin-local		#The vital one	F31
//--------------------------------
Create Local Repo, ...
To automatically add all "directly installed" rpms to the /global/local_repos/31 repo

geany /etc/dnf/plugins/local.conf and add repodir = /global/local_repos/31 to the [main] section
Better?		Yes	F31
sed -i.ja.bak '+# repodir = /var/lib/dnf/plugins/local$+arepodir = /global/local_repos/31' \
/etc/dnf/plugins/local.conf
//--------------------------------
mkdir /global/local_repos/31				If not present		F31
NB the 'EOF_LOCAL_REPO' quotes disables parameter substitution within its body.
cat > /etc/yum.repos.d/local-31-update.repo << 'EOF_LOCAL_REPO'
[local-31-update]
name=Local ja Fedora $releasever $basearch Updates
cost=1
baseurl=file:///global/local_repos/$releasever/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$release-$basearch
metadata_expire=120
EOF_LOCAL_REPO
//-----------------------------------
Testing
dnf install pcmanfm
dnf config-manager --dump or dnf config-manager local-30-update --dump
dnf repoquery --repoid=local-31-update		Tested OK	F31

//-----------------------------------
dnf update           should then populate the local repo.
//-----------------------------------
All my machines are set up like this so that when dnf update is run
on what ever machine the local repo is updated and subsequent updates of
the same packages on other machines are taken from there.

John

[Tim]

On Mon, 2020-04-06 at 07:20 +0200, Fabrice BAUZAC-STEHLY wrote:
> In your situation I think I would look into setting up a caching
> proxy such as Squid.  It should be able to cache many kinds of files
> you may need on several of your machines, not only rpms downloaded by
> dnf but maybe also some common webpages, javascript or images you
> download via a browser from several machines.

Since each download from a Fedora installation could fetch RPMs from
different repo servers, and Squid will cache based on the whole
internet address of each file, you'd need to change the DNF config on
them to always use the same repo server.

I think you can configure DNF to go through a proxy, so you wouldn't
have to specially configure Squid, it'd just cache in its normal way.

I used to do that kind of thing for Windows updates, 20 years ago.  But
I'm not sure if it's worth going to this trouble with Fedora.  Once
installed, the next set of updates can be the small delta RPM files
(just the difference between the prior install and the bits of it that
are actually updated).  A lot of those delta files are very small.

I seem to recall that there was a package for easily setting up a local
repo service to do this kind of thing (one PC will act as your cache
for the rest, only downloading the wanted RPMs rather than mirroring
everything on the external repo).