Squid proxy setup for caching RPMs

[Richard Shaw]

Since I have >7 Fedora installations I setup a squid proxy server
specifically for the purpose of reducing my download data since many of the
same packages are going to get downloaded over and over again.

I seem to be getting some hits (still looking for a GOOD squid analyzer!)
but not as much as I would expect. I'm no regex expert but I tried to
update the regex to pick up delta RPMS as well and may have made a
mistake...

Here's my modifications to the default squid conf file...

have not seen a larger RPM yet
maximum_object_size 1 GB

# cache RPMs only
acl rpm_only urlpath_regex .[d]?rpm
cache allow rpm_only
cache deny all
cache_replacement_policy heap LFUDA

Any ideas?

Thanks,
Richard

[Tim]

On Mon, 2019-09-02 at 08:53 -0500, Richard Shaw wrote:
> Since I have >7 Fedora installations I setup a squid proxy server
> specifically for the purpose of reducing my download data since many
> of the same packages are going to get downloaded over and over again.
>
> I seem to be getting some hits (still looking for a GOOD squid
> analyzer!) but not as much as I would expect. I'm no regex expert but
> I tried to update the regex to pick up delta RPMS as well and may
> have made a mistake...

To use a caching proxy to help with yum/dnf/packagekit, you need to
ensure that those updaters always use the same repo mirror.  So you
need to configure your updater, first.

While configuring it (yum, dnf, etc) to use a particular mirror, you
can set it to always go through your proxy.

You don't have to specially configure your proxy.  It'll just cache
anything going through it, unless you have some other rules to prevent
it.

That's what I used to do.  I set up my /etc/yum.conf files to always
use the Linux mirror that my ISP provided, set them to use my Squid
proxy, and just set Squid up to be a caching proxy in general.

[Richard Shaw]

On Mon, Sep 2, 2019 at 9:29 AM Tim via users 
wrote:
> On Mon, 2019-09-02 at 08:53 -0500, Richard Shaw wrote:
> > Since I have >7 Fedora installations I setup a squid proxy server
> > specifically for the purpose of reducing my download data since many
> > of the same packages are going to get downloaded over and over again.
> >
> > I seem to be getting some hits (still looking for a GOOD squid
> > analyzer!) but not as much as I would expect. I'm no regex expert but
> > I tried to update the regex to pick up delta RPMS as well and may
> > have made a mistake...
>
> To use a caching proxy to help with yum/dnf/packagekit, you need to
> ensure that those updaters always use the same repo mirror.  So you
> need to configure your updater, first.

I found some methods that use url rewriting but it's rather ugly...

One[1] suggested this:

/fedora/linux/releases/([0-9]+)/Everything/x86_64/(.*)$
http://repo.mirrors.squid.internal/fedora/releases/$1/$2
/fedora/linux/updates/([0-9]+)/x86_64/(.*)$
http://repo.mirrors.squid.internal/fedora/updates/$1/$2

There's got to be a proper and generic way of doing this. Keeping up with a
list of mirrors isn't sustainable.

I'd be willing to create a new project to try and come up with a package
that would automagically get this working but I'd need a lot of help...

Thanks,
Richard

[1]
https://serverfault.com/questions/837291/squid-and-caching-of-dnf-yum-downlo...

[Samuel Sieb]

On 9/2/19 8:51 AM, Richard Shaw wrote:
> /fedora/linux/releases/([0-9]+)/Everything/x86_64/(.*)$ 
> http://repo.mirrors.squid.internal/fedora/releases/$1/$2
> /fedora/linux/updates/([0-9]+)/x86_64/(.*)$ 
> http://repo.mirrors.squid.internal/fedora/updates/$1/$2
>
> There's got to be a proper and generic way of doing this. Keeping up 
> with a list of mirrors isn't sustainable.

The bigger problem is that most mirrors default to using https which you 
can't cache with a proxy.
> I'd be willing to create a new project to try and come up with a package 
> that would automagically get this working but I'd need a lot of help...

I have a proxy that I made myself which is a rather ugly hack (but it 
mostly works).  It returns an error for https attempts forcing dnf to 
retry with the http version.

What would really be useful is for a dnf plugin to be able to intercept 
download requests, but it doesn't appear to be possible.  They go 
straight to curl.  It's possible that I missed something, so if anyone 
knows differently, please let me know.

[Richard Shaw]

On Mon, Sep 2, 2019 at 3:26 PM Samuel Sieb  wrote:
> On 9/2/19 8:51 AM, Richard Shaw wrote:
> > /fedora/linux/releases/([0-9]+)/Everything/x86_64/(.*)$
> > http://repo.mirrors.squid.internal/fedora/releases/$1/$2
> > /fedora/linux/updates/([0-9]+)/x86_64/(.*)$
> > http://repo.mirrors.squid.internal/fedora/updates/$1/$2
> >
> > There's got to be a proper and generic way of doing this. Keeping up
> > with a list of mirrors isn't sustainable.
>
> The bigger problem is that most mirrors default to using https which you
> can't cache with a proxy.

Looks like it's possible but not simple...

https://elatov.github.io/2019/01/using-squid-to-proxy-ssl-sites/

Thanks,
Richard

[Ted Roche]

Would this be a good use for a local private mirror?

https://fedoraproject.org/wiki/Infrastructure/Mirroring#How_can_someone_make...

On Mon, Sep 2, 2019 at 9:54 AM Richard Shaw  wrote:
> Since I have >7 Fedora installations I setup a squid proxy server
> specifically for the purpose of reducing my download data since many of the
> same packages are going to get downloaded over and over again.
>
> I seem to be getting some hits (still looking for a GOOD squid analyzer!)
> but not as much as I would expect. I'm no regex expert but I tried to
> update the regex to pick up delta RPMS as well and may have made a
> mistake...
>
> Here's my modifications to the default squid conf file...
>
> have not seen a larger RPM yet
> maximum_object_size 1 GB
>
> # cache RPMs only
> acl rpm_only urlpath_regex .[d]?rpm
> cache allow rpm_only
> cache deny all
> cache_replacement_policy heap LFUDA
>
> Any ideas?
>
> Thanks,
> Richard
> _______________________________________________
> users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Richard Shaw]

On Wed, Sep 4, 2019 at 10:16 AM Ted Roche  wrote:
> Would this be a good use for a local private mirror?
>
> https://fedoraproject.org/wiki/Infrastructure/Mirroring#How_can_someone_make...

It looked like you have some ability to control what gets mirrored but for
the most part you get everything, so may easily be a net increase in data
usage.

Thanks,
Richard

[stan]

On Wed, 4 Sep 2019 09:51:58 -0500
Richard Shaw  wrote:
> On Mon, Sep 2, 2019 at 3:26 PM Samuel Sieb  wrote:
> > On 9/2/19 8:51 AM, Richard Shaw wrote:
> > > /fedora/linux/releases/([0-9]+)/Everything/x86_64/(.*)$
> > > http://repo.mirrors.squid.internal/fedora/releases/$1/$2
> > > /fedora/linux/updates/([0-9]+)/x86_64/(.*)$
> > > http://repo.mirrors.squid.internal/fedora/updates/$1/$2
> > >
> > > There's got to be a proper and generic way of doing this. Keeping
> > > up with a list of mirrors isn't sustainable.
> >
> > The bigger problem is that most mirrors default to using https
> > which you can't cache with a proxy.
>
> Looks like it's possible but not simple...
>
> https://elatov.github.io/2019/01/using-squid-to-proxy-ssl-sites/

A hack you might consider if the computers can exchange files. On one
machine, the machine that will get the packages, turn off the switch
that deletes packages after install in /etc/dnf/dnf.conf.  Then, 
use rsync to put those packages in the appropriate directory on each of
the dependent computers (something like var/cache/dnf/updates-[unique
key]/)

Then you can update the dependent computers as normal, and they will get
their own metadata, but when the update occurs, they will skip the
download because they will see that the packages are already downloaded.

You could put the rsync commands in a script, and run the script after
updating the master computer.

[Samuel Sieb]

On 9/4/19 11:42 AM, stan via users wrote:
> A hack you might consider if the computers can exchange files. On one
> machine, the machine that will get the packages, turn off the switch
> that deletes packages after install in /etc/dnf/dnf.conf.  Then,
> use rsync to put those packages in the appropriate directory on each of
> the dependent computers (something like var/cache/dnf/updates-[unique
> key]/)
>
> Then you can update the dependent computers as normal, and they will get
> their own metadata, but when the update occurs, they will skip the
> download because they will see that the packages are already downloaded.
>
> You could put the rsync commands in a script, and run the script after
> updating the master computer.

I have used that in the past for doing system upgrades, but with general 
updates and a lot of computers, that gets unwieldy really fast.

[Tim]

On Mon, 2019-09-02 at 13:24 -0700, Samuel Sieb wrote:
> The bigger problem is that most mirrors default to using https which
> you can't cache with a proxy.

Pick a mirror that doesn't use HTTPS.