Since I have >7 Fedora installations I setup a squid proxy server specifically for the purpose of reducing my download data since many of the same packages are going to get downloaded over and over again.
I seem to be getting some hits (still looking for a GOOD squid analyzer!) but not as much as I would expect. I'm no regex expert but I tried to update the regex to pick up delta RPMS as well and may have made a mistake...
Here's my modifications to the default squid conf file...
have not seen a larger RPM yet maximum_object_size 1 GB
# cache RPMs only acl rpm_only urlpath_regex .[d]?rpm cache allow rpm_only cache deny all cache_replacement_policy heap LFUDA
Any ideas?
Thanks, Richard
On Mon, 2019-09-02 at 08:53 -0500, Richard Shaw wrote:
Since I have >7 Fedora installations I setup a squid proxy server specifically for the purpose of reducing my download data since many of the same packages are going to get downloaded over and over again.
I seem to be getting some hits (still looking for a GOOD squid analyzer!) but not as much as I would expect. I'm no regex expert but I tried to update the regex to pick up delta RPMS as well and may have made a mistake...
To use a caching proxy to help with yum/dnf/packagekit, you need to ensure that those updaters always use the same repo mirror. So you need to configure your updater, first.
While configuring it (yum, dnf, etc) to use a particular mirror, you can set it to always go through your proxy.
You don't have to specially configure your proxy. It'll just cache anything going through it, unless you have some other rules to prevent it.
That's what I used to do. I set up my /etc/yum.conf files to always use the Linux mirror that my ISP provided, set them to use my Squid proxy, and just set Squid up to be a caching proxy in general.
On Mon, Sep 2, 2019 at 9:29 AM Tim via users users@lists.fedoraproject.org wrote:
On Mon, 2019-09-02 at 08:53 -0500, Richard Shaw wrote:
Since I have >7 Fedora installations I setup a squid proxy server specifically for the purpose of reducing my download data since many of the same packages are going to get downloaded over and over again.
I seem to be getting some hits (still looking for a GOOD squid analyzer!) but not as much as I would expect. I'm no regex expert but I tried to update the regex to pick up delta RPMS as well and may have made a mistake...
To use a caching proxy to help with yum/dnf/packagekit, you need to ensure that those updaters always use the same repo mirror. So you need to configure your updater, first.
I found some methods that use url rewriting but it's rather ugly...
One[1] suggested this:
/fedora/linux/releases/([0-9]+)/Everything/x86_64/(.*)$ http://repo.mirrors.squid.internal/fedora/releases/$1/$2 /fedora/linux/updates/([0-9]+)/x86_64/(.*)$ http://repo.mirrors.squid.internal/fedora/updates/$1/$2
There's got to be a proper and generic way of doing this. Keeping up with a list of mirrors isn't sustainable.
I'd be willing to create a new project to try and come up with a package that would automagically get this working but I'd need a lot of help...
Thanks, Richard
[1] https://serverfault.com/questions/837291/squid-and-caching-of-dnf-yum-downlo...
On 9/2/19 8:51 AM, Richard Shaw wrote:
/fedora/linux/releases/([0-9]+)/Everything/x86_64/(.*)$ http://repo.mirrors.squid.internal/fedora/releases/$1/$2 /fedora/linux/updates/([0-9]+)/x86_64/(.*)$ http://repo.mirrors.squid.internal/fedora/updates/$1/$2
There's got to be a proper and generic way of doing this. Keeping up with a list of mirrors isn't sustainable.
The bigger problem is that most mirrors default to using https which you can't cache with a proxy.
I'd be willing to create a new project to try and come up with a package that would automagically get this working but I'd need a lot of help...
I have a proxy that I made myself which is a rather ugly hack (but it mostly works). It returns an error for https attempts forcing dnf to retry with the http version.
What would really be useful is for a dnf plugin to be able to intercept download requests, but it doesn't appear to be possible. They go straight to curl. It's possible that I missed something, so if anyone knows differently, please let me know.
On Mon, Sep 2, 2019 at 3:26 PM Samuel Sieb samuel@sieb.net wrote:
On 9/2/19 8:51 AM, Richard Shaw wrote:
/fedora/linux/releases/([0-9]+)/Everything/x86_64/(.*)$ http://repo.mirrors.squid.internal/fedora/releases/$1/$2 /fedora/linux/updates/([0-9]+)/x86_64/(.*)$ http://repo.mirrors.squid.internal/fedora/updates/$1/$2
There's got to be a proper and generic way of doing this. Keeping up with a list of mirrors isn't sustainable.
The bigger problem is that most mirrors default to using https which you can't cache with a proxy.
Looks like it's possible but not simple...
https://elatov.github.io/2019/01/using-squid-to-proxy-ssl-sites/
Thanks, Richard
Would this be a good use for a local private mirror?
https://fedoraproject.org/wiki/Infrastructure/Mirroring#How_can_someone_make...
On Mon, Sep 2, 2019 at 9:54 AM Richard Shaw hobbes1069@gmail.com wrote:
Since I have >7 Fedora installations I setup a squid proxy server specifically for the purpose of reducing my download data since many of the same packages are going to get downloaded over and over again.
I seem to be getting some hits (still looking for a GOOD squid analyzer!) but not as much as I would expect. I'm no regex expert but I tried to update the regex to pick up delta RPMS as well and may have made a mistake...
Here's my modifications to the default squid conf file...
have not seen a larger RPM yet maximum_object_size 1 GB
# cache RPMs only acl rpm_only urlpath_regex .[d]?rpm cache allow rpm_only cache deny all cache_replacement_policy heap LFUDA
Any ideas?
Thanks, Richard _______________________________________________ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
On Wed, Sep 4, 2019 at 10:16 AM Ted Roche tedroche@gmail.com wrote:
Would this be a good use for a local private mirror?
https://fedoraproject.org/wiki/Infrastructure/Mirroring#How_can_someone_make...
It looked like you have some ability to control what gets mirrored but for the most part you get everything, so may easily be a net increase in data usage.
Thanks, Richard
On Wed, 4 Sep 2019 09:51:58 -0500 Richard Shaw hobbes1069@gmail.com wrote:
On Mon, Sep 2, 2019 at 3:26 PM Samuel Sieb samuel@sieb.net wrote:
On 9/2/19 8:51 AM, Richard Shaw wrote:
/fedora/linux/releases/([0-9]+)/Everything/x86_64/(.*)$ http://repo.mirrors.squid.internal/fedora/releases/$1/$2 /fedora/linux/updates/([0-9]+)/x86_64/(.*)$ http://repo.mirrors.squid.internal/fedora/updates/$1/$2
There's got to be a proper and generic way of doing this. Keeping up with a list of mirrors isn't sustainable.
The bigger problem is that most mirrors default to using https which you can't cache with a proxy.
Looks like it's possible but not simple...
https://elatov.github.io/2019/01/using-squid-to-proxy-ssl-sites/
A hack you might consider if the computers can exchange files. On one machine, the machine that will get the packages, turn off the switch that deletes packages after install in /etc/dnf/dnf.conf. Then, use rsync to put those packages in the appropriate directory on each of the dependent computers (something like var/cache/dnf/updates-[unique key]/)
Then you can update the dependent computers as normal, and they will get their own metadata, but when the update occurs, they will skip the download because they will see that the packages are already downloaded.
You could put the rsync commands in a script, and run the script after updating the master computer.
On 9/4/19 11:42 AM, stan via users wrote:
A hack you might consider if the computers can exchange files. On one machine, the machine that will get the packages, turn off the switch that deletes packages after install in /etc/dnf/dnf.conf. Then, use rsync to put those packages in the appropriate directory on each of the dependent computers (something like var/cache/dnf/updates-[unique key]/)
Then you can update the dependent computers as normal, and they will get their own metadata, but when the update occurs, they will skip the download because they will see that the packages are already downloaded.
You could put the rsync commands in a script, and run the script after updating the master computer.
I have used that in the past for doing system upgrades, but with general updates and a lot of computers, that gets unwieldy really fast.
On Mon, 2019-09-02 at 13:24 -0700, Samuel Sieb wrote:
The bigger problem is that most mirrors default to using https which you can't cache with a proxy.
Pick a mirror that doesn't use HTTPS.