Please hardlink archive/epel and epel

[Anderson, Charles R]

My disk filled up due to new copies of epel appearing under archive.
Can someone please hardlink this content from epel to archive/epel?

Thanks.

[Stephen John Smoogen]

On Sat, 27 Jul 2019 at 01:00, Anderson, Charles R <cra(a)wpi.edu&gt; wrote:

>  My disk filled up due to new copies of epel appearing under archive.
>  Can someone please hardlink this content from epel to archive/epel?
>
>  
OK my sincere apologies. I broke a lot of promises in doing this.

I did not announce this before hand
I did not announce after words.
I did not put in a ticket in our work queues to cover this.

I did a bunch of copying of packages over to archives at the end of May. I
do all the copies using cp -l to preserve links. I then ran the
update-archives for full file list but then did not see it did not work. I
did not announce this

This we got several requests from mirrors to get rid of f28 out of normal
space. I confirmed that f28 was copied over and then I synced over f29 and
f30 so I would not be behind in a couple of months. All of these were also
done with cp -l also and I checked that the hardlink numbers had increased
on files. I gave the info to Adrian and he let me know that the
update-archives was not updated. I then fixed that and again didnt send any
email about this.

So at this point I have majorly screwed you guys over 4 times.



>  Thanks.
>  _______________________________________________
>  Mirror-admin mailing list -- mirror-admin(a)lists.fedoraproject.org
>  To unsubscribe send an email to mirror-admin-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  

-- 
Stephen J Smoogen.

[Anderson, Charles R]

On Sat, Jul 27, 2019 at 10:04:06AM -0400, Stephen John Smoogen wrote:
>  I did a bunch of copying of packages over to archives at the end of
> May. I
>  do all the copies using cp -l to preserve links. I then ran the
>  update-archives for full file list but then did not see it did not work.
>  
>  This we got several requests from mirrors to get rid of f28 out of normal
>  space. I confirmed that f28 was copied over and then I synced over f29 and
>  f30 so I would not be behind in a couple of months. All of these were also
>  done with cp -l also and I checked that the hardlink numbers had increased
>  on files. I gave the info to Adrian and he let me know that the
>  update-archives was not updated. I then fixed that and again didnt send any
>  email about this. 
I'm not sure whether q-f-m didn't work quite right, whether there was
a server-side issue (update-archives?), or whether this was just all
due to timing of cp -l vs. update-archives vs. my q-f-m runs, but
somehow I received over 1 TB of downloads that were not hardlinked.
It did not finish (I ran out of disk space) so I cleaned that up
yesterday, moved the files from .~tmp/* to their final locations, ran
a local hardlink.py, and re-ran q-f-m.  The hardlink restored the 1+
TB of free space.  The files are all staying hardlinked, but due to
q-f-m optimizing the rsync runs, that doesn't provide any assurance
that the master mirror is as hardlinked as it could be.

To focus on the good parts:

My local hardlink.py run saved at least 600 GB additional space
compared to before Friday.  I think some of the additional savings
comes from internal hardlinking between e.g. Workstation, Everything,
Spins, noarch across different arch directories, etc. that one might
expect to be already hardlinked, but weren't for some reason.

I'm going to look into running quick-fedora-harlink locally regularly.

[Stephen John Smoogen]

On Sun, 28 Jul 2019 at 13:17, Anderson, Charles R <cra(a)wpi.edu&gt; wrote:

>  On Sat, Jul 27, 2019 at 10:04:06AM -0400, Stephen John Smoogen
> wrote:
>  > I did a bunch of copying of packages over to archives at the end of May.
>  I
>  > do all the copies using cp -l to preserve links. I then ran the
>  > update-archives for full file list but then did not see it did not work.
>  >
>  > This we got several requests from mirrors to get rid of f28 out of normal
>  > space. I confirmed that f28 was copied over and then I synced over f29
>  and
>  > f30 so I would not be behind in a couple of months. All of these were
>  also
>  > done with cp -l also and I checked that the hardlink numbers had
>  increased
>  > on files. I gave the info to Adrian and he let me know that the
>  > update-archives was not updated. I then fixed that and again didnt send
>  any
>  > email about this.
>
>  I'm not sure whether q-f-m didn't work quite right, whether there was
>  a server-side issue (update-archives?), or whether this was just all
>  due to timing of cp -l vs. update-archives vs. my q-f-m runs, but
>  somehow I received over 1 TB of downloads that were not hardlinked.
>  It did not finish (I ran out of disk space) so I cleaned that up
>  yesterday, moved the files from .~tmp/* to their final locations, ran
>  a local hardlink.py, and re-ran q-f-m.  The hardlink restored the 1+
>  TB of free space.  The files are all staying hardlinked, but due to
>  q-f-m optimizing the rsync runs, that doesn't provide any assurance
>  that the master mirror is as hardlinked as it could be.
>
>  To focus on the good parts:
>
>  My local hardlink.py run saved at least 600 GB additional space
>  compared to before Friday.  I think some of the additional savings
>  comes from internal hardlinking between e.g. Workstation, Everything,
>  Spins, noarch across different arch directories, etc. that one might
>  expect to be already hardlinked, but weren't for some reason.
>
>  I'm going to look into running quick-fedora-harlink locally regularly.
>  

So I am going to look at using the hardlink command with a 'do not do this,
just tell me what you would do' option first. I will report to the list
what I find. This would be with the hardlink we normally ship, but I was
wondering if anyone has any new versions they recommend.


>  _______________________________________________
>  Mirror-admin mailing list -- mirror-admin(a)lists.fedoraproject.org
>  To unsubscribe send an email to mirror-admin-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]...
>  

-- 
Stephen J Smoogen.

[Anderson, Charles R]

On Sun, Jul 28, 2019 at 01:21:16PM -0400, Stephen John Smoogen wrote:
>  On Sun, 28 Jul 2019 at 13:17, Anderson, Charles R <cra(a)wpi.edu&gt;
> wrote:
>  
>  > On Sat, Jul 27, 2019 at 10:04:06AM -0400, Stephen John Smoogen wrote:
>  > > I did a bunch of copying of packages over to archives at the end of May.
>  > I
>  > > do all the copies using cp -l to preserve links. I then ran the
>  > > update-archives for full file list but then did not see it did not work.
>  > >
>  > > This we got several requests from mirrors to get rid of f28 out of normal
>  > > space. I confirmed that f28 was copied over and then I synced over f29
>  > and
>  > > f30 so I would not be behind in a couple of months. All of these were
>  > also
>  > > done with cp -l also and I checked that the hardlink numbers had
>  > increased
>  > > on files. I gave the info to Adrian and he let me know that the
>  > > update-archives was not updated. I then fixed that and again didnt send
>  > any
>  > > email about this.
>  >
>  > I'm not sure whether q-f-m didn't work quite right, whether there was
>  > a server-side issue (update-archives?), or whether this was just all
>  > due to timing of cp -l vs. update-archives vs. my q-f-m runs, but
>  > somehow I received over 1 TB of downloads that were not hardlinked.
>  > It did not finish (I ran out of disk space) so I cleaned that up
>  > yesterday, moved the files from .~tmp/* to their final locations, ran
>  > a local hardlink.py, and re-ran q-f-m.  The hardlink restored the 1+
>  > TB of free space.  The files are all staying hardlinked, but due to
>  > q-f-m optimizing the rsync runs, that doesn't provide any assurance
>  > that the master mirror is as hardlinked as it could be.
>  >
>  > To focus on the good parts:
>  >
>  > My local hardlink.py run saved at least 600 GB additional space
>  > compared to before Friday.  I think some of the additional savings
>  > comes from internal hardlinking between e.g. Workstation, Everything,
>  > Spins, noarch across different arch directories, etc. that one might
>  > expect to be already hardlinked, but weren't for some reason.
>  >
>  > I'm going to look into running quick-fedora-harlink locally regularly.
>  >
>  
>  
>  So I am going to look at using the hardlink command with a 'do not do this,
>  just tell me what you would do' option first. I will report to the list
>  what I find. This would be with the hardlink we normally ship, but I was
>  wondering if anyone has any new versions they recommend. 
I know of at least 5 implementations.  Despite hardlink.py claiming to be faster than the
original C version, it still took over 12 hours when I ran it yesterday.

- hardlink (the C version we ship in Fedora).
- hardlink in util-linux (maybe the same as above).
- hardlink from Debian that may end up being adopted into util-linux (see
https://github.com/karelzak/util-linux/issues/808).
- hardlink.py by John Villalovos that I used
(https://code.google.com/archive/p/hardlinkpy/, there appear to be forks on github).

# John Villalovos
# email: john(a)sodarock.com
# http://www.sodarock.com/
#
# Inspiration for this program came from the hardlink.c code. I liked what it
# did but did not like the code itself, to me it was very unmaintainable.  So I
# rewrote in C++ and then I rewrote it in python.  In reality this code is
# nothing like the original hardlink.c, since I do things quite differently.
# Even though this code is written in python the performance of the python
# version is much faster than the hardlink.c code, in my limited testing.  This
# is mainly due to use of different algorithms.
#
# Original inspirational hardlink.c code was written by:  Jakub Jelinek
# <jakub(a)redhat.com&gt;

- quick-fedora-hardlink
(https://docs.pagure.org/quick-fedora-mirror/quick-fedora-hardlink.rst) that can run much
quicker using the filelists in the repos.

[Jason L Tibbitts III]

> >>>> "ACR" == Anderson, Charles R
> <cra(a)wpi.edu&gt; writes: 
ACR> I'm not sure whether q-f-m didn't work quite right, whether there
ACR> was a server-side issue (update-archives?), or whether this was
ACR> just all due to timing of cp -l vs. update-archives vs. my q-f-m
ACR> runs, but somehow I received over 1 TB of downloads that were not
ACR> hardlinked.

Here's the issue, put simply:

Currently quick-fedora-mirror will copy hardlinks between rsync modules
as hardlinks if, and only if, it sees the changes in the file lists for
both modules in the same run.

If you hardlink, and then don't update the file lists of both modules
simulteneously, then q-f-m will see a bunch of timestamp updates in one
module on one run, and then in a subsequent run it will see a bunch of
new files in a different module.  (The same thing in the reverse order.)

The file lists include a timestamp, which is the newer of mtime and
ctime.  Hardlinking will update the timestamp on both files, which is
how it detects that hardlinks were made at all.  Then it tells rsync to
transfer both linked files and (due to rsync's '-H' flag) they get
transferred as links.

I have a strategy for working around this, but I haven't implemented it
yet.

ACR> I'm going to look into running quick-fedora-harlink locally
ACR> regularly.

It should always be safe to run.  Because links can always be created or
broken on the server in ways that might not be picked up by q-f-m, it's
not a bad idea to run it occasionally.  It tries to be "faster" by using
the information in the file lists, but there's still so much stuff in
there that it takes a while.  (For more fun, there are even files with
identical names, sizes, and timestamps but not identical content.)

 - J<

[Jason L Tibbitts III]

> >>>> "SJS" == Stephen John Smoogen
> <smooge(a)gmail.com&gt; writes: 
SJS> So I am going to look at using the hardlink command with a 'do not
SJS> do this, just tell me what you would do' option first. I will
SJS> report to the list what I find. This would be with the hardlink we
SJS> normally ship, but I was wondering if anyone has any new versions
SJS> they recommend.

Well, do note that the hardlinker in quick-fedora-mirror tries to use
the file lists to make things faster.  There is an 'ignore timestamps'
mode intended for use on the mirror server.  Then the primary
optimization is the assumption that linked files will have identical
filenames, which I don't think a general hardlink program will do.

 - J<

[Japheth Cleaver]

On 7/29/2019 11:53 AM, Jason L Tibbitts III wrote:
>  "SJS" == Stephen John Smoogen <smooge(a)gmail.com&gt;
> writes:
>
>  SJS> So I am going to look at using the hardlink command with a 'do not
>  SJS> do this, just tell me what you would do' option first. I will
>  SJS> report to the list what I find. This would be with the hardlink we
>  SJS> normally ship, but I was wondering if anyone has any new versions
>  SJS> they recommend.
>
>  Well, do note that the hardlinker in quick-fedora-mirror tries to use
>  the file lists to make things faster.  There is an 'ignore timestamps'
>  mode intended for use on the mirror server.  Then the primary
>  optimization is the assumption that linked files will have identical
>  filenames, which I don't think a general hardlink program will do.
>
>    - J< 
Would it be possible to convert some of the hardlinks to softlinks at 
the generating source? That seems like it would help make this process 
much easier for rsync users overall.

This would especially be the helpful for the few entire directories... 
Not sure if it's still the case, but IIRC 
rawhide/Everything/source/tree/ and rawhide/Everything/SRPMS/ at one 
point seemed functionally identical.

-jc

[Jason L Tibbitts III]

> >>>> "JC" == Japheth Cleaver
> <cleaver(a)terabithia.org&gt; writes: 
JC> Would it be possible to convert some of the hardlinks to softlinks
JC> at the generating source? That seems like it would help make this
JC> process much easier for rsync users overall.

I think in general that would cause more issues than it would solve.
For example, which of fedora and fedora-secondary is the "master" tree that gets
the
noarch RPMS?  If I want to mirror just one, I'd have to figure out how
to convert those symlinks back.

That said, if entire directory trees are expected to have the same
content in the long term, then it does make much more sense for one to
be a simple symlink.  But that wouldn't cover the archive, so I think
its utility would be limited.

 - J<