12:31 p.m.
Heya,
We're adding some more storage to our mirrors - how big is buffet?
--
Disclaimer: This message contains information which may be confidential and
privileged. Unless you are the intended recipient you may not use, copy,
disseminate or disclose to anyone the message or any information contained
in the message. Thank you.
3:50 p.m.
On Wed, Apr 29, 2020 at 04:44:08PM -0400, James Cassell wrote:
On Wed, Apr 29, 2020, at 1:31 PM, Theo Morra wrote:
> Heya,
>
> We're adding some more storage to our mirrors - how big is buffet?
>
Someone said 3.8 TB the other day.
Unfortunately I was off by one. It is 38TB. Knowing that, my proposal to
require mirroring fedora-buffet for new tier 1 mirrors sounds not so
good anymore as that is really a lot.
Adrian
7:22 p.m.
New subject: [EXT] Re: fedora-buffet
On Wed, Apr 29, 2020 at 10:50:00PM +0200, Adrian Reber wrote:
On Wed, Apr 29, 2020 at 04:44:08PM -0400, James Cassell wrote:
>
> On Wed, Apr 29, 2020, at 1:31 PM, Theo Morra wrote:
> > Heya,
> >
> > We're adding some more storage to our mirrors - how big is buffet?
> >
>
> Someone said 3.8 TB the other day.
Unfortunately I was off by one. It is 38TB. Knowing that, my proposal to
require mirroring fedora-buffet for new tier 1 mirrors sounds not so
good anymore as that is really a lot.
It is a lot, but I think it should still be required. Tier1 should
have everything any Tier2 might want to pull a subset of.
4:27 a.m.
New subject: [EXT] Re: fedora-buffet
On Wed, 29 Apr 2020 at 20:22, Chuck Anderson <cra(a)wpi.edu> wrote:
On Wed, Apr 29, 2020 at 10:50:00PM +0200, Adrian Reber wrote:
> On Wed, Apr 29, 2020 at 04:44:08PM -0400, James Cassell wrote:
> >
> > On Wed, Apr 29, 2020, at 1:31 PM, Theo Morra wrote:
> > > Heya,
> > >
> > > We're adding some more storage to our mirrors - how big is buffet?
> > >
> >
> > Someone said 3.8 TB the other day.
>
> Unfortunately I was off by one. It is 38TB. Knowing that, my proposal to
> require mirroring fedora-buffet for new tier 1 mirrors sounds not so
> good anymore as that is really a lot.
It is a lot, but I think it should still be required. Tier1 should
have everything any Tier2 might want to pull a subset of.
In the past I would agree, but we (aka Fedora) had to drop mirroring
archives from 2 of our systems (download-ib01/download-i2 and
download-cc-rdu01) because we didn't have the disk space on these
servers to do that much storage. I am not sure trying to shove every
use case for a top mirror into one box is going to work anymore.
At the moment I only see the following mirrors offering archives via
mirrormanager:
mirror.math.princeton.edu
pubmirror1.math.uh.edu
d2lzkl7pfhq30w.cloudfront.net # a fedora project caching server
dl.fedoraproject.org
I expect there are probably private ones which I am not seeing via my
simple curl |& grep but the majority of users who are grabbing stuff
from archived data are only getting it from those 4 places. While it
would be good for more, I can also see that the majority of mirror
traffic is EPEL driven with only the latest 3 Fedora's taking up the
majority of users. Having those fed fast is probably more important
than Fedora 18.
Would a different tiering system work better in the same way as we
have different rsync targets?
--
Stephen J Smoogen.
7:41 a.m.
New subject: [EXT] Re: fedora-buffet
On Thu, Apr 30, 2020 at 05:27:18AM -0400, Stephen John Smoogen wrote:
On Wed, 29 Apr 2020 at 20:22, Chuck Anderson <cra(a)wpi.edu>
wrote:
>
> On Wed, Apr 29, 2020 at 10:50:00PM +0200, Adrian Reber wrote:
> > On Wed, Apr 29, 2020 at 04:44:08PM -0400, James Cassell wrote:
> > >
> > > On Wed, Apr 29, 2020, at 1:31 PM, Theo Morra wrote:
> > > > Heya,
> > > >
> > > > We're adding some more storage to our mirrors - how big is
buffet?
> > > >
> > >
> > > Someone said 3.8 TB the other day.
> >
> > Unfortunately I was off by one. It is 38TB. Knowing that, my proposal to
> > require mirroring fedora-buffet for new tier 1 mirrors sounds not so
> > good anymore as that is really a lot.
>
> It is a lot, but I think it should still be required. Tier1 should
> have everything any Tier2 might want to pull a subset of.
In the past I would agree, but we (aka Fedora) had to drop mirroring
archives from 2 of our systems (download-ib01/download-i2 and
download-cc-rdu01) because we didn't have the disk space on these
servers to do that much storage. I am not sure trying to shove every
use case for a top mirror into one box is going to work anymore.
At the moment I only see the following mirrors offering archives via
mirrormanager:
mirror.math.princeton.edu
pubmirror1.math.uh.edu
d2lzkl7pfhq30w.cloudfront.net # a fedora project caching server
dl.fedoraproject.org
I see a couple more from my location:
ftp-stud.hs-esslingen.de
fedora-archive.ip-connect.info
mirrors.dotsrc.org
fedora-archive.ip-connect.vn.ua
fedora-mirror02.rbc.ru
I asked MirrorManager for Fedora 26 based mirrors, so this does not mean
that all these mirrors mirror the complete archive.
I expect there are probably private ones which I am not seeing via
my
simple curl |& grep but the majority of users who are grabbing stuff
from archived data are only getting it from those 4 places. While it
would be good for more, I can also see that the majority of mirror
traffic is EPEL driven with only the latest 3 Fedora's taking up the
majority of users. Having those fed fast is probably more important
than Fedora 18.
Would a different tiering system work better in the same way as we
have different rsync targets?
What I would love to see in any tiering system we set up is full support
for quick-fedora-mirror. If quick-fedora-mirror could handle a partial
tree (not only via fedora-buffet) that would be perfect.
Adrian
1446
days inactive
1451
days old
mirror-admin@lists.fedoraproject.org
5 comments
5 participants
participants (5)
-
Adrian Reber -
Chuck Anderson -
James Cassell -
Stephen John Smoogen -
Theo Morra