What is the problem with the mirrors?

[John Ellson]

Few of the Fedora mirrors have March 19th updates yet, never mind March 
22nd.

What is the problem?

[Jakub Jelinek]

On Tue, Mar 23, 2004 at 09:22:03AM -0500, John Ellson wrote:
>  Few of the Fedora mirrors have March 19th updates yet, never mind
> March 
>  22nd. 
There aren't many March 22 changes, just rpmdb-redhat...
Several mirrors have even March 22 changes.  What arch are you looking for?

	Jakub

[John Ellson]

Jakub Jelinek wrote:

> On Tue, Mar 23, 2004 at 09:22:03AM -0500, John Ellson wrote:
>   
>
> >Few of the Fedora mirrors have March 19th updates yet, never mind March 
> >22nd.
> >    
> >
>
> There aren't many March 22 changes, just rpmdb-redhat...
> Several mirrors have even March 22 changes.  What arch are you looking for?
>
> 	Jakub
>
>
>   
>  
i386

ibiblio doesn't have any rpmdb

kernel.org has rpmdb for March 19, but not March 22.

duke seems to have died altogether

redhat mirrors are impossible to log into this morning

John

[seth vidal]

>  >
>  i386
>  
>  ibiblio doesn't have any rpmdb
>  
>  kernel.org has rpmdb for March 19, but not March 22.
>  
>  duke seems to have died altogether
>  
>  redhat mirrors are impossible to log into this morning 
The problem is simple.

1. LOTS of changes last week to development
2. 5KB/s download rates from the mirror master to mirrors
3. everyone in the world wanting everything updated at any given second.


It's just not possible to count on that much data synced world-wide if
any one of the mirrors can't get any rates.

oh and 'duke seems to have died' is b/c a world of people repointed
their rsyncs to us when their download rates off of the red hat master
dropped to modem speeds.

The theory was that I was somehow magically able to get the bits. The
theory is wrong but...

-sv

[John Ellson]

seth vidal wrote:

> >i386
> >
> >ibiblio doesn't have any rpmdb
> >
> >kernel.org has rpmdb for March 19, but not March 22.
> >
> >duke seems to have died altogether
> >
> >redhat mirrors are impossible to log into this morning
> >    
> >
>
> The problem is simple.
>
> 1. LOTS of changes last week to development
> 2. 5KB/s download rates from the mirror master to mirrors
> 3. everyone in the world wanting everything updated at any given second.
>
>
> It's just not possible to count on that much data synced world-wide if
> any one of the mirrors can't get any rates.
>
> oh and 'duke seems to have died' is b/c a world of people repointed
> their rsyncs to us when their download rates off of the red hat master
> dropped to modem speeds.
>
> The theory was that I was somehow magically able to get the bits. The
> theory is wrong but...
>
> -sv
>
>
>
>   
>  
Isn't the solution also simple?

Give the mirror sites a priority access somehow, or perhaps a private 
master server?


John

[seth vidal]

>  Isn't the solution also simple?
>  
>  Give the mirror sites a priority access somehow, or perhaps a private 
>  master server? 
Can you keep a secret?

We are getting priority access.

Something is "Not Right"(TM)

-sv

[Jack Neely]

On Tue, Mar 23, 2004 at 10:14:14AM -0500, seth vidal wrote:
>  > Isn't the solution also simple?
>  > 
>  > Give the mirror sites a priority access somehow, or perhaps a private 
>  > master server?
>  
>  Can you keep a secret?
>  
>  We are getting priority access.
>  
>  Something is "Not Right"(TM)
>  
>  -sv
>  
>  
>  
>  -- 
>  fedora-devel-list mailing list
>  fedora-devel-list(a)redhat.com
>  http://www.redhat.com/mailman/listinfo/fedora-devel-list
>   
The master server on I-2 seems to have "gone away".  Its not in DNS
anymore.  I don't have a master to sync to that's anywhere near
ftp.linux.ncsu.edu.  

I can sync...only if I shut down my FTP site so I have no load and sync
for *days* at a time.  Normally, all I get is rsync error messages.

Jack

-- 
Jack Neely <slack(a)quackmaster.net&gt;
Realm Linux Administration and Development
PAMS Computer Operations at NC State University
GPG Fingerprint: 1917 5AC1 E828 9337 7AA4  EA6B 213B 765F 3B6A 5B89

[Jack Neely]

Looks like the I-2 master will be back up today...I'll try to get a good
sync as soon as possible.

Jack
-- 
Jack Neely <slack(a)quackmaster.net&gt;
Realm Linux Administration and Development
PAMS Computer Operations at NC State University
GPG Fingerprint: 1917 5AC1 E828 9337 7AA4  EA6B 213B 765F 3B6A 5B89

[Ryan Dooley]

[*snip*]

>  >It's just not possible to count on that much data synced
> world-wide if
>  >any one of the mirrors can't get any rates. 
I knew 300 baud would come back in style ;-)

[*snip*]

>  Isn't the solution also simple?
>  
>  Give the mirror sites a priority access somehow, or perhaps a private
>  master server? 
Yup... let the global mirrors talk with a private rsync server with a
username/password combo.

Ryan

[Chris Adams]

Once upon a time, John Ellson <ellson(a)research.att.com&gt; said:
>  Give the mirror sites a priority access somehow, or perhaps a private
>
>  master server? 
The mirror sites have priority access, but there is something afoot with
the Red Hat master servers, so that priority access results in modem
download speeds.

-- 
Chris Adams <cmadams(a)hiwaay.net&gt;
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.

[Volodymyr M. Lisivka]

вт, 2004-03-23 у 16:53, seth vidal написав:

>  The problem is simple.
>  
>  1. LOTS of changes last week to development 
rpmdelta and subversion?

http://toast.debian.net/~may/rpmdelta/

-- 
Best regards, Volodymyr M. Lisivka
icq#14549856

[seth vidal]

On Tue, 2004-03-23 at 11:07, Volodymyr M. Lisivka wrote:
>  вт, 2004-03-23 у 16:53, seth vidal написав:
>  
>  > The problem is simple.
>  > 
>  > 1. LOTS of changes last week to development
>  rpmdelta and subversion? 
We still need all the data on the mirror servers.


-sv

[Volodymyr M. Lisivka]

вт, 2004-03-23 у 18:22, seth vidal написав:
>  On Tue, 2004-03-23 at 11:07, Volodymyr M. Lisivka wrote:
>  > п╡я┌, 2004-03-23 я┐ 16:53, seth vidal пҐп╟п©п╦я│п╟п╡:
>  > 
>  > > The problem is simple.
>  > > 
>  > > 1. LOTS of changes last week to development
>  > rpmdelta and subversion?
>  
>  We still need all the data on the mirror servers. 
Quick list of advantages/disadvantages of the new method:

 + only differences between versions of compiled RPM will be transfered
(rpmdelta <= RPM);
 + can get any version of the RPM at any time;
 - two scripts needed to store/restore RPM in/from repository;
 - can't reexport SVN repository (+ if do rsync on SVN repository
itself);
 - need to keep all files on the server to apply deltas;
 - can't use apt/yum directly.

PS.
Even just plain rpm-deltas will be much better than simple downloading
of the whole RPM every time when it changed (eg. size of a quick
security fix or translation update in libc will be only 1/10000 to size
of the whole libc package).

-- 
Best regards, Volodymyr M. Lisivka
icq#14549856

[Michael Schwendt]

On 23 Mar 2004 09:53:26 -0500, seth vidal wrote:

>  The problem is simple.
>  
>  1. LOTS of changes last week to development
>  2. 5KB/s download rates from the mirror master to mirrors
>  3. everyone in the world wanting everything updated at any given second. 
4. many users in the world switch to the master server when a
mirror has yesterday's packages only

[Jakub Jelinek]

On Tue, Mar 23, 2004 at 09:41:39AM -0500, John Ellson wrote:
>  >There aren't many March 22 changes, just rpmdb-redhat...
>  >Several mirrors have even March 22 changes.  What arch are you looking for? 
>  i386
>  
>  ibiblio doesn't have any rpmdb
>  
>  kernel.org has rpmdb for March 19, but not March 22.
>  
>  duke seems to have died altogether
>  
>  redhat mirrors are impossible to log into this morning 
If you want i386, then
{ftp,http}://sunsite.mff.cuni.cz/pub/fedora/development/
has everything up to and including March 22. Ditto ia64.
Other arches are not 100% up2date though, since the current download
speed from master is not very good and if you have ~ 35GB of
constantly changing rawhide...

	Jakub

[Noa Resare]

tis 2004-03-23 klockan 16.03 skrev Jakub Jelinek:
>  On Tue, Mar 23, 2004 at 09:41:39AM -0500, John Ellson wrote:
>  > >There aren't many March 22 changes, just rpmdb-redhat...
>  > >Several mirrors have even March 22 changes.  What arch are you looking for?
>  
>  > i386
>  > 
>  > ibiblio doesn't have any rpmdb
>  > 
>  > kernel.org has rpmdb for March 19, but not March 22.
>  > 
>  > duke seems to have died altogether
>  > 
>  > redhat mirrors are impossible to log into this morning
>  
>  If you want i386, then
>  {ftp,http}://sunsite.mff.cuni.cz/pub/fedora/development/
>  has everything up to and including March 22. Ditto ia64.
>  Other arches are not 100% up2date though, since the current download
>  speed from master is not very good and if you have ~ 35GB of
>  constantly changing rawhide... 

I think it's obvious from this that development/rawhide should be
distribtuted independently from the released fredora trees. Having
stable updates distribution being delayed for days because all mirrors
are busy downloading contless gigabyes of development updates to
architectures very few uses seems like a problem that should get some
priority in fixing.

With numerous well connected mirrors as well as lots of bandwith to the
master site there should be no problem creating a rock solid fast
mirroring system with a little work.

/noa

-- 
And the lions ate the christians and the christians burned the witches,
and even I am out of explanations -- Ola Salo
gpg fingerprint: F3C4 AC90 B885 FE15 344B  4D05 220B 7662 A190 6F09

[John Ellson]

Noa Resare wrote:

> tis 2004-03-23 klockan 16.03 skrev Jakub Jelinek:
>   
>
> >On Tue, Mar 23, 2004 at 09:41:39AM -0500, John Ellson wrote:
> >    
> >
> >>>There aren't many March 22 changes, just rpmdb-redhat...
> >>>Several mirrors have even March 22 changes.  What arch are you looking for?
> >>>        
> >>>
> >>i386
> >>
> >>ibiblio doesn't have any rpmdb
> >>
> >>kernel.org has rpmdb for March 19, but not March 22.
> >>
> >>duke seems to have died altogether
> >>
> >>redhat mirrors are impossible to log into this morning
> >>      
> >>
> >If you want i386, then
> >{ftp,http}://sunsite.mff.cuni.cz/pub/fedora/development/
> >has everything up to and including March 22. Ditto ia64.
> >Other arches are not 100% up2date though, since the current download
> >speed from master is not very good and if you have ~ 35GB of
> >constantly changing rawhide...
> >    
> >
>
>
> I think it's obvious from this that development/rawhide should be
> distribtuted independently from the released fredora trees. Having
> stable updates distribution being delayed for days because all mirrors
> are busy downloading contless gigabyes of development updates to
> architectures very few uses seems like a problem that should get some
> priority in fixing.
>
> With numerous well connected mirrors as well as lots of bandwith to the
> master site there should be no problem creating a rock solid fast
> mirroring system with a little work.
>
> /noa
>
>   
>  
I guess I don't see the need to separate.  If the mirroring system was 
able to handle the large number
of development updates in a timely fashion then it would presumably have 
little problem with a small
number of stable-release updates.

The problem as I understand it is with the master server(s) that supply 
private/prioritized access to the mirrors.

John

[Dan Goodes]

On Tue, 23 Mar 2004 at 16:34, Noa Resare wrote:

>  I think it's obvious from this that development/rawhide should
> be
>  distribtuted independently from the released fredora trees. Having
>  stable updates distribution being delayed for days because all mirrors
>  are busy downloading contless gigabyes of development updates to
>  architectures very few uses seems like a problem that should get some
>  priority in fixing. 
I've had a setup that fetches "released" updates for the core tree
separately from the development tree, for some time (that is, one sync
that runs for /pub/fedora/linux/core/updates/ and a separate tree that
runs for /pub/fedora/linux/core/development/ at different times).

It helps to a degree, but I agree with the many posts on this thread that
something maybe "up" with redhat/fedora's "priority access" setup.
It's
difficult to get good speeds at the best of times, and at release times
(such as next monday/tuesday when -test2 is released) it's nigh-on
_impossible_ to get anything decent from the master server/s. I usually
end up talking with other well-connected mirrors to get IP-based access to
their mirrors so I can have the release before it's announced.

>  With numerous well connected mirrors as well as lots of bandwith to
> the
>  master site there should be no problem creating a rock solid fast
>  mirroring system with a little work. 
Sure - and for what it's worth, the RedHat / Fedora mirroring system is
among the most solid that I'm involved with.

Regards,

Dan Goodes  :  Systems Programmer  :  dang(a)planetmirror.com

Help support PlanetMirror - Australia's largest Internet archive
by signing up for PlanetMirror Premium : http://planetmirror.com

[Gene C.]

On Tuesday 23 March 2004 09:30, Jakub Jelinek wrote:
>  On Tue, Mar 23, 2004 at 09:22:03AM -0500, John Ellson wrote:
>  > Few of the Fedora mirrors have March 19th updates yet, never mind March
>  > 22nd.
>
>  There aren't many March 22 changes, just rpmdb-redhat...
>  Several mirrors have even March 22 changes.  What arch are you looking for? 
I know I have been having a problem finding a recent version for x86_64.  
Although some mirrors have the updates to Fedora/RPMS/*, they do not have 
anything (or anything current) for images/boot.iso or Fedora/base/*
-- 
Gene