Re: Rawhide 20080328 Snapshot Released

[John Summerfield]

I replied to fedora-list but I'm expecting a bounce any moment, I 
unsubscribe from it. This seems a better place anyway.

Jesse Keating wrote:
>  As part of our development schedule, we are releasing a snapshot of
>  Rawhide in iso and Live form.  We are releasing these via bittorrent
>  only as it is a much lighter weight method to get bits out the door than
>  to go through our mirroring system.  If you cannot use bittorrent we
>  apologize for the inconvenience. 
Is it difficult to release jigdos?

A problem with bittorrent (other than intolerant firewalls) its that
many broadband offerings, such as ADSL, that are designed around the
assumption that most users download more than they upload, and I gather
peer2peer networking, such as bittorrent, doesn't work so well as it's
designed to prefer sending over receiving[1].

In Oz - I can't speak for other locations - IAPs tend to either mirror
popular downloads (including rawhide in my case) or peer with some other
IAPs and share mirrors.

And then some of us download the whole box and dice ourselves, and could
easily and quickly assemble it off their own network.




[1] It's serious enough a problem that some are seeking to "manage"
traffic by breaking connexions, and others are reviewing their charging
regimes. IAPs here that previously did not charge for uploads are now
reversing that practice.

_I_ think they must, they're not getting the expected return from their
investment and shareholders care about that.



-- 

Cheers
John

-- spambait
1aaaaaaa(a)coco.merseine.nu  Z1aaaaaaa(a)coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

Rawhide 20080328 Snapshot Released

[Jesse Keating]

On Sat, 2008-03-29 at 22:59 +0900, John Summerfield wrote:
>  Is it difficult to release jigdos? 
The problem is this.

Jigdo would require the exploaded bits be available by http.  Trying to
sync all that content around the mirrors in any sort of reasonable time
frame is not going to happen.  Hanging the entire tree off of a single
http point is not going to happen either, that point would quickly drown
under the connections.  We can't just rely on rawhide as tomorrow the
rawhide content will be different and you wouldn't be able to complete
your jigdo.

If we're to have any sort of fast to the public snapshoting, we have to
use a delivery mechanism that is capable of spreading the bandwidth load
throughout the users without bringing down the host.  Right now, that
only leaves us with bittorrent as an option.

Now, if there were some way to combine bittorrent and jigdo and if jigdo
had better failover methods when mirrors are hit without the content we
could potentially do something better.  Jigdo + rawhide + whatever you
already have for the majority of the content, bittorrent to suck down
the very last little bit or something along those lines.

-- 
Jesse Keating
Fedora -- All my bits are free, are yours?

Rawhide 20080328 Snapshot Released

[Andrew Farris]

Jesse Keating wrote:
>  On Sat, 2008-03-29 at 22:59 +0900, John Summerfield wrote:
> > Is it difficult to release jigdos?
>  
>  The problem is this.
>  
>  Jigdo would require the exploaded bits be available by http.  Trying to
>  sync all that content around the mirrors in any sort of reasonable time
>  frame is not going to happen.  Hanging the entire tree off of a single
>  http point is not going to happen either, that point would quickly drown
>  under the connections.  We can't just rely on rawhide as tomorrow the
>  rawhide content will be different and you wouldn't be able to complete
>  your jigdo.
>  
>  If we're to have any sort of fast to the public snapshoting, we have to
>  use a delivery mechanism that is capable of spreading the bandwidth load
>  throughout the users without bringing down the host.  Right now, that
>  only leaves us with bittorrent as an option.
>  
>  Now, if there were some way to combine bittorrent and jigdo and if jigdo
>  had better failover methods when mirrors are hit without the content we
>  could potentially do something better.  Jigdo + rawhide + whatever you
>  already have for the majority of the content, bittorrent to suck down
>  the very last little bit or something along those lines. 
A suggestion I made awhile back re: breaking network installs would help here. 
Rawhide currently changes monotonically (probably by directory renaming?) and 
when the primary mirror finishes it's repo compose it ends up removing all the 
prior repo bits.  Now, network installs request files by name, and they already 
have the list of file names to get from anaconda doing the metadata fetch and 
dep solving.  If a network install is going during the repo change it breaks and 
cannot complete even if it was fetching the very last package when the repo 
changed.  Each mirror rsyncs at random times, but whenever they do that mirror 
ends up with the same behavior (almost monotonic change from one repodata 
package set to the next).

Keeping the prior day (or two days) packages in place, but not including them in 
the current repodata would fix this.  It would allow installs already started to 
fetch the package they expect to find.  The mirrors already have that data, so 
it means a very light load increase on disk space; files that would normally be 
deleted during that repo changeover would be kept, but all new files coming in 
would also be there, so a small increase exists on average.

Anyone syncing that repo tree would already have the files that are excess 
space, so it shouldn't impact bandwidths much because still only the new 
packages need fetched.

For the jigdo issue, that would also leave the packages that need to be fetched 
for a day or two.  If people reacted quickly it would be possible to build the 
images.  If they were lazy, they would need to build off the next jigdo 
snapshot, whenever that happened.

-- 
Andrew Farris <lordmorgul(a)gmail.com&gt; www.lordmorgul.net
  gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
----                                                                       ----

Rawhide 20080328 Snapshot Released

[Andrew Farris]

Andrew Farris wrote:
>  A suggestion I made awhile back re: breaking network installs would
> help 
>  here. Rawhide currently changes monotonically (probably by directory  
Hmm, been looking at dynamic domino logic all day... s/monotonically/atomically, 
just meaning it changes the whole repo package set at once.

-- 
Andrew Farris <lordmorgul(a)gmail.com&gt; www.lordmorgul.net
  gpg 0xC99B1DF3 fingerprint CDEC 6FAD BA27 40DF 707E A2E0 F0F6 E622 C99B 1DF3
No one now has, and no one will ever again get, the big picture. - Daniel Geer
----                                                                       ----

Rawhide 20080328 Snapshot Released

[Development discussions related to Fedora]

Jesse Keating wrote:
>  On Sat, 2008-03-29 at 22:59 +0900, John Summerfield wrote:
> > Is it difficult to release jigdos?
>  
>  The problem is this.
>  
>  Jigdo would require the exploaded bits be available by http.  Trying to
>  sync all that content around the mirrors in any sort of reasonable time
>  frame is not going to happen.  Hanging the entire tree off of a single
>  http point is not going to happen either, that point would quickly drown
>  under the connections.  We can't just rely on rawhide as tomorrow the
>  rawhide content will be different and you wouldn't be able to complete
>  your jigdo.
>  
>  If we're to have any sort of fast to the public snapshoting, we have to
>  use a delivery mechanism that is capable of spreading the bandwidth load
>  throughout the users without bringing down the host.  Right now, that
>  only leaves us with bittorrent as an option.
>  
>  Now, if there were some way to combine bittorrent and jigdo and if jigdo
>  had better failover methods when mirrors are hit without the content we
>  could potentially do something better.  Jigdo + rawhide + whatever you
>  already have for the majority of the content, bittorrent to suck down
>  the very last little bit or something along those lines. 
One way to make bittorrent for the {iso} faster may be to:
- build iso as normal
- mount the iso
- build the torrent of the mounted iso
     now we have a torrent in little rpm {+ other bits} chunks.
- user gets set to download this torrent, but puts the download location 
in the same location that he had his previous copy of mounted iso.
   note: the practice of including a subfolder in the .torrent 
definition makes this difficult - because you need to mess with what the 
path inside the new torrent will be.

- force recheck function {azureus}
- each pre-downloaded {existing} file is checked against the torrent
- bt client only downloads the changed chunks / new files.

- need a tool to rebuild the identical iso out of the copy of the 
mounted tree...  including the iso structure and boot parts.

Further it would be cool if bt client could know about:
- fastest: bits I may already have on my local machine
- faster: bits I have on my local network
- fast: bits that my ISP is mirroring {and not charging me download for}
- medium: in country mirrors.
- bittorrent normal method.
and use them in that order.

If any of that actually makes sense, let me know if I can be of assistance.

DaveT.

Rawhide 20080328 Snapshot Released

[John Summerfield]

Jesse Keating wrote:
>  On Sat, 2008-03-29 at 22:59 +0900, John Summerfield wrote:
> > Is it difficult to release jigdos?
>  
>  The problem is this.
>  
>  Jigdo would require the exploaded bits be available by http.  Trying to
>  sync all that content around the mirrors in any sort of reasonable time
>  frame is not going to happen.  Hanging the entire tree off of a single
>  http point is not going to happen either, that point would quickly drown
>  under the connections.  We can't just rely on rawhide as tomorrow the
>  rawhide content will be different and you wouldn't be able to complete
>  your jigdo.
>  
>  If we're to have any sort of fast to the public snapshoting, we have to
>  use a delivery mechanism that is capable of spreading the bandwidth load
>  throughout the users without bringing down the host.  Right now, that
>  only leaves us with bittorrent as an option.
>  
>  Now, if there were some way to combine bittorrent and jigdo and if jigdo
>  had better failover methods when mirrors are hit without the content we
>  could potentially do something better.  Jigdo + rawhide + whatever you
>  already have for the majority of the content, bittorrent to suck down
>  the very last little bit or something along those lines. 

I like Andrew's suggestion of keeping stuff around in the mirrors for a 
time, and that will probably be beneficial to the mirrors as they will 
become more reliable.

What problems a user might have in completing  jigdo would depend on 
timing wrt their mirror's update (and I will assume mirrors manage theri 
refresh so the are consistent), and their download speed. I'm on ADSL2+ 
at a handy distance from my exchange and I can pull stuff pretty quickly 
- 1.2 Mytes/sec and better, but I well remember pulling 3 Gbytes/month 
through a modem - not everyone does so well. For me filling out a jigdo 
isn't hard.

Using bittorrent at all is still a problem for those with unfriendly 
firewalls; is rsync from a master copy an option?

Ideally someone would write a script to wrap the whole procedure so it's 
not "too much trouble" for users.







-- 

Cheers
John

-- spambait
1aaaaaaa(a)coco.merseine.nu  Z1aaaaaaa(a)coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)