[Debtorrent-devel] Fwd: BitTorrent Protocol Expansion (Google SoC)

[Cameron Dale]

---------- Forwarded message ----------
From: Cameron Dale <camrdale at gmail.com>
Date: Apr 10, 2007 11:12 AM
Subject: Re: BitTorrent Protocol Expansion (Google SoC)
To: Anthony Towns <aj at azure.humbug.org.au>


Anthony Towns wrote:
> On Tue, Apr 10, 2007 at 12:16:26AM -0700, Cameron Dale wrote:
>> I haven't tried the sorting by popularity (hard), or by dependencies
>> (harder) yet, so they might be better. If you have any other ideas for
>> statistics you'd like to see related to this, let me know.
>
> Can't you just grab the data from http://popcon.debian.org/by_vote for
> sorting by popularity?

I could, I would just have to find a way to import it and I haven't
gotten around to it yet (I shouldn't have said "hard", just "work" ;) ).
I'll try tonight.

> Actually doing that might start getting too complicated on the server
> side, though.

I agree, it might not be possible, but might just let us see how optimal
an ordering can be in minimizing wastage.

>> This gives the
>> following for the 19091 packages in the same main-amd64 Packages file:
>>
>> Piece  |  Total # |         Number (%) of Packages that are
>>  Size  | of Pieces|     1 piece    |  2-8 pieces  | > 8 pieces
>> ------------------------------------------------------------------
>> 32 KB  |  476464  |   5388 (28.2%) | 7789 (40.8%) | 5914 (31.0%)
>> 64 KB  |  243926  |   8098 (42.4%) | 7008 (36.7%) | 3985 (20.9%)
>> 128 KB |  128265  |  10814 (56.6%) | 5906 (30.9%) | 2371 (12.4%)
>> 256 KB |   71209  |  13177 (69.0%) | 4586 (24.0%) | 1328 ( 7.0%)
>> 512 KB |   43331  |  15106 (79.1%) | 3321 (17.4%) |  664 ( 3.5%)
>>  1 MB  |   29920  |  16720 (87.6%) | 2053 (10.7%) |  318 ( 1.7%)
>>
>> This is again looking like 256 KB or 512 KB are the way to go, as any
>> smaller creates too many pieces, and larger means almost all the
>> packages are a single piece.
>
> Is that a bad thing? I download dpkg, you download coreutils, then we
> swap doesn't sound any worse than downloading half of coreutils each,
> then swapping.

>From the BitTorrent FAQ: "In general, the smaller the piece size, the
more efficient the BitTorrent download will be, but will result in a
larger .torrent file. 256 kB seems to be the most common piece size in
use these days, but you can experiment with other settings if you want."

This doesn't necessarily hold true in our case. For one, the torrent
file size is already quite large, and contains most of the information
for the torrent already, so adding piece information to it is not going
to affect it as much (though it's something to consider). Also, we're
already talking about a large number of pieces, but not many people are
downloading all of them, so we may (or may not) already be efficient
enough. Memory may also become a consideration, as the more pieces will
probably mean more memory to store that information. Lots of tradeoffs
to consider here, and not all is clear yet.

> I was figuring assuming "1GB pieces" (ie, every package is a single piece)
> will be the first step anyway, just to get something working.

Definitely we'll start with this, but I think you want to move to
something smaller to increase the efficiency.

> Looks good; want to put what you've got up on the wiki?

Doing that now.

Cameron
-------------- next part --------------
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFGG9NwDx924g0gNq0RAjkbAJ4mKN9ui4yEaLyyu92Hf0JtSmVqNQCgoMXj
NUxouktQZzfstCUEmOezCVE=
=X4Hr
-----END PGP SIGNATURE-----