[Debtorrent-devel] Fwd: BitTorrent Protocol Expansion (Google SoC)

[Cameron Dale]

---------- Forwarded message ----------
From: Anthony Towns <aj at azure.humbug.org.au>
Date: Apr 9, 2007 4:47 AM
Subject: Re: BitTorrent Protocol Expansion (Google SoC)
To: Cameron Dale <camrdale at gmail.com>


On Sun, Apr 08, 2007 at 10:59:48PM -0700, Cameron Dale wrote:
> Anthony Towns wrote:
> >>>   - sharing files between torrents seems a worry, but a necessary
> >>>     one since we'll want to not double the space people need
> >>>     to watch testing and unstable.
> >> Something like the pooling done by the archive would seem to make sense.
> >> We just need to be careful not to run into problems where 2 torrents
> >> are trying to update the same pool file.
> > I guess we could just do locking?
> I'm not sure what you mean. File locking?

Hrm, I'm not sure what I mean either. I guess locking the file to a particular
torrent. Dunno.

> So, if I read this correctly, this means we can make the Packages file into a
> torrent easy enough in the short term, but adding unique piece numbers would be
> more difficult, as they require some kind of state information be kept. What
> info do you have in mind when you say "adding information that's based on the
> package but not specific to a version"?

I don't /think/ it's relevant, but it's things like Section: or Task:
fields, that stay the same across multiple versions.

> What about adding torrent identifiers to
> the Release files, how hard is that?

If it's just an extra descriptive field that only changes rarely, easy.

> > The hard change needs changes to apt which might be difficult to code,
> > and will require mvo checking them in some detail at least; and also
> > will require changes to dak, which will require more testing and review
> > by ftpmaster (me, James Troup, Ryan Murray).
> This is just the unique piece numbers, right? The rest is easy?

Well, as easy as these things get. :)

> > Sounds like the next steps would be:
> >    * break packages into smaller pieces
> >    * determine usage patterns
> >    * based on measured usage patterns analyse ways to optimise sharing
> >      amongst different Packages files (arch:all, different versions,
> >      testing/unstable, tesintg/stable at release)
> > Hrm. I'm not sure "determine usage patterns" can happen quickly enough for
> > the "analyse" step to actually happen as part of the GSoC. Any thoughts?
> I think the uptake of early adopters will be too slow. Maybe by then we will
> have a good enough idea on what to do anyway, without too much analysis. Or,
> just go ahead and blindly pick one and start implementing it, then decide later
> if it needs to be changed.

Hmmm. Something to think about as we go along then I guess.

> > For the first half, ordering as:
> >    1. will share all downloaded pieces with other interested clients
> >     (already what bittornado does!)
> >    2. implements variable size pieces for all packages
> >     (use hacked up .torrent files that will get you an "interesting" bit
> >      of the archive)
> >    3. uses current Packages files as torrent files
> >     (add Packages -> torrent parsing into bittornado so you don't need to
> >      download duplicate information)
> >    4. receives input from apt about which packages/pieces to download
> >     (add separate scripts to parse /var/lib/dpkg/available and/or apt
> >      to prioritise pieces?)
> >    5. runs as a daemon, started from /etc/init.d
> >     (automate it all)
> > might work well -- that way it's possible to start releasing usable
> > betas right from step (2).
> Looks good to me. I'm not sure what you mean by "add separate scripts to parse
> /var/lib/dpkg/available and/or apt to prioritise pieces" though.

Hrm, I meant /var/lib/dpkg/status which'll tell you what packages are
currently installed; adding scripts to apt would be a pre-invoke script
or similar, I guess.

Sounds good.

I've sent a mail to the mozilla folks you mentioned btw.

Cheers,
aj


-----BEGIN PGP SIGNATURE-----

iD8DBQFGGie+Oxe8dCpOPqoRAgjiAJ9lcsGQmQzK3f55oCztYlkkaFo09wCdHDOF
RhQWI7DQ82kYGGygnKoec1Y=
=gpXi
-----END PGP SIGNATURE-----