[Debtorrent-devel] Early testing version available

[Anthony Towns]

On Mon, Apr 30, 2007 at 05:16:26PM -0700, Cameron Dale wrote:
> I'm not sure if mean for these to be complete before a 0.1 release
> implementing the first step (variable-sized pieces) in the development
> plan, or whether you just mean in general things to do. I was thinking
> the 0.1 release was pretty much done.

Yup, I agree.

> >    * fix debtorrent to work independent of allocation method
> >[... snip...]
> >The allocation stuff (which is also one of the things you've marked with
> >a TODO, I see) probably needs to be changed a bit before it can be made
> >to work more nicely with apt, I'd say.
> This is not fixable, nor do I think it needs to be fixed, but my notes
> on this were probably deceptive so I'll clarify here.
>
> The original implementation would store downloaded pieces temporarily
> in other piece locations, [...]

Right. In theory that is fixable (you'd keep a list of offsets for the
pieces you've stored), but it's not worth the effort -- preallocation
is only important to keep the chunks within a file in order, having the
files themselves in random order on the disk doesn't much matter.

Two things do matter though. One is making sure the user can't choose an
option that'll break, which is presumably pretty easy to fix. 

The other is that having partially downloaded files around (or simply
pre-allocated files that are all null, whether sparse or not) means you
can't make the download area available for mirroring over http/rsync/ftp
or made available to apt until the debtorrent download is complete. Some
options:

	- declare it to not be a problem

	- don't preallocate files until you have a piece for that file
	  (minimise the problem)

	- download files to a temporary name (like rsync does) until
	  they're completed

	- have two areas, one that debtorrent downloads to, and one
	  that's made available to other processes, and add symlinks or
	  hardlinks for completed files/torrents

We don't need to do anything about this immediately by any means, but
I expect we'll need some solution for it so it's worth keeping in mind.

> >    * fix "TODO" annotated changes
> There's a few of these in the TODO file (if you mean the ones
> interspersed throughout the code, those are not important):

I meant the ones throughout the code; making sure the codebase is clean
at this point's worthwhile; even if that just means deciding "this is
the behaviour we want now" and moving/removing the TODO.

> Statistics for the swarm are incorrect
> This is not really important, as the statistics will become less
> important later when the program is run as a daemon and not
> interactively. Something to keep an eye on though.

It'll work better when the piece sizes are capped, too.

> >    * (re-)enable other frontends
> The only other frontend is btlaunchmany, which should work fine right
> now (though I haven't tried it). I was planning on using it for future
> work though, as we'll obviously want to run multiple torrents at once.

Hrm, what happened to btdownload{curses,gui} ? I haven't poked at
upstream, just noticed they're in the bittornado deb.

> >    * create Packages files with sub-package pieces so pieces can be 
> >    limited to 256kB/512kB or similar
> I thought this was for much later. We haven't even completed the
> initial 4 steps of the implementation from the wiki, and I think this
> requires apt changes to really make work. Do you want to change the
> original plan, and instead do this now?

Sorry, miscommunication: I was looking at those 4 steps as broader things
that'd get us to about the midway point, not 0.1. So I was thinking of
sub-package pieces as part of the "variable size pieces" phase.

Hacking on apt-ftparchive (the part of apt that generates Packages files)
is easy enough, but if you're happy with the dtorrents as they are atm,
it makes sense to delay it and keep working on debtorrent itself for
the time being.

> >Any particular preferences on which?
> 1. Implement a single variable size piece for each package. (complete)

Ack.

> 2. Add parsing of current Packages files so they can be used
> unmodified as torrent files. (partially complete)

I'd actually consider that complete for the moment, although having
a simple way to go from /var/lib/apt/lists/* to dtorrent files (a
"debtorrent update" command, eg) would be perfect.

> 3. Implement proxy functionality to receive input from apt about which
> packages to download. (might be the hardest part of the first 4 steps)

I'd consider that out of scope for 0.1 / the first four steps
really. Or I'd creatively misinterpret it so that looking through
/var/lib/apt/lists/* for Packages files and generating torrents for
them was sufficient (ie, "ah, they only want testing/main/i386 packages"
rather than "ah, they want gnome, not kde").

> 4. Automate the process to run as a daemon started from /etc/init.d

FWIW, I'd be inclined to either skip this for the moment, or create a
debtorrent user (with a home directory of /var/lib/debtorrent) which would
both make it a bit safer for people doing alpha/beta testing, and avoid
you having to worry too much about using $HOME/.DebTorrent for cache info.

Cheers,
aj

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 155 bytes
Desc: Digital signature
Url : http://lists.alioth.debian.org/pipermail/debtorrent-devel/attachments/20070501/52e8f1e2/attachment.pgp