[Debtorrent-devel] Early testing version available

[Cameron Dale]

On 4/30/07, Anthony Towns <aj at azure.humbug.org.au> wrote:
> So, I guess next things on the list are:

I'm not sure if mean for these to be complete before a 0.1 release
implementing the first step (variable-sized pieces) in the development
plan, or whether you just mean in general things to do. I was thinking
the 0.1 release was pretty much done.

>     * fix debtorrent to work independent of allocation method
> [... snip...]
> The allocation stuff (which is also one of the things you've marked with
> a TODO, I see) probably needs to be changed a bit before it can be made
> to work more nicely with apt, I'd say.

This is not fixable, nor do I think it needs to be fixed, but my notes
on this were probably deceptive so I'll clarify here.

The original implementation would store downloaded pieces temporarily
in other piece locations, starting at the beginning of the list of
files, and then moving them into place as the download continues. This
allowed for allocation to proceed at the minimum possible speed (the
download speed), while somewhat minimizing the amount of fragmentation
created.

This no longer works, as none of the pieces are the same size. It is
also undesirable, as it means the files can not be read until the
download is complete, as the pieces in the file might not be in place,
whereas we would like to serve completed files to apt while
downloading continues. Pre-allocation also shouldn't be a problem for
us, as downloads will start out with all files disabled, which means
they wont be pre-allocated, and then files to download will be added
one at a time, which also minimizes fragmentation and allocation speed
required. So, I think pre-allocation is win-win for us at this stage.

When (or if) we introduce a maximum piece size a lot of the pieces
will still be different sizes, making this impossible between files,
but it could then be used to allocate pieces within large files, but
this is something for the future.

>     * fix "TODO" annotated changes

There's a few of these in the TODO file (if you mean the ones
interspersed throughout the code, those are not important):

Move the data cache from ~/.DebTorrent

This is probably fine for now, and I haven't really come up with a
better place to store this information. This will probably change at
implementation step 4 when the program is made into a daemon.

Comments

I could certainly do some of this now, but whether it's really
necessary at this stage is debatable, and might break merging in new
upstreams, if that's desirable. Though I'm always disappointed with
uncommented code, which is all there is right now. :)

Statistics for the swarm are incorrect

This is not really important, as the statistics will become less
important later when the program is run as a daemon and not
interactively. Something to keep an eye on though.

>     * (re-)enable other frontends

The only other frontend is btlaunchmany, which should work fine right
now (though I haven't tried it). I was planning on using it for future
work though, as we'll obviously want to run multiple torrents at once.

>     * create Packages files with sub-package pieces so pieces can be limited
>       to 256kB/512kB or similar

I thought this was for much later. We haven't even completed the
initial 4 steps of the implementation from the wiki, and I think this
requires apt changes to really make work. Do you want to change the
original plan, and instead do this now?

>     * put together a tarball or .deb of debtorrent for easy testing

Definitely required for a release. I was thinking of preparing a
tarball and posting it on the alioth project for download once 0.1 is
ready.

> Any particular preferences on which?

I was thinking the implementation would proceed more along the
previously discussed steps of:

1. Implement a single variable size piece for each package. (complete)

2. Add parsing of current Packages files so they can be used
unmodified as torrent files. (partially complete)

3. Implement proxy functionality to receive input from apt about which
packages to download. (might be the hardest part of the first 4 steps)

4. Automate the process to run as a daemon started from /etc/init.d

And then release a public beta for testing purposes. Do you no longer
like this plan?

Cameron