[Debtorrent-devel] Early testing version available

[Cameron Dale]

On 4/30/07, Anthony Towns <aj at azure.humbug.org.au> wrote:
> Two things do matter though. One is making sure the user can't choose an
> option that'll break, which is presumably pretty easy to fix.

Good idea, I'll do that. Should be pretty easy.

> The other is that having partially downloaded files around (or simply
> pre-allocated files that are all null, whether sparse or not) means you
> can't make the download area available for mirroring over http/rsync/ftp
> or made available to apt until the debtorrent download is complete. Some
> options:

It's definitely a problem for rsync/http/ftp, but I didn't think it
would be a problem for apt. Doesn't apt only use files you provide
directly to it? I didn't think it would parse a directory looking for
files, and therefore finding incomplete ones.

>         - declare it to not be a problem
>
>         - don't preallocate files until you have a piece for that file
>           (minimise the problem)
>
>         - download files to a temporary name (like rsync does) until
>           they're completed
>
>         - have two areas, one that debtorrent downloads to, and one
>           that's made available to other processes, and add symlinks or
>           hardlinks for completed files/torrents
>
> We don't need to do anything about this immediately by any means, but
> I expect we'll need some solution for it so it's worth keeping in mind.

I'm definitely liking "not a problem" for now, :) though it is
something to think about.

> > >    * fix "TODO" annotated changes
> > There's a few of these in the TODO file (if you mean the ones
> > interspersed throughout the code, those are not important):
>
> I meant the ones throughout the code; making sure the codebase is clean
> at this point's worthwhile; even if that just means deciding "this is
> the behaviour we want now" and moving/removing the TODO.

I was going to leave them in, as they're good markers for things that
need doing, and it's only an alpha release, but if you think it's
worthwhile then I'll remove them.

> Hrm, what happened to btdownload{curses,gui} ? I haven't poked at
> upstream, just noticed they're in the bittornado deb.

I removed the curses and gui front ends, as we're looking to create a
daemon, not an interactive program. Do you think it's worth putting
them in to encourage alpha-level testing, especially considering they
won't be in the beta (daemon will be implemented at that stage)?

> > >    * create Packages files with sub-package pieces so pieces can be
> > >    limited to 256kB/512kB or similar
> > I thought this was for much later. We haven't even completed the
> > initial 4 steps of the implementation from the wiki, and I think this
> > requires apt changes to really make work. Do you want to change the
> > original plan, and instead do this now?
>
> Sorry, miscommunication: I was looking at those 4 steps as broader things
> that'd get us to about the midway point, not 0.1. So I was thinking of
> sub-package pieces as part of the "variable size pieces" phase.

Definitely agree with you on the 4 steps being midway point. I have it
on the wiki that "break up large packages into a number of pieces" is
something that comes after the initial 4 steps.

> Hacking on apt-ftparchive (the part of apt that generates Packages files)
> is easy enough, but if you're happy with the dtorrents as they are atm,
> it makes sense to delay it and keep working on debtorrent itself for
> the time being.

I'm happy with the way it is for alpha stage testing. If we could get
it added to apt-ftparchive sometime before the beta release (after the
first 4 steps), then we could maybe make a 5th step of the sub-package
pieces, and then release the beta with that. Either way's fine with
me. It might just come down to time considerations.

> > >Any particular preferences on which?
> > 1. Implement a single variable size piece for each package. (complete)
>
> Ack.

That a good "Ack" or a bad one?

> > 2. Add parsing of current Packages files so they can be used
> > unmodified as torrent files. (partially complete)
>
> I'd actually consider that complete for the moment, although having
> a simple way to go from /var/lib/apt/lists/* to dtorrent files (a
> "debtorrent update" command, eg) would be perfect.

I was going to add a function, so that instead of calling it with the
intermediate dtorrent file as argument, you just call it with a
Packages file as argument. The function is mostly written (pretty
simple). The only downside is the tracker announce, which will have to
default to the dttracker address. Eventually, I'm thinking this will
be something we'll add to apt-ftparchive to get put in the
Releases/Packages files.

> > 3. Implement proxy functionality to receive input from apt about which
> > packages to download. (might be the hardest part of the first 4 steps)
>
> I'd consider that out of scope for 0.1 / the first four steps
> really. Or I'd creatively misinterpret it so that looking through
> /var/lib/apt/lists/* for Packages files and generating torrents for
> them was sufficient (ie, "ah, they only want testing/main/i386 packages"
> rather than "ah, they want gnome, not kde").
>
> > 4. Automate the process to run as a daemon started from /etc/init.d
>
> FWIW, I'd be inclined to either skip this for the moment, or create a
> debtorrent user (with a home directory of /var/lib/debtorrent) which would
> both make it a bit safer for people doing alpha/beta testing, and avoid
> you having to worry too much about using $HOME/.DebTorrent for cache info.

I guess we're misunderstanding each other again with these last 2
steps. The 0.1 I'm think of is only the first step, and so doesn't
include these 2. After that, adding step 2 can be 0.2, or 0.1.1, the
key point being there is a new alpha release after each step. Then,
after step 4 (the mid point), there can be a beta release, which I
think would need to include some kind of proxying functionality (i.e.
recognizing gnome over kde, rather than just suite), for it to be at
all useful. I don't think many people will want to use it if it means
having to download the entire archive. The daemon-izing in step 4 I
think is also necessary for a beta release.

So I guess I see the things to do before a 0.1 release as:

   * fix debtorrent to not allow changes to allocation method
   * fix "TODO" annotated changes
   * tarball it and upload to alioth

Are we on the same page now? Seems like it, sorry for the confusion.

Cameron