[Debtorrent-devel] Early testing version available

Tue May 1 08:54:26 UTC 2007

On Mon, Apr 30, 2007 at 11:04:45PM -0700, Cameron Dale wrote:
> >The other is that having partially downloaded files around (or simply
> >pre-allocated files that are all null, whether sparse or not) means you
> >can't make the download area available for mirroring over http/rsync/ftp
> >or made available to apt until the debtorrent download is complete. Some
> >options:
> It's definitely a problem for rsync/http/ftp, but I didn't think it
> would be a problem for apt. 

Well the easiest way to hook it up to apt locally is to give apt a file
URL, like:

	deb file:/srv/debian/ etch main

and having pool/ populated via debtorrent, and dists/ populated with
Release and Packages files by some other method. 

But the dists/**/Packages files will list all the .debs not just the
ones that have downloaded successfully, so when apt looks for any package
that's not fully downloaded, it'll see a file with the right name, right
size and a recentish timestamp, try using it and then fail. That might
not be too bad (although thinking about it, I'm not 100% sure that it
does md5sum the file before trying to use it), but it'd be better if
there just wasn't a file there until debtorrent's got the whole thing.

Longer term we might be talking a sources.list line like "deb debtorrent:
etch main" or similar and that won't matter, of course, but that's
a whole other thing, afaics. (Doing the apt-ftparchive / piece-size
capping stuff would come before this eg, IMO)

> >        - declare it to not be a problem
> >We don't need to do anything about this immediately by any means, but
> >I expect we'll need some solution for it so it's worth keeping in mind.
> I'm definitely liking "not a problem" for now, :) though it is
> something to think about.

Righto.

> >> >    * fix "TODO" annotated changes
> >> There's a few of these in the TODO file (if you mean the ones
> >> interspersed throughout the code, those are not important):
> >I meant the ones throughout the code; making sure the codebase is clean
> >at this point's worthwhile; even if that just means deciding "this is
> >the behaviour we want now" and moving/removing the TODO.
> I was going to leave them in, as they're good markers for things that
> need doing, and it's only an alpha release, but if you think it's
> worthwhile then I'll remove them.

Well, if they're things you've broken (like the allocation option)
it's definitely best to "fix" them now from a user's perspective,
even if that means saying "this is no longer a possibility", rather
than leaving broken bits in the program. "This should be a possibility
(again) in the future" is worth keeping around in whatever form you like
(though I think the ./TODO file is probably better than notes in the
code), but it's better to do the tidying up asap.

> >Hrm, what happened to btdownload{curses,gui} ? I haven't poked at
> >upstream, just noticed they're in the bittornado deb.
> I removed the curses and gui front ends, as we're looking to create a
> daemon, not an interactive program. Do you think it's worth putting
> them in to encourage alpha-level testing, especially considering they
> won't be in the beta (daemon will be implemented at that stage)?

Hrm. I guess I'm thinking more along the lines of keeping the changes
from BitTornado minimal, so that the DebTorrent stuff can potentially
be included upstream if it turns out to be useful generally; and also
that the more code there is that uses the underlying DebTorrent code,
the more likely that is to catch any bugs in earlier testing.

> Definitely agree with you on the 4 steps being midway point. I have it
> on the wiki that "break up large packages into a number of pieces" is
> something that comes after the initial 4 steps.

Okay, that sounds fine. I'd say that'll be a prerequisite for beta-testing
though -- 50MB+ pieces just don't seem feasible to me at large scale,
so perhaps that means we aim for alpha releases of 0.1 - 0.4 up to the
mid-point, and beta releases after that?

> I'm happy with the way it is for alpha stage testing. If we could get
> it added to apt-ftparchive sometime before the beta release (after the
> first 4 steps), then we could maybe make a 5th step of the sub-package
> pieces, and then release the beta with that. Either way's fine with
> me. It might just come down to time considerations.

debconf is just before the midpoint, so there's a reasonable chance it
can get hacked on during that couple of weeks.

> >> >Any particular preferences on which?
> >> 1. Implement a single variable size piece for each package. (complete)
> >Ack.
> That a good "Ack" or a bad one?

A good "ack". Bad acks will be more like "Erk" and include an explanation
or at least a question. :)

> >> 2. Add parsing of current Packages files so they can be used
> >> unmodified as torrent files. (partially complete)
> >I'd actually consider that complete for the moment, although having
> >a simple way to go from /var/lib/apt/lists/* to dtorrent files (a
> >"debtorrent update" command, eg) would be perfect.
> I was going to add a function, so that instead of calling it with the
> intermediate dtorrent file as argument, you just call it with a
> Packages file as argument. 

I think it's worth keeping the option/flexibility of calling it with a
.dtorrent file directly, particularly as far as reusing this for other
purposes goes, or for "unit-testing" or similar.

> The function is mostly written (pretty
> simple). The only downside is the tracker announce, which will have to
> default to the dttracker address. Eventually, I'm thinking this will
> be something we'll add to apt-ftparchive to get put in the
> Releases/Packages files.

The tracker should definitely go in the Release file eventually, and
it's probably quite reasonable to invoke debtorrent with both a Release
and a Packages file in all cases. Hardcoding it somewhere sounds like
the way to go for the moment though.

> I guess we're misunderstanding each other again with these last 2
> steps. The 0.1 I'm think of is only the first step, and so doesn't
> include these 2. After that, adding step 2 can be 0.2, or 0.1.1, the
> key point being there is a new alpha release after each step. Then,
> after step 4 (the mid point), 

Ack.

>                               there can be a beta release, which I
> think would need to include some kind of proxying functionality (i.e.
> recognizing gnome over kde, rather than just suite), for it to be at
> all useful. I don't think many people will want to use it if it means
> having to download the entire archive. The daemon-izing in step 4 I
> think is also necessary for a beta release.

I'd suggest letting 0.4.alpha live a while, then relabel it as 0.4.beta
once you're confident there's no bugs (ie it "works" for some value of
"working"), then add "proxying functionality", "capped piece sizes"
etc as further milestones that make it "work" for less trivial classes
of users.  Once it works for a suitably non-trivial number of users,
that's a real release, even if there's more features we've got.

> So I guess I see the things to do before a 0.1 release as:
>   * fix debtorrent to not allow changes to allocation method
>   * fix "TODO" annotated changes
>   * tarball it and upload to alioth

Ack.

> Are we on the same page now? Seems like it, sorry for the confusion.

Sounds good. Excellent progress for week, hrm, -5... :)

Cheers,
aj

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 155 bytes
Desc: Digital signature
Url : http://lists.alioth.debian.org/pipermail/debtorrent-devel/attachments/20070501/ec2dc1f5/attachment-0001.pgp