[cut-team] "weather" report, what to include?

[Drake Diedrich]

On Tue, Aug 10, 2010 at 4:22 AM, Stefano Zacchiroli <zack at debian.org> wrote:
> On Mon, Aug 09, 2010 at 03:12:40PM -0400, Joey Hess wrote:
>> So I want to take advantage of the freeze:
> <snip>
>> 3. Deal with upgrades. (Someone suggested makign a "weather report"
>>    that detects trouble in testing.)
>
> Just a brief comment on this, I totaly agree that we need something like
> that. For what concerns "number/ratio of uninstallable packages", that
> already exists for testing and is actively maintained by the
> EDOS/Mancoosi team I'm a member of. Data can be find starting from
> http://edos.debian.net (sorry for not having a more specific address for
> esting, but I'm writing this offline). We also have GNOME weather-like
> applets in the unofficial repository mancoosi.debian.net: they can be
> easily moved to the Debian archive and suggested to cut/rolling users.
>
> However, if we go this way, we probably need a more specific
> "cut/rolling weather" which takes into account various factors. One of
> those factor will be the number of uninstallable packages, but there can
> be more.
>
> For instance I'm still quite worried about packages removed from testing
> from transition reasons. Others have argued that "we lack evidence" and
> that they are rare cases anyhow; I'm sure that they are rare,
> nevertheless if we go ahead promoting a new suite, we need to be able to
> give *more guarantees* than the current testing, otherwise there is no
> point in doing all this. If we don't find/agree upon a technical way to
> ensure those transients removals won't affect rolling users, than at
> least we should include information about them in the weather report.
>
> What else should be include in a hypothetical weather report for
> cut/rollling to improve user experience?
>


I'm using EDOS to judge individual numbered snapshots of squeeze and
sid, and increment named snapshots when certain criteria are met (no
most errors than previously, or no errors at all).  This is an
evaluation of the entire snapshot, it might also be interesting to
only evaluate the packages in standard installs.

Examples:

http://debmarshal.debian.net/debian/dists/squeeze/0/main/binary-i386/Packages.edos
http://debmarshal.debian.net/debian/dists/squeeze/3/main/binary-i386/Packages.edos
  These two are the same, so "squeeze/better" got upgraded from
snapshot 0 to snapshot 3.  snapshots 1&2 had a temporary additional
problem, were skipped, and deleted early from the archive as no
symbolic name was pointing to them.

These are generated independently, but should be the same except for
mirror pulse timing differences:
http://edos.debian.net/edos-debcheck/results/testing/latest/i386/list.php
http://debmarshal.debian.net/debian/dists/squeeze/latest/main/binary-i386/Packages.edos

http://debmarshal.debian.net/debian/dists/sid/0/main/binary-i386/Packages.edos
http://debmarshal.debian.net/debian/dists/sid/108/main/binary-i386/Packages.edos
   The EDOS report for sid has grown from 10K to 34K in the past week
and a half, and doesn't appear to be converging.  (sorry it wasn't
working again for a few days - was out with a kidney stone following
Debconf10 - the heat and dehydration got me).  Analysing the whole
archive for sid doesn't look productive right now, but a more limited
analysis subset might pick out good snapshots to keep and base
something off of.

What if we snapshot the best (for some definition - installable,
upgradeable, local EDOS error minimums, ...) snapshots of squeeze or
sid, and then introduce an additional updates distribution to fix up
the last problems that block usability (like building installers,
hardware support, etc)?  We'll already be starting with the best
snapshots as far as dependencies go.

One thing I'm not checking yet is undeclared package contents -
opening up the .debs, finding out the actual files in each one, and
making sure any overlapping files have an appropriate Conflicts: in at
least one of the .debs.  This was happening in sid fairly frequently 4
years ago (when we were testing for it), particularly during
transitions and package renames, and is something we want to try to
pick minimums in as well (or generate our own minimums).  It looks
like EDOS is checking these manually and filing bugs.

-Drake