[Popcon-developers] Bug#350934: popularity-contest: HTTP method fails to recognize success

[Bill Allombert]

On Thu, Apr 06, 2006 at 11:45:05PM -0300, Javier Kohen wrote:
> Hello Bill
> 
> El vie, 07-04-2006 a las 00:54 +0200, Bill Allombert escribió:
> > > Popcon-upload fails to recognize a successful upload. The popcon web server replies within 2 seconds with the following banner, still popcon ends up aborting with a timeout after 30 seconds:
> > > 
> > > 
> > > HTTP/1.1 200 OK
> > > Transfer-Encoding: chunked
> > > Date: Wed, 01 Feb 2006 19:59:48 GMT
> > > Content-Type: text/plain; charset=utf-8
> > > Server: Apache/1.3.33 (Debian GNU/Linux) PHP/4.3.10-16 DAV/1.0.3
> > > 
> > > 27 
> > > Compressed (x-gzip) encoding detected.
> > > 
> > > 54 
> > > Thanks for your submission to Debian Popularity-Contest!
> > > DEBIAN POPCON HTTP-POST OK
> > > 
> > > 0
> > > 
> > > 
> > > Running popcon-upload with the -d switch doesn't give any useful output at all, as it never reaches the line where the content is matched against the "DEBIAN POPCON HTTP-POST OK" pattern.
> > > 
> > 
> > Hello Javier,
> > 
> > It seems popcon-upload will timeout after 30 seconds when the script to
> > send the data, so if your connection is slow it might take more than 30
> > seconds to complete, and the timeout will occur.
> 
> My connection is definitely not slow. In fact, if you read my original
> report, the server is reporting having received all data after *2
> seconds*. Moreover, I have a fair amount of packages and popcon sends
> less than 16kB of data; even at 9600bps there is plenty of time to send
> it.

Hello Javier,

Could you try

time /usr/share/popularity-contest/popcon-upload < /var/log/popularity-contest && echo $?

Here I get
time /usr/share/popularity-contest/popcon-upload < /var/log/popularity-contest;echo $?
/usr/share/popularity-contest/popcon-upload < /var/log/popularity-contest  1,79s user 0,08s system 9% cpu 20,398 total
0

Note the 0 that means success.

> I don't want to appear ungrateful, but I would certainly appreciate it
> if you had actually read my original report. Don't hesitate to ask me
> any questions you might have.

There is no contradiction with what you wrote: on my box we get the
reply in about 2 seconds but the script take actually 20 seconds to 
complete. If it tooks 30 seconds, the timeout might be a problem.

Since I cannot reproduce the problem I have to rely on guesswork.

I hope Petter can investigate. 

I think the problem is that the client side rely does not parse the
banner to detect the end but instead rely on the socket to be closed,
and this happen too late. 

Could you try the attached popcon-upload script ? This one will
terminate as soon as it receive the "DEBIAN POPCON HTTP-POST OK"
string instead of waiting for the socket to be closed. This way
I get:

time ./popcon-upload < /var/log/popularity-contest ;echo $?
./popcon-upload < /var/log/popularity-contest  0,41s user 0,05s system 36% cpu 1,255 total
0

which means the script finishes in less than 2 seconds instead of 20s.

Cheers,
-- 
Bill. <ballombe at debian.org>

Imagine a large red swirl here. 
-------------- next part --------------
#!/usr/bin/perl -w
# Written by Bill Allombert for the Debian popularity-contest project.
# This file is placed in the public domain.

use strict;
use IO::Socket;
use Getopt::Std;

my %opts;
getopts("Cdu:f:", \%opts);

sub usage {
    print <<"EOF";
Usage: $0 [-Cd] [-u <url>] [-f <file>]
  -C        send submissions in clear text, and not compressed
  -d        enable debugging
  -u <url>  submit to the given URL (default popcon.debian.org)
  -f <file> read popcon report from file (default stdin)
EOF
}

my $compressed = 1; # Submit reports in a compressed form?

my ($submiturl)  = $opts{'u'} || "http://popcon.debian.org/cgi-bin/popcon.cgi";
my ($file)  = $opts{'f'} || "-";
$compressed = 0 if ($opts{'C'});

my ($host) = $submiturl =~ m%http://([^/]+)%;

print "Unable to parse url\n" if ($opts{'d'} && ! $host);

# Configure the proxy:
my ($http_proxy,$proxy,$port,$remote);

$http_proxy=$ENV{'http_proxy'};
if (defined($http_proxy))
{
  $http_proxy =~ m{http://([^:]*)(?::([0-9]+))?} 
        or die ("unrecognized http_proxy");
  $proxy=$1; $port=$2;
}
  
$proxy=$host unless (defined($proxy));
$port=80 unless (defined($port));

# Compress the report:
my ($str,$len);
my $encoding;
if ($compressed) {
    open FILE, "gzip -c $file |" or die "gzip -c $file";
    $encoding = "x-gzip";
} else {
    open FILE, "< $file" or die "reading from '$file'";
    $encoding = "identity";
}
$str .= $_ while(<FILE>); 
close(FILE);
$len = length($str);

# 30 second timeout on http connections
$SIG{ALRM} = sub { die "timeout in popcon-upload\n" };
alarm(30);

# Connect to server
$remote = IO::Socket::INET->new(Proto => "tcp", PeerAddr => $proxy, 
                                                PeerPort => $port); 
unless ($remote) { die "cannot connect to $proxy:$port" }

my $boundary = "----------ThIs_Is_tHe_bouNdaRY_\$";

#Content-Length: $len
# text/plain; charset=utf-8
my $ORS = "\r\n"; # Use DOS line endings to make HTTP happy
my $form;
$form .= "--${boundary}$ORS";
$form .= "Content-Disposition: form-data; name=\"popcondata\"; filename=\"popcon-data\"$ORS";
$form .= "Content-Encoding: $encoding$ORS";
$form .= "Content-Type: application/octet-stream$ORS$ORS";
$form .= "$str$ORS";
$form .= "--${boundary}--$ORS";
$form .= "$ORS";

my $formlen = length($form);

#Send data
print $remote <<"EOF";
POST $submiturl HTTP/1.1
User-Agent: popcon-upload
Host: $host
content-type: multipart/form-data; boundary=$boundary
content-length: $formlen

$form
EOF

#Get answer
my($answer)="";
while(<$remote>)
{
  $answer.=$_;
  m/DEBIAN POPCON HTTP-POST OK/ and last;
}
close ($remote);
#Check answer
my $status = ($answer =~ m/DEBIAN POPCON HTTP-POST OK/) ? 0 : 1;
print "Failed to upload, answer '$answer'\n" if $status && $opts{'d'};
exit $status;