[Teammetrics-discuss] Never ending nntpstat problems

[Andreas Tille]

On Wed, Dec 07, 2011 at 12:22:18PM +0530, Sukhbir Singh wrote:
> At least this is what the code says. Because we are populating the
> database per list, say if it fails for 'debian-X', we can just clear
> the records of debian-X from the database and restart the script. The
> exact behaviour cannot be simulated in this case (I guess), so we just
> have to see when it happens I guess :)

OK, I tried again, but ... :-(

$ ./nntpstat.py
Traceback (most recent call last):
  File "./nntpstat.py", line 353, in <module>
    main()
  File "./nntpstat.py", line 337, in main
    body, first, last)
  File "./nntpstat.py", line 226, in nntp_to_mbox
    liststat.parse_and_save({lst_url: mbox_file_path}, nntp=True)
  File "/home/tille/debian-maintain/teammetrics/liststat.py", line 185, in parse_and_save
    for key, message in mbox_file.iteritems():
  File "/usr/lib/python2.6/mailbox.py", line 126, in iteritems
    value = self[key]
  File "/usr/lib/python2.6/mailbox.py", line 82, in __getitem__
    return self.get_message(key)
  File "/usr/lib/python2.6/mailbox.py", line 690, in get_message
    msg = self._message_factory(string.replace(os.linesep, '\n'))
  File "/usr/lib/python2.6/mailbox.py", line 1492, in __init__
    Message.__init__(self, message)
  File "/usr/lib/python2.6/mailbox.py", line 1353, in __init__
    self._become_message(email.message_from_string(message))
  File "/usr/lib/python2.6/email/__init__.py", line 57, in message_from_string
    return Parser(*args, **kws).parsestr(s)
  File "/usr/lib/python2.6/email/parser.py", line 82, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/lib/python2.6/email/parser.py", line 72, in parse
    return feedparser.close()
  File "/usr/lib/python2.6/email/feedparser.py", line 168, in close
    self._call_parse()
  File "/usr/lib/python2.6/email/feedparser.py", line 161, in _call_parse
    self._parse()
  File "/usr/lib/python2.6/email/feedparser.py", line 429, in _parsegen
    self._cur.set_payload(EMPTYSTRING.join(lines))
MemoryError

The logfile implies that this happens in the mbox parsing phase.

Kind regards

          Andreas.

-- 
http://fam-tille.de