[Debtorrent-commits] r71 - in /debtorrent/branches/http-listen: ./ DebTorrent/ DebTorrent/BT1/ docs/ docs/pstat/ downloads/ dtorrents/
camrdale-guest at users.alioth.debian.org
camrdale-guest at users.alioth.debian.org
Tue May 29 05:58:07 UTC 2007
Author: camrdale-guest
Date: Tue May 29 05:58:07 2007
New Revision: 71
URL: http://svn.debian.org/wsvn/debtorrent/?sc=1&rev=71
Log:
Merged revisions 45-50,52,54-61,63-70 via svnmerge from
svn+ssh://camrdale-guest@svn.debian.org/svn/debtorrent/debtorrent/trunk
........
r48 | camrdale-guest | 2007-05-16 22:19:57 -0700 (Wed, 16 May 2007) | 3 lines
Fixed a bug in bittornado that prevented using priorities with pre-allocation.
For more information: http://forums.degreez.net/viewtopic.php?t=7155
........
r49 | camrdale-guest | 2007-05-16 22:59:50 -0700 (Wed, 16 May 2007) | 1 line
Add assertion to check that file boundaries are piece boundaries
........
r50 | camrdale-guest | 2007-05-16 23:48:59 -0700 (Wed, 16 May 2007) | 1 line
Another little fix for the bittornado priority bug.
........
r52 | camrdale-guest | 2007-05-18 11:25:40 -0700 (Fri, 18 May 2007) | 1 line
Modify parsedir so btlaunchmany works with Packages files directly (tracker still only serves dtorrent files)
........
r54 | camrdale-guest | 2007-05-18 14:20:03 -0700 (Fri, 18 May 2007) | 1 line
Directories are no longer created when they will contain no files.
........
r55 | camrdale-guest | 2007-05-20 16:24:29 -0700 (Sun, 20 May 2007) | 1 line
Modify the README for the new method.
........
r56 | camrdale-guest | 2007-05-20 20:37:41 -0700 (Sun, 20 May 2007) | 1 line
Fix up the docs directory for use with epydoc
........
r57 | camrdale-guest | 2007-05-20 20:38:40 -0700 (Sun, 20 May 2007) | 1 line
Add some initial documentation (docstrings)
........
r58 | camrdale-guest | 2007-05-22 00:56:32 -0700 (Tue, 22 May 2007) | 1 line
More documentation
........
r59 | camrdale-guest | 2007-05-22 22:22:54 -0700 (Tue, 22 May 2007) | 1 line
More debug messages to help with connection problems.
........
r63 | camrdale-guest | 2007-05-26 14:04:19 -0700 (Sat, 26 May 2007) | 1 line
Improve profiling.
........
r64 | camrdale-guest | 2007-05-26 14:04:52 -0700 (Sat, 26 May 2007) | 1 line
Speed up _reset_ranges.
........
r65 | camrdale-guest | 2007-05-26 17:41:11 -0700 (Sat, 26 May 2007) | 1 line
Remove unneeded error function from get_packages.
........
r66 | camrdale-guest | 2007-05-26 17:41:37 -0700 (Sat, 26 May 2007) | 1 line
Make btshowmetainfo work on Packages files.
........
r67 | camrdale-guest | 2007-05-26 18:38:36 -0700 (Sat, 26 May 2007) | 1 line
Updated instructions for testing next release.
........
r68 | camrdale-guest | 2007-05-28 17:12:42 -0700 (Mon, 28 May 2007) | 1 line
Preparation for release.
........
r70 | camrdale-guest | 2007-05-28 22:27:34 -0700 (Mon, 28 May 2007) | 1 line
Add pstat file for btlaunchmany
........
Added:
debtorrent/branches/http-listen/docs/epydoc.config
- copied unchanged from r59, debtorrent/trunk/docs/epydoc.config
debtorrent/branches/http-listen/docs/pstat/
- copied from r59, debtorrent/trunk/docs/pstat/
debtorrent/branches/http-listen/docs/pstat/btlaunchmany.pstat
- copied unchanged from r70, debtorrent/trunk/docs/pstat/btlaunchmany.pstat
debtorrent/branches/http-listen/downloads/
- copied from r70, debtorrent/trunk/downloads/
Removed:
debtorrent/branches/http-listen/dtorrents/ftp.us.debian.org_debian_dists_stable_contrib_binary-i386_Packages.dtorrent
debtorrent/branches/http-listen/dtorrents/ftp.us.debian.org_debian_dists_stable_non-free_binary-i386_Packages.dtorrent
Modified:
debtorrent/branches/http-listen/ (props changed)
debtorrent/branches/http-listen/CHANGELOG
debtorrent/branches/http-listen/DebTorrent/BT1/Choker.py
debtorrent/branches/http-listen/DebTorrent/BT1/Connecter.py
debtorrent/branches/http-listen/DebTorrent/BT1/Downloader.py
debtorrent/branches/http-listen/DebTorrent/BT1/Encrypter.py
debtorrent/branches/http-listen/DebTorrent/BT1/FileSelector.py
debtorrent/branches/http-listen/DebTorrent/BT1/Rerequester.py
debtorrent/branches/http-listen/DebTorrent/BT1/Storage.py
debtorrent/branches/http-listen/DebTorrent/BT1/__init__.py
debtorrent/branches/http-listen/DebTorrent/BT1/btformats.py
debtorrent/branches/http-listen/DebTorrent/BTcrypto.py
debtorrent/branches/http-listen/DebTorrent/ConfigDir.py
debtorrent/branches/http-listen/DebTorrent/ConnChoice.py
debtorrent/branches/http-listen/DebTorrent/CurrentRateMeasure.py
debtorrent/branches/http-listen/DebTorrent/__init__.py
debtorrent/branches/http-listen/DebTorrent/bencode.py
debtorrent/branches/http-listen/DebTorrent/bitfield.py
debtorrent/branches/http-listen/DebTorrent/clock.py
debtorrent/branches/http-listen/DebTorrent/download_bt1.py
debtorrent/branches/http-listen/DebTorrent/inifile.py
debtorrent/branches/http-listen/DebTorrent/parsedir.py
debtorrent/branches/http-listen/README.txt
debtorrent/branches/http-listen/TODO
debtorrent/branches/http-listen/btcompletedir.py
debtorrent/branches/http-listen/btcopyannounce.py
debtorrent/branches/http-listen/btdownloadheadless.py
debtorrent/branches/http-listen/btlaunchmany.py
debtorrent/branches/http-listen/btmakemetafile.py
debtorrent/branches/http-listen/btreannounce.py
debtorrent/branches/http-listen/btrename.py
debtorrent/branches/http-listen/btsethttpseeds.py
debtorrent/branches/http-listen/btshowmetainfo.py
debtorrent/branches/http-listen/bttrack.py
debtorrent/branches/http-listen/docs/ (props changed)
debtorrent/branches/http-listen/setup.py
Propchange: debtorrent/branches/http-listen/
------------------------------------------------------------------------------
--- svnmerge-integrated (original)
+++ svnmerge-integrated Tue May 29 05:58:07 2007
@@ -1,1 +1,1 @@
-/debtorrent/trunk:1-44
+/debtorrent/trunk:1-50,52,54-70
Modified: debtorrent/branches/http-listen/CHANGELOG
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/CHANGELOG?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/CHANGELOG (original)
+++ debtorrent/branches/http-listen/CHANGELOG Tue May 29 05:58:07 2007
@@ -1,3 +1,14 @@
+debtorrent (0.1.1)
+
+ * Add ability to parse dpkg status for priorities of files to download
+ * Modify btlaunchmany and btshowmetainfo to work with Packages files directly
+ * Fixed a bug in bittornado that prevented using priorities with
+ pre-allocation: http://forums.degreez.net/viewtopic.php?t=7155
+ * Directories are no longer pre-allocated when they will contain no files
+ * Added lots of documentation
+
+ -- Cameron Dale <camrdale at gmail.com> Mon, 28 May 2007 17:11:00 -0700
+
debtorrent (0.1.0)
* Initial release, based on BitTornado 0.3.18
Modified: debtorrent/branches/http-listen/DebTorrent/BT1/Choker.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/BT1/Choker.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/BT1/Choker.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/BT1/Choker.py Tue May 29 05:58:07 2007
@@ -1,8 +1,10 @@
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+
+"""Contains the Choker class."""
from random import randrange, shuffle
from DebTorrent.clock import clock
@@ -13,7 +15,47 @@
False = 0
class Choker:
+ """Manages the choking and unchoking of other downloaders.
+
+ @type config: C{dictonary}
+ @ivar config: the configuration variables
+ @type round_robin_period: C{int}
+ @ivar round_robin_period: the number of seconds between the client's
+ switching of upload targets
+ @type schedule: C{method}
+ @ivar schedule: the method to call to schedule future events
+ @type picker: L{PiecePicker}
+ @ivar picker: the PiecePicker to get connection information from
+ @type connections: C{list} of L{Connecter.Connection}
+ @ivar connections: the connections from peers to the client
+ @type last_preferred: C{int}
+ @ivar last_preferred: the number of preferred connections found in the
+ last examination
+ @type last_round_robin: C{long}
+ @ivar last_round_robin: the last time the connections were examined
+ @type done: C{Event}
+ @ivar done: flag to indicate when the download is complete
+ @type super_seed: C{boolean}
+ @ivar super_seed: whether super-seeding is enabled
+ @type paused: C{Event}
+ @ivar paused: flag to indicate when the download is paused
+
+ """
+
def __init__(self, config, schedule, picker, done = lambda: False):
+ """Initialize the Choker instance.
+
+ @type config: C{dictonary}
+ @param config: the configuration variables
+ @type schedule: C{method}
+ @param schedule: the method to call to schedule future events
+ @type picker: L{PiecePicker}
+ @param picker: the piece picker to use to
+ @type done: C{Event}
+ @param done: flag to indicate when the download is complete
+
+ """
+
self.config = config
self.round_robin_period = config['round_robin_period']
self.schedule = schedule
@@ -27,9 +69,18 @@
schedule(self._round_robin, 5)
def set_round_robin_period(self, x):
+ """Set a new round-robin period.
+
+ @type x: C{int}
+ @param x: the new round-robin period
+ @see: L{Choker.round_robin_period}
+
+ """
+
self.round_robin_period = x
def _round_robin(self):
+ """Periodically determine the ordering for connections and call the choker."""
self.schedule(self._round_robin, 5)
if self.super_seed:
cons = range(len(self.connections))
@@ -59,6 +110,13 @@
self._rechoke()
def _rechoke(self):
+ """Unchoke some connections.
+
+ Reads the current upload and download rates from the connections,
+ as well as the connection state, and unchokes the most preferable
+ ones.
+
+ """
preferred = []
maxuploads = self.config['max_uploads']
if self.paused:
@@ -101,31 +159,70 @@
u.unchoke()
def connection_made(self, connection, p = None):
+ """Adds a new connection to the list.
+
+ @type connection: L{Connecter.Connection}
+ @param connection: the connection to the client from the peer
+ @type p: C{int}
+ @param p: the location to insert the new connection into the list
+ (optional, default is to choose a random location)
+
+ """
+
if p is None:
p = randrange(-2, len(self.connections) + 1)
self.connections.insert(max(p, 0), connection)
self._rechoke()
def connection_lost(self, connection):
+ """Removes a lost connection from the list.
+
+ @type connection: L{Connecter.Connection}
+ @param connection: the connection to the client from the peer
+
+ """
+
self.connections.remove(connection)
self.picker.lost_peer(connection)
if connection.get_upload().is_interested() and not connection.get_upload().is_choked():
self._rechoke()
def interested(self, connection):
+ """Indicate the connection is now interesting.
+
+ @type connection: L{Connecter.Connection}
+ @param connection: the connection to the client from the peer
+
+ """
+
if not connection.get_upload().is_choked():
self._rechoke()
def not_interested(self, connection):
+ """Indicate the connection is no longer interesting.
+
+ @type connection: L{Connecter.Connection}
+ @param connection: the connection to the client from the peer
+
+ """
+
if not connection.get_upload().is_choked():
self._rechoke()
def set_super_seed(self):
+ """Change to super seed state."""
while self.connections: # close all connections
self.connections[0].close()
self.picker.set_superseed()
self.super_seed = True
def pause(self, flag):
+ """Pause the choker.
+
+ @type flag: C{Event}
+ @param flag: flag to indicate when pausing is finished
+
+ """
+
self.paused = flag
self._rechoke()
Modified: debtorrent/branches/http-listen/DebTorrent/BT1/Connecter.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/BT1/Connecter.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/BT1/Connecter.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/BT1/Connecter.py Tue May 29 05:58:07 2007
@@ -1,8 +1,35 @@
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+
+"""For maintaining connections to peers.
+
+ at type DEBUG1: C{boolean}
+ at var DEBUG1: whether to print debugging information for the L{Connection}
+ at type DEBUG2: C{boolean}
+ at var DEBUG2: whether to print debugging information for the L{Connecter}
+ at type CHOKE: C{char}
+ at var CHOKE: the code for choke messages
+ at type UNCHOKE: C{char}
+ at var UNCHOKE: the code for unchoke messages
+ at type INTERESTED: C{char}
+ at var INTERESTED: the code for interested messages
+ at type NOT_INTERESTED: C{char}
+ at var NOT_INTERESTED: the code for not interested messages
+ at type HAVE: C{char}
+ at var HAVE: the code for have messages
+ at type BITFIELD: C{char}
+ at var BITFIELD: the code for bitfield messages
+ at type REQUEST: C{char}
+ at var REQUEST: the code for request messages
+ at type PIECE: C{char}
+ at var PIECE: the code for piece messages
+ at type CANCEL: C{char}
+ at var CANCEL: the code for cancel messages
+
+"""
from DebTorrent.bitfield import Bitfield
from DebTorrent.clock import clock
@@ -18,9 +45,23 @@
DEBUG2 = True
def toint(s):
+ """Convert four-byte big endian representation to a long.
+
+ @type s: C{string}
+ @param s: the string to convert
+
+ """
+
return long(b2a_hex(s), 16)
def tobinary(i):
+ """Convert an integer to a four-byte big endian representation.
+
+ @type i: C{int}
+ @param i: the integer to convert
+
+ """
+
return (chr(i >> 24) + chr((i >> 16) & 0xFF) +
chr((i >> 8) & 0xFF) + chr(i & 0xFF))
@@ -40,7 +81,46 @@
CANCEL = chr(8)
class Connection:
+ """A connection to an individual peer.
+
+ @type connection: unknown
+ @ivar connection: the connection
+ @type connecter: L{Connecter}
+ @ivar connecter: the collection of all connections
+ @type ccount: C{int}
+ @ivar ccount: the number of the connection
+ @type got_anything: C{boolean}
+ @ivar got_anything: whether a message has ever been received on the connection
+ @type next_upload: unknown
+ @ivar next_upload: the connection that will next be allowed to upload
+ @type outqueue: C{list}
+ @ivar outqueue: the queue of messages to send on the connection that are
+ waiting for the current piece to finish sending
+ @type partial_message: C{string}
+ @ivar partial_message: the remaining data in the current piece being sent
+ @type upload: C{Uploader.Upload}
+ @ivar upload: the Uploader instance to use for the connection
+ @type download: C{Downloader.Downloader}
+ @ivar download: the Downloader instance to use for the connection
+ @type send_choke_queued: C{boolean}
+ @ivar send_choke_queued: whether to suppress the next L{CHOKE} message
+ @type just_unchoked: C{long}
+ @ivar just_unchoked: the time of a recent L{UNCHOKE}, if it was the first
+
+ """
+
def __init__(self, connection, connecter, ccount):
+ """Initialize the class.
+
+ @type connection: unknown
+ @param connection: the connection
+ @type connecter: L{Connecter}
+ @param connecter: the collection of all connections
+ @type ccount: C{int}
+ @param ccount: the number of the connection
+
+ """
+
self.connection = connection
self.connecter = connecter
self.ccount = ccount
@@ -53,32 +133,71 @@
self.just_unchoked = None
def get_ip(self, real=False):
+ """Get the IP address of the connection.
+
+ @type real: C{boolean}
+ @param real: unknown (optional, defaults to False)
+
+ """
+
return self.connection.get_ip(real)
def get_id(self):
+ """Get the Peer ID of the connection.
+
+ @rtype: C{string}
+ @return: the ID of the connection
+
+ """
+
return self.connection.get_id()
def get_readable_id(self):
+ """Get a human readable version of the ID of the connection.
+
+ @rtype: C{string}
+ @return: the ID of the connection
+
+ """
+
return self.connection.get_readable_id()
def close(self):
+ """Close the connection."""
if DEBUG1:
- print (self.ccount,'connection closed')
+ print (self.get_ip(),'connection closed')
self.connection.close()
def is_locally_initiated(self):
+ """Check whether the connection was established by the client.
+
+ @rtype: C{boolean}
+ @return: whether the connection was established by the client
+
+ """
+
return self.connection.is_locally_initiated()
def is_encrypted(self):
+ """Check whether the connection is encrypted.
+
+ @rtype: C{boolean}
+ @return: whether the connection is encrypted
+
+ """
+
return self.connection.is_encrypted()
def send_interested(self):
+ """Send the L{INTERESTED} message."""
self._send_message(INTERESTED)
def send_not_interested(self):
+ """Send the L{NOT_INTERESTED} message."""
self._send_message(NOT_INTERESTED)
def send_choke(self):
+ """Send the L{CHOKE} message."""
if self.partial_message:
self.send_choke_queued = True
else:
@@ -87,10 +206,11 @@
self.just_unchoked = 0
def send_unchoke(self):
+ """Send the L{UNCHOKE} message."""
if self.send_choke_queued:
self.send_choke_queued = False
if DEBUG1:
- print (self.ccount,'CHOKE SUPPRESSED')
+ print (self.get_ip(),'CHOKE SUPPRESSED')
else:
self._send_message(UNCHOKE)
if ( self.partial_message or self.just_unchoked is None
@@ -100,32 +220,78 @@
self.just_unchoked = clock()
def send_request(self, index, begin, length):
+ """Send the L{REQUEST} message.
+
+ @type index: C{int}
+ @param index: the piece to request some of
+ @type begin: C{int}
+ @param begin: the starting offset within the piece
+ @type length: C{int}
+ @param length: the length of the part of the piece to get
+
+ """
+
self._send_message(REQUEST + tobinary(index) +
tobinary(begin) + tobinary(length))
if DEBUG1:
- print (self.ccount,'sent request',index,begin,begin+length)
+ print (self.get_ip(),'sent request',index,begin,begin+length)
def send_cancel(self, index, begin, length):
+ """Send the L{CANCEL} message.
+
+ Cancels a previously sent L{REQUEST} message.
+
+ @type index: C{int}
+ @param index: the piece that was requested
+ @type begin: C{int}
+ @param begin: the starting offset within the piece
+ @type length: C{int}
+ @param length: the length of the part of the piece to get
+
+ """
+
self._send_message(CANCEL + tobinary(index) +
tobinary(begin) + tobinary(length))
if DEBUG1:
- print (self.ccount,'sent cancel',index,begin,begin+length)
+ print (self.get_ip(),'sent cancel',index,begin,begin+length)
def send_bitfield(self, bitfield):
+ """Send the L{BITFIELD} message.
+
+ @type bitfield: C{string}
+ @param bitfield: the bitfield to send
+
+ """
+
self._send_message(BITFIELD + bitfield)
def send_have(self, index):
+ """Send the L{HAVE} message.
+
+ @type index: C{int}
+ @param index: the piece index to indicate having
+
+ """
+
self._send_message(HAVE + tobinary(index))
def send_keepalive(self):
+ """Send a keepalive message."""
self._send_message('')
def _send_message(self, s):
+ """Actually send the message.
+
+ @type s: C{string}
+ @param s: the message to send
+
+ """
+
if DEBUG2:
if s:
- print (self.ccount,'SENDING MESSAGE',ord(s[0]),len(s))
+ print (self.get_ip(),'SENDING MESSAGE',ord(s[0]),len(s))
else:
- print (self.ccount,'SENDING MESSAGE',-1,0)
+ print (self.get_ip(),'SENDING MESSAGE','keepalive',0)
s = tobinary(len(s))+s
if self.partial_message:
self.outqueue.append(s)
@@ -133,6 +299,15 @@
self.connection.send_message_raw(s)
def send_partial(self, bytes):
+ """Send a L{PIECE} message containing part of a piece.
+
+ @type bytes: C{int}
+ @param bytes: the number of bytes of piece data to send
+ @rtype: C{int}
+ @return: the actual number of bytes sent
+
+ """
+
if self.connection.closed:
return 0
if self.partial_message is None:
@@ -144,7 +319,7 @@
tobinary(len(piece) + 9), PIECE,
tobinary(index), tobinary(begin), piece.tostring() ))
if DEBUG1:
- print (self.ccount,'sending chunk',index,begin,begin+len(piece))
+ print (self.get_ip(),'sending chunk',index,begin,begin+len(piece))
if bytes < len(self.partial_message):
self.connection.send_message_raw(self.partial_message[:bytes])
@@ -165,18 +340,57 @@
return len(q)
def get_upload(self):
+ """Get the L{Uploader.Upload} instance for this connection.
+
+ @rtype: L{Uploader.Upload}
+ @return: the Upload instance
+
+ """
+
return self.upload
def get_download(self):
+ """Get the L{Downloader.Downloader} instance for this connection.
+
+ @rtype: L{Downloader.Downloader}
+ @return: the Downloader instance
+
+ """
+
return self.download
def set_download(self, download):
+ """Set the L{Downloader.Downloader} instance for this connection.
+
+ @type download: L{Downloader.Downloader}
+ @param download: the Downloader instance
+
+ """
+
self.download = download
def backlogged(self):
+ """Check whether the connection is ready to send.
+
+ @rtype: C{boolean}
+ @return: whether the connection is backlogged
+
+ """
+
return not self.connection.is_flushed()
def got_request(self, i, p, l):
+ """Process a request from a peer for a part of a piece.
+
+ @type i: C{int}
+ @param i: the piece index
+ @type p: C{int}
+ @param p: the position to start at
+ @type l: C{int}
+ @param l: the length to send
+
+ """
+
self.upload.got_request(i, p, l)
if self.just_unchoked:
self.connecter.ratelimiter.ping(clock() - self.just_unchoked)
@@ -186,8 +400,59 @@
class Connecter:
+ """A collection of all connections to peers.
+
+ @type downloader: L{Downloader.Downloader}
+ @ivar downloader: the Downloader instance to use
+ @type make_upload: C{method}
+ @ivar make_upload: the method to create a new L{Uploader.Upload}
+ @type choker: L{Choker.Choker}
+ @ivar choker: the Choker instance to use
+ @type numpieces: C{int}
+ @ivar numpieces: the number of pieces in the download
+ @type config: C{dictionary}
+ @ivar config: the configration information
+ @type ratelimiter: L{RateLimiter.RateLimiter}
+ @ivar ratelimiter: the RateLimiter instance to use
+ @type rate_capped: unknown
+ @ivar rate_capped: unknown
+ @type sched: unknown
+ @ivar sched: unknown
+ @type totalup: L{Debtorrent.CurrentRateMeasure.Measure}
+ @ivar totalup: the Measure instance to use
+ @type connections: C{dictionary}
+ @ivar connections: the collection of connections that are open
+ @type external_connection_made: C{int}
+ @ivar external_connection_made: greater than 0 if there have been external connections
+ @type ccount: C{int}
+ @ivar ccount: the largest connection number used
+
+ """
+
def __init__(self, make_upload, downloader, choker, numpieces,
totalup, config, ratelimiter, sched = None):
+ """
+
+ @type make_upload: C{method}
+ @param make_upload: the method to create a new L{Uploader.Upload}
+ @type downloader: L{Downloader.Downloader}
+ @param downloader: the Downloader instance to use
+ @type choker: L{Choker.Choker}
+ @param choker: the Choker instance to use
+ @type numpieces: C{int}
+ @param numpieces: the number of pieces in the download
+ @type totalup: L{Debtorrent.CurrentRateMeasure.Measure}
+ @param totalup: the Measure instance to use
+ @type config: C{dictionary}
+ @param config: the configration information
+ @type ratelimiter: L{RateLimiter.RateLimiter}
+ @param ratelimiter: the RateLimiter instance to use
+ @type sched: C{method}
+ @param sched: the method to call to schedule future actions
+ (optional, default is None)
+
+ """
+
self.downloader = downloader
self.make_upload = make_upload
self.choker = choker
@@ -203,13 +468,29 @@
self.ccount = 0
def how_many_connections(self):
+ """Get the number of currently open connections.
+
+ @rtype: C{int}
+ @return: the number of open connections
+
+ """
+
return len(self.connections)
def connection_made(self, connection):
+ """Make a new connection.
+
+ @type connection: unknown
+ @param connection: the new connection to make
+ @rtype: L{Connection}
+ @return: the new connection
+
+ """
+
self.ccount += 1
c = Connection(connection, self, self.ccount)
if DEBUG2:
- print (c.ccount,'connection made')
+ print (c.get_ip(),'connection made')
self.connections[connection] = c
c.upload = self.make_upload(c, self.ratelimiter, self.totalup)
c.download = self.downloader.make_download(c)
@@ -217,39 +498,69 @@
return c
def connection_lost(self, connection):
+ """Process a lost connection.
+
+ @type connection: unknown
+ @param connection: the connection that was lost
+
+ """
+
c = self.connections[connection]
if DEBUG2:
- print (c.ccount,'connection closed')
+ print (c.get_ip(),'connection closed')
del self.connections[connection]
if c.download:
c.download.disconnected()
self.choker.connection_lost(c)
def connection_flushed(self, connection):
+ """Process a flushed connection.
+
+ @type connection: unknown
+ @param connection: the connection that was flushed
+
+ """
+
conn = self.connections[connection]
if conn.next_upload is None and (conn.partial_message is not None
or len(conn.upload.buffer) > 0):
self.ratelimiter.queue(conn)
def got_piece(self, i):
+ """Alert all the open connections that a piece was received.
+
+ @type i: C{int}
+ @param i: the piece index that was received
+
+ """
+
for co in self.connections.values():
co.send_have(i)
def got_message(self, connection, message):
+ """Process a received message on a connection.
+
+ @type connection: unknown
+ @param connection: the connection that the message was received on
+ @type message: C{string}
+ @param message: the message that was received
+
+ """
+
c = self.connections[connection]
t = message[0]
if DEBUG2:
- print (c.ccount,'message received',ord(t))
+ print (c.get_ip(),'message received',ord(t))
if t == BITFIELD and c.got_anything:
if DEBUG2:
- print (c.ccount,'misplaced bitfield')
+ print (c.get_ip(),'misplaced bitfield')
connection.close()
return
c.got_anything = True
if (t in [CHOKE, UNCHOKE, INTERESTED, NOT_INTERESTED] and
len(message) != 1):
if DEBUG2:
- print (c.ccount,'bad message length')
+ print (c.get_ip(),'bad message length')
connection.close()
return
if t == CHOKE:
@@ -264,13 +575,13 @@
elif t == HAVE:
if len(message) != 5:
if DEBUG2:
- print (c.ccount,'bad message length')
+ print (c.get_ip(),'bad message length')
connection.close()
return
i = toint(message[1:])
if i >= self.numpieces:
if DEBUG2:
- print (c.ccount,'bad piece number')
+ print (c.get_ip(),'bad piece number')
connection.close()
return
if c.download.got_have(i):
@@ -280,7 +591,7 @@
b = Bitfield(self.numpieces, message[1:])
except ValueError:
if DEBUG2:
- print (c.ccount,'bad bitfield')
+ print (c.get_ip(),'bad bitfield')
connection.close()
return
if c.download.got_have_bitfield(b):
@@ -288,13 +599,13 @@
elif t == REQUEST:
if len(message) != 13:
if DEBUG2:
- print (c.ccount,'bad message length')
+ print (c.get_ip(),'bad message length')
connection.close()
return
i = toint(message[1:5])
if i >= self.numpieces:
if DEBUG2:
- print (c.ccount,'bad piece number')
+ print (c.get_ip(),'bad piece number')
connection.close()
return
c.got_request(i, toint(message[5:9]),
@@ -302,13 +613,13 @@
elif t == CANCEL:
if len(message) != 13:
if DEBUG2:
- print (c.ccount,'bad message length')
+ print (c.get_ip(),'bad message length')
connection.close()
return
i = toint(message[1:5])
if i >= self.numpieces:
if DEBUG2:
- print (c.ccount,'bad piece number')
+ print (c.get_ip(),'bad piece number')
connection.close()
return
c.upload.got_cancel(i, toint(message[5:9]),
@@ -316,16 +627,18 @@
elif t == PIECE:
if len(message) <= 9:
if DEBUG2:
- print (c.ccount,'bad message length')
+ print (c.get_ip(),'bad message length')
connection.close()
return
i = toint(message[1:5])
if i >= self.numpieces:
if DEBUG2:
- print (c.ccount,'bad piece number')
+ print (c.get_ip(),'bad piece number')
connection.close()
return
if c.download.got_piece(i, toint(message[5:9]), message[9:]):
self.got_piece(i)
else:
+ if DEBUG2:
+ print (c.get_ip(),'unknown message type')
connection.close()
Modified: debtorrent/branches/http-listen/DebTorrent/BT1/Downloader.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/BT1/Downloader.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/BT1/Downloader.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/BT1/Downloader.py Tue May 29 05:58:07 2007
@@ -1,8 +1,15 @@
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+
+"""Download pieces from remote peers.
+
+ at type EXPIRE_TIME: C{int}
+ at var EXPIRE_TIME: number of seconds after which disconnected seeds are expired
+
+"""
from DebTorrent.CurrentRateMeasure import Measure
from DebTorrent.bitfield import Bitfield
@@ -16,8 +23,31 @@
EXPIRE_TIME = 60 * 60
-class PerIPStats:
+class PerIPStats:
+ """Statistics relating to downloads from a single peer.
+
+ @type numgood: C{int}
+ @ivar numgood: the number of good pieces received
+ @type bad: C{dictionary}
+ @ivar bad: keys are piece numbers, values are the number of bad copies
+ of the piece received from the peer
+ @type numconnections: C{int}
+ @ivar numconnections: the number of connections made to the peer
+ @type lastdownload: L{SingleDownload}
+ @ivar lastdownload: the most recent SingleDownload instance
+ @type peerid: C{string}
+ @ivar peerid: the peer's ID
+
+ """
+
def __init__(self, ip):
+ """Initialize the statistics.
+
+ @type ip: unknown
+ @param ip: the IP address of the peer (not used)
+
+ """
+
self.numgood = 0
self.bad = {}
self.numconnections = 0
@@ -25,7 +55,29 @@
self.peerid = None
class BadDataGuard:
+ """Process good and bad received pieces from a single peer.
+
+ @type download: L{SingleDownload}
+ @ivar download: the SingleDownload instance
+ @type ip: C{string}
+ @ivar ip: IP address of the peer
+ @type downloader: L{Downloader}
+ @ivar downloader: the Downloader instance
+ @type stats: L{PerIPStats}
+ @ivar stats: the PerIPStats instance
+ @type lastindex: C{int}
+ @ivar lastindex: the last good piece that was received
+
+ """
+
def __init__(self, download):
+ """Initialize the class.
+
+ @type download: L{SingleDownload}
+ @param download: the SingleDownload instance for the download
+
+ """
+
self.download = download
self.ip = download.ip
self.downloader = download.downloader
@@ -33,6 +85,16 @@
self.lastindex = None
def failed(self, index, bump = False):
+ """Process the failed piece.
+
+ @type index: C{int}
+ @param index: the piece that failed
+ @type bump: C{boolean}
+ @param bump: whether to increase the interest level in the
+ L{PiecePicker.PiecePicker} (optional, defaults to False)
+
+ """
+
self.stats.bad.setdefault(index, 0)
self.downloader.gotbaddata[self.ip] = 1
self.stats.bad[index] += 1
@@ -47,6 +109,13 @@
self.downloader.picker.bump(index)
def good(self, index):
+ """Process the successful piece.
+
+ @type index: C{int}
+ @param index: the piece that succeeded
+
+ """
+
# lastindex is a hack to only increase numgood by one for each good
# piece, however many chunks come from the connection(s) from this IP
if index != self.lastindex:
@@ -54,7 +123,49 @@
self.lastindex = index
class SingleDownload:
+ """Manage downloads from a single peer.
+
+ @type downloader: L{Downloader}
+ @ivar downloader: the Downloader instance
+ @type connection: unknown
+ @ivar connection: the connection to the peer
+ @type choked: C{boolean}
+ @ivar choked: whether the peer is choking the download
+ @type interested: C{boolean}
+ @ivar interested: whether the peer is interesting
+ @type active_requests: C{list}
+ @ivar active_requests: unknown
+ @type measure: L{DebTorrent.CurrentRateMeasure.Measure}
+ @ivar measure: for measuring the download rate from the peer
+ @type peermeasure: L{DebTorrent.CurrentRateMeasure.Measure}
+ @ivar peermeasure: for measuring the download rate of the peer
+ @type have: L{DebTorrent.bitfield.Bitfield}
+ @ivar have: the bitfield the peer has
+ @type last: C{float}
+ @ivar last: the last time a chunk was received from the peer
+ @type last2: C{float}
+ @ivar last2: the last time a chunk or an unchoke was received
+ @type example_interest: C{int}
+ @ivar example_interest: an example piece to request
+ @type backlog: C{int}
+ @ivar backlog: the current backlog of chunk requests
+ @type ip: C{string}
+ @ivar ip: the IP address of the peer
+ @type guard: L{BadDataGuard}
+ @ivar guard: the guard to use to process pieces
+
+ """
+
def __init__(self, downloader, connection):
+ """Initialize the instance.
+
+ @type downloader: L{Downloader}
+ @param downloader: the parent Downloader instance
+ @type connection: unknown
+ @param connection: the connection to the peer
+
+ """
+
self.downloader = downloader
self.connection = connection
self.choked = True
@@ -71,6 +182,15 @@
self.guard = BadDataGuard(self)
def _backlog(self, just_unchoked):
+ """Calculate the backlog of chunk requests to the peer.
+
+ @type just_unchoked: C{boolean}
+ @param just_unchoked: whether the connection was just unchoked
+ @rtype: C{int}
+ @return: the new backlog
+
+ """
+
self.backlog = min(
2+int(4*self.measure.get_rate()/self.downloader.chunksize),
(2*just_unchoked)+self.downloader.queue_limit() )
@@ -79,6 +199,7 @@
return self.backlog
def disconnected(self):
+ """Remove the newly disconnected peer."""
self.downloader.lost_peer(self)
if self.have.complete():
self.downloader.picker.lost_seed()
@@ -92,6 +213,14 @@
self.guard.download = None
def _letgo(self):
+ """Remove the oustanding requests to the peer.
+
+ For each active request that was unfulfilled by the peer, inform the
+ Storage that the request was lost, and send interested messages to any
+ remaining peers that have the piece.
+
+ """
+
if self.downloader.queued_out.has_key(self):
del self.downloader.queued_out[self]
if not self.active_requests:
@@ -119,11 +248,13 @@
break
def got_choke(self):
+ """Update the choked status and remove any active requests."""
if not self.choked:
self.choked = True
self._letgo()
def got_unchoke(self):
+ """Update the status and request any needed pieces."""
if self.choked:
self.choked = False
if self.interested:
@@ -131,12 +262,27 @@
self.last2 = clock()
def is_choked(self):
+ """Get the choked status of the connection.
+
+ @rtype: C{boolean}
+ @return: whether the peer is choking the connection
+
+ """
+
return self.choked
def is_interested(self):
+ """Get the interest in the peer.
+
+ @rtype: C{boolean}
+ @return: whether the peer is interesting
+
+ """
+
return self.interested
def send_interested(self):
+ """Send the interested message to the peer."""
if not self.interested:
self.interested = True
self.connection.send_interested()
@@ -144,11 +290,28 @@
self.last2 = clock()
def send_not_interested(self):
+ """Send the not interested message to the peer."""
if self.interested:
self.interested = False
self.connection.send_not_interested()
def got_piece(self, index, begin, piece):
+ """Process a received chunk.
+
+ Add the newly received chunk to the Storage, remove any oustanding
+ requests for it, and request more chunks from the peer.
+
+ @type index: C{int}
+ @param index: the piece index
+ @type begin: C{int}
+ @param begin: the offset within the piece
+ @type piece: C{string}
+ @param piece: the chunk
+ @rtype: C{boolean}
+ @return: whether the piece was accepted by the Storage (valid)
+
+ """
+
length = len(piece)
try:
self.active_requests.remove((index, begin, length))
@@ -187,6 +350,14 @@
return self.downloader.storage.do_I_have(index)
def _request_more(self, new_unchoke = False):
+ """Request more chunks from the peer.
+
+ @type new_unchoke: C{boolean}
+ @param new_unchoke: whether this request was the result of a recent
+ unchoke (optional, defaults to False)
+
+ """
+
assert not self.choked
if self.downloader.endgamemode:
self.fix_download_endgame(new_unchoke)
@@ -241,6 +412,14 @@
def fix_download_endgame(self, new_unchoke = False):
+ """Request more chunks from the peer in endgame mode.
+
+ @type new_unchoke: C{boolean}
+ @param new_unchoke: whether this request was the result of a recent
+ unchoke (optional, defaults to False)
+
+ """
+
if self.downloader.paused:
return
if len(self.active_requests) >= self._backlog(new_unchoke):
@@ -263,6 +442,15 @@
self.downloader.chunk_requested(length)
def got_have(self, index):
+ """Receive a Have message from the peer.
+
+ @type index: C{int}
+ @param index: the piece the peer now has
+ @rtype: C{boolean}
+ @return: whether the peer is now a seed
+
+ """
+
self.downloader.totalmeasure.update_rate(self.downloader.storage.piece_lengths[index])
self.peermeasure.update_rate(self.downloader.storage.piece_lengths[index])
if not self.have[index]:
@@ -285,6 +473,7 @@
return self.have.complete()
def _check_interests(self):
+ """Check if the peer is now interesting."""
if self.interested or self.downloader.paused:
return
for i in xrange(len(self.have)):
@@ -295,6 +484,15 @@
return
def got_have_bitfield(self, have):
+ """Receive a Bitfield message from the peer.
+
+ @type have: L{DebTorrent.bitfield.Bitfield}
+ @param have: the bitfield received from the peer
+ @rtype: C{boolean}
+ @return: whether the peer is a seed
+
+ """
+
if self.downloader.storage.am_I_complete() and have.complete():
if self.downloader.super_seeding:
self.connection.send_bitfield(have.tostring()) # be nice, show you're a seed too
@@ -318,9 +516,23 @@
return have.complete()
def get_rate(self):
+ """Get the current download rate from the peer.
+
+ @rtype: C{float}
+ @return: the peer's download rate
+
+ """
+
return self.measure.get_rate()
def is_snubbed(self):
+ """Check if the peer is snubbing the download.
+
+ @rtype: C{boolean}
+ @return: whether the peer is snubbing the connection
+
+ """
+
if ( self.interested and not self.choked
and clock() - self.last2 > self.downloader.snub_time ):
for index, begin, length in self.active_requests:
@@ -330,9 +542,109 @@
class Downloader:
+ """A collection of all single downloads.
+
+ @type storage: L{StorageWrapper.StorageWrapper}
+ @ivar storage: the StorageWrapper instance
+ @type picker: L{PiecePicker.PiecePicker}
+ @ivar picker: the PiecePicker instance
+ @type backlog: C{int}
+ @ivar backlog: the maximum number of requests for a single connection
+ @type max_rate_period: C{float}
+ @ivar max_rate_period: maximum amount of time to guess the current
+ rate estimate represents
+ @type measurefunc: C{method}
+ @ivar measurefunc: the method to call to add downloaded data to the
+ measurement of the download rate
+ @type totalmeasure: L{DebTorrent.CurrentRateMeasure.Measure}
+ @ivar totalmeasure: for measuring the total download rate from all peers
+ @type numpieces: C{int}
+ @ivar numpieces: total number of pieces in the download
+ @type chunksize: C{int}
+ @ivar chunksize: the number of bytes to query for per request
+ @type snub_time: C{float}
+ @ivar snub_time: seconds to wait for data to come in over a connection
+ before assuming it's semi-permanently choked
+ @type kickfunc: C{method}
+ @ivar kickfunc: method to call to kick a peer
+ @type banfunc: C{method}
+ @ivar banfunc: method to call to ban a peer
+ @type disconnectedseeds: C{dictionary}
+ @ivar disconnectedseeds: unknown
+ @type downloads: C{list} of C{SingleDownload}
+ @ivar downloads: unknown
+ @type perip: C{dictionary}
+ @ivar perip: unknown
+ @type gotbaddata: C{dictionary}
+ @ivar gotbaddata: unknown
+ @type kicked: C{dictionary}
+ @ivar kicked: unknown
+ @type banned: C{dictionary}
+ @ivar banned: unknown
+ @type kickbans_ok: C{boolean}
+ @ivar kickbans_ok: whether to automatically kick/ban peers that send
+ bad data
+ @type kickbans_halted: C{boolean}
+ @ivar kickbans_halted: unknown
+ @type super_seeding: C{boolean}
+ @ivar super_seeding: unknown
+ @type endgamemode: C{boolean}
+ @ivar endgamemode: unknown
+ @type endgame_queued_pieces: C{list}
+ @ivar endgame_queued_pieces: unknown
+ @type all_requests: C{list}
+ @ivar all_requests: unknown
+ @type discarded: C{long}
+ @ivar discarded: unknown
+ @type download_rate: C{float}
+ @ivar download_rate: the maximum rate to download at
+ @type bytes_requested: C{int}
+ @ivar bytes_requested: the number of bytes in oustanding requests
+ @type last_time: C{float}
+ @ivar last_time: the last time the queue limit was calculated
+ @type queued_out: C{dictionary}
+ @ivar queued_out: unknown
+ @type requeueing: C{boolean}
+ @ivar requeueing: unknown
+ @type paused: C{boolean}
+ @ivar paused: unknown
+
+ """
+
def __init__(self, storage, picker, backlog, max_rate_period,
numpieces, chunksize, measurefunc, snub_time,
kickbans_ok, kickfunc, banfunc):
+ """Initialize the instance.
+
+ @type storage: L{StorageWrapper.StorageWrapper}
+ @param storage: the StorageWrapper instance
+ @type picker: L{PiecePicker.PiecePicker}
+ @param picker: the PiecePicker instance
+ @type backlog: C{int}
+ @param backlog: the maximum number of requests for a single connection
+ @type max_rate_period: C{float}
+ @param max_rate_period: maximum amount of time to guess the current
+ rate estimate represents
+ @type numpieces: C{int}
+ @param numpieces: total number of pieces in the download
+ @type chunksize: C{int}
+ @param chunksize: the number of bytes to query for per request
+ @type measurefunc: C{method}
+ @param measurefunc: the method to call to add downloaded data to the
+ measurement of the download rate
+ @type snub_time: C{float}
+ @param snub_time: seconds to wait for data to come in over a connection
+ before assuming it's semi-permanently choked
+ @type kickbans_ok: C{boolean}
+ @param kickbans_ok: whether to automatically kick/ban peers that send
+ bad data
+ @type kickfunc: C{method}
+ @param kickfunc: method to call to kick a peer
+ @type banfunc: C{method}
+ @param banfunc: method to call to ban a peer
+
+ """
+
self.storage = storage
self.picker = picker
self.backlog = backlog
@@ -366,10 +678,24 @@
self.paused = False
def set_download_rate(self, rate):
+ """Set the maximum download rate for all downloads.
+
+ @type rate: C{float}
+ @param rate: maximum kB/s to download at (0 = no limit)
+
+ """
+
self.download_rate = rate * 1000
self.bytes_requested = 0
def queue_limit(self):
+ """Get the maximum number of bytes to request.
+
+ @rtype: C{int}
+ @return: the limit on the number of bytes to request
+
+ """
+
if not self.download_rate:
return 10e10 # that's a big queue!
t = clock()
@@ -388,11 +714,27 @@
return max(int(-self.bytes_requested/self.chunksize),0)
def chunk_requested(self, size):
+ """Add the new request size to the tally.
+
+ @type size: C{int}
+ @param size: the number of bytes that were requested
+
+ """
+
self.bytes_requested += size
external_data_received = chunk_requested
def make_download(self, connection):
+ """Create a new L{SingleDownload} instance for a new connection.
+
+ @type connection: unknown
+ @param connection: the connection that was received
+ @rtype: L{SingleDownload}
+ @return: the newly created SingleDownload instance
+
+ """
+
ip = connection.get_ip()
if self.perip.has_key(ip):
perip = self.perip[ip]
@@ -406,6 +748,13 @@
return d
def piece_flunked(self, index):
+ """Request a failed piece from other peers.
+
+ @type index: C{int}
+ @param index: the piece index that failed
+
+ """
+
if self.paused:
return
if self.endgamemode:
@@ -428,9 +777,23 @@
d.send_interested()
def has_downloaders(self):
+ """Get the number of active downloads.
+
+ @rtype: C{int}
+ @return: the number of active download connections
+
+ """
+
return len(self.downloads)
def lost_peer(self, download):
+ """Remove a lost peer from the collection of downloads.
+
+ @type download: L{SingleDownload}
+ @param download: the download peer that was lost
+
+ """
+
ip = download.ip
self.perip[ip].numconnections -= 1
if self.perip[ip].lastdownload == download:
@@ -439,7 +802,8 @@
if self.endgamemode and not self.downloads: # all peers gone
self._reset_endgame()
- def _reset_endgame(self):
+ def _reset_endgame(self):
+ """Stop the endgame mode."""
self.storage.reset_endgame(self.all_requests)
self.endgamemode = False
self.all_requests = []
@@ -447,6 +811,13 @@
def add_disconnected_seed(self, id):
+ """Save the time of a disconnected seed.
+
+ @type id: C{string}
+ @param id: the peer ID of the disconnected seed
+
+ """
+
# if not self.disconnectedseeds.has_key(id):
# self.picker.seed_seen_recently()
self.disconnectedseeds[id]=clock()
@@ -454,6 +825,13 @@
# def expire_disconnected_seeds(self):
def num_disconnected_seeds(self):
+ """Expire old disconnected seeds and calculate the recent number.
+
+ @rtype: C{int}
+ @return: the number of recently seen disconnected seeds
+
+ """
+
# first expire old ones
expired = []
for id,t in self.disconnectedseeds.items():
@@ -467,12 +845,26 @@
# it should be scheduled to run every minute or two.
def _check_kicks_ok(self):
+ """Check whether peers can be kicked for bad data.
+
+ @rtype: C{boolean}
+ @return: whether it is OK to kick peers
+
+ """
+
if len(self.gotbaddata) > 10:
self.kickbans_ok = False
self.kickbans_halted = True
return self.kickbans_ok and len(self.downloads) > 2
def try_kick(self, download):
+ """If allowed, kick a peer.
+
+ @type download: L{SingleDownload}
+ @param download: the peer's download connection
+
+ """
+
if self._check_kicks_ok():
download.guard.download = None
ip = download.ip
@@ -482,6 +874,13 @@
self.kickfunc(download.connection)
def try_ban(self, ip):
+ """If allowed, ban a peer.
+
+ @type ip: C{string}
+ @param ip: the IP address of the peer
+
+ """
+
if self._check_kicks_ok():
self.banfunc(ip)
self.banned[ip] = self.perip[ip].peerid
@@ -489,9 +888,22 @@
del self.kicked[ip]
def set_super_seed(self):
+ """Enable super-seed mode."""
self.super_seeding = True
def check_complete(self, index):
+ """Check whether the download is complete.
+
+ If it is complete, send the piece to the connected seeds and then
+ disconnect them.
+
+ @type index: C{int}
+ @param index: the last received piece
+ @rtype: C{boolean}
+ @return: whether the download is complete
+
+ """
+
if self.endgamemode and not self.all_requests:
self.endgamemode = False
if self.endgame_queued_pieces and not self.endgamemode:
@@ -507,10 +919,25 @@
return False
def too_many_partials(self):
+ """Check whether there are too many outstanding incomplete pieces.
+
+ @rtype: C{boolean}
+ @return: if the number of incomplete pieces is greater than half the
+ number of connected downloads
+
+ """
+
return len(self.storage.dirty) > (len(self.downloads)/2)
def cancel_piece_download(self, pieces):
+ """Cancel any outstanding requests for the pieces.
+
+ @type pieces: C{list} of C{int}
+ @param pieces: the list of pieces to cancel
+
+ """
+
if self.endgamemode:
if self.endgame_queued_pieces:
for piece in pieces:
@@ -542,6 +969,13 @@
d._check_interests()
def requeue_piece_download(self, pieces = []):
+ """Request more pieces.
+
+ @type pieces: C{list} of C{int}
+ @param pieces: the list of pieces to requeue
+
+ """
+
if self.endgame_queued_pieces:
for piece in pieces:
if not piece in self.endgame_queued_pieces:
@@ -563,6 +997,7 @@
d._request_more()
def start_endgame(self):
+ """Switch to endgame mode."""
assert not self.endgamemode
self.endgamemode = True
assert not self.all_requests
@@ -576,6 +1011,13 @@
d.fix_download_endgame()
def pause(self, flag):
+ """Pause or unpause the download.
+
+ @type flag: C{boolean}
+ @param flag: whether to pause of unpause.
+
+ """
+
self.paused = flag
if flag:
for d in self.downloads:
Modified: debtorrent/branches/http-listen/DebTorrent/BT1/Encrypter.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/BT1/Encrypter.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/BT1/Encrypter.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/BT1/Encrypter.py Tue May 29 05:58:07 2007
@@ -586,6 +586,8 @@
if self.config['security'] and ip != 'unknown' and ip == dns[0]:
return True
try:
+ if DEBUG:
+ print 'initiating connection to:', dns, id, encrypted
c = self.raw_server.start_connection(dns)
con = Connection(self, c, id, encrypted = encrypted)
self.connections[c] = con
Modified: debtorrent/branches/http-listen/DebTorrent/BT1/FileSelector.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/BT1/FileSelector.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/BT1/FileSelector.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/BT1/FileSelector.py Tue May 29 05:58:07 2007
@@ -117,6 +117,30 @@
self.new_partials = None
+ def _initialize_files_disabled(self, old_priority, new_priority):
+ old_disabled = [p == -1 for p in old_priority]
+ new_disabled = [p == -1 for p in new_priority]
+ files_updated = False
+ try:
+ for f in xrange(self.numfiles):
+ if new_disabled[f] and not old_disabled[f]:
+ self.storage.disable_file(f)
+ files_updated = True
+ if old_disabled[f] and not new_disabled[f]:
+ self.storage.enable_file(f)
+ files_updated = True
+ except (IOError, OSError), e:
+ if new_disabled[f]:
+ msg = "can't open partial file for "
+ else:
+ msg = 'unable to open '
+ self.failfunc(msg + self.files[f][0] + ': ' + str(e))
+ return False
+ if files_updated:
+ self.storage.reset_file_status()
+ return True
+
+
def _set_files_disabled(self, old_priority, new_priority):
old_disabled = [p == -1 for p in old_priority]
new_disabled = [p == -1 for p in new_priority]
@@ -202,6 +226,15 @@
return new_piece_priority
+ def initialize_priorities_now(self, new_priority = None):
+ if not new_priority:
+ return
+ old_priority = self.priority
+ self.priority = new_priority
+ if not self._initialize_files_disabled(old_priority, new_priority):
+ return
+# self.piece_priority = self._set_piece_priority(new_priority)
+
def set_priorities_now(self, new_priority = None):
if not new_priority:
new_priority = self.new_priority
Modified: debtorrent/branches/http-listen/DebTorrent/BT1/Rerequester.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/BT1/Rerequester.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/BT1/Rerequester.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/BT1/Rerequester.py Tue May 29 05:58:07 2007
@@ -28,6 +28,8 @@
except:
True = 1
False = 0
+
+DEBUG = True
mapbase64 = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz.-'
keys = {}
@@ -383,6 +385,8 @@
x = p[i]
peers.append(((x['ip'].strip(), x['port']),
x.get('peer id',0), cflags[i]))
+ if DEBUG:
+ print 'received from tracker:', peers
ps = len(peers) + self.howmany()
if ps < self.maxpeers:
if self.doneflag.isSet():
Modified: debtorrent/branches/http-listen/DebTorrent/BT1/Storage.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/BT1/Storage.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/BT1/Storage.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/BT1/Storage.py Tue May 29 05:58:07 2007
@@ -1,14 +1,34 @@
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+
+"""Low-level writing of files.
+
+
+
+ at type DEBUG: C{boolean}
+ at var DEBUG: whether to enable printing of debug messages
+ at type MAXREADSIZE: C{long}
+ at var MAXREADSIZE: the maximum number of bytes that can be read at a time
+ at type MAXLOCKSIZE: C{long}
+ at var MAXLOCKSIZE: the maximum size to lock at a time (windows only)
+ at type MAXLOCKRANGE: C{long}
+ at var MAXLOCKRANGE: the maximum range to lock in a file (windows only)
+ at type _pool: L{DebTorrent.piecebuffer.BufferPool}
+ at var _pool: the buffer for temporary storage of pieces
+ at type PieceBuffer: C{method} L{DebTorrent.piecebuffer.BufferPool.new}
+ at var PieceBuffer: returns a new or existing L{DebTorrent.piecebuffer.SingleBuffer}
+
+"""
from DebTorrent.piecebuffer import BufferPool
from threading import Lock
from time import time, strftime, localtime
import os
-from os.path import exists, getsize, getmtime, basename
+from os.path import exists, getsize, getmtime, basename, split
+from os import makedirs
from traceback import print_exc
try:
from os import fsync
@@ -32,12 +52,108 @@
PieceBuffer = _pool.new
def dummy_status(fractionDone = None, activity = None):
+ """Dummy function to print nothing."""
pass
class Storage:
+ """Low-level writing of files.
+
+ Control the low-level management of files in the download. Contains
+ functions to open and close, read and write, enable and disable,
+ flush and delete, all the files in the download. Also serves as an
+ abstraction layer, as the reading and writing is called with no
+ knowledge of file boundaries.
+
+ @type files: C{list} of C{tuple} of (C{string}, C{long})
+ @ivar files: the files list from the info of the metainfo dictionary
+ @type piece_lengths: C{list} of C{long}
+ @ivar piece_lengths: the list of piece lengths
+ @type doneflag: unknown
+ @ivar doneflag: unknown
+ @type disabled: C{list} of C{boolean}
+ @ivar disabled: list of true for the files that are disabled
+ @type file_ranges: C{list} of (C{long}, C{long}, C{long}, C{string})
+ @ivar file_ranges: for each file, the start offset within the download,
+ end offset, offset within the file, and file name
+ @type file_pieces: C{list} of (C{int}, C{int})
+ @ivar file_pieces: for each file, the starting and ending piece of the file
+ @type disabled_ranges: C{list} of C{tuple}
+ @ivar disabled_ranges: for each file, a tuple containing the working range,
+ shared pieces, and disabled range (see L{_get_disabled_ranges} for their
+ meaning)
+ @type working_ranges: C{list} of C{list} of (C{long}, C{long}, C{long}, C{string})
+ @ivar working_ranges: For each file, the list of files to be written when
+ writing to that file (may not be the actual file, i.e if it is
+ disabled). Ranges are temporarily stored here, before eventually being
+ written to self.ranges by L{_reset_ranges} to be used.
+ @type handles: C{dictionary} of {C{string}, C{file handle}}
+ @ivar handles: the file handles that are open, keys are file names and
+ values are the file handles
+ @type whandles: C{dictionary} of {C{string}, C{int}}
+ @ivar whandles: the files that are open for writing, keys are the file
+ names and values are all 1
+ @type tops: C{dictionary} of {C{string}, C{long}}
+ @ivar tops: the current length of each file (by name)
+ @type sizes: C{dictionary} of {C{string}, C{long}}
+ @ivar sizes: the desired length of each file (by name)
+ @type mtimes: C{dictionary} of {C{string}, C{long}}
+ @ivar mtimes: the last modified time of each file (by name)
+ @type lock_file: C{method}
+ @ivar lock_file: locks a file (if file locking is enabled, otherwise does
+ nothing)
+ @type unlock_file: C{method}
+ @ivar unlock_file: unlocks a file (if file locking is enabled, otherwise
+ does nothing)
+ @type lock_while_reading: C{boolean}
+ @ivar lock_while_reading: whether to lock files while reading them
+ @type lock: C{lock}
+ @ivar lock: a threading lock object for synchorizing threads (semaphore)
+ @type total_length: C{long}
+ @ivar total_length: the total length in bytes of the download
+ @type max_files_open: C{int}
+ @ivar max_files_open: the maximum number of files to have open at a time
+ (0 means no maximum)
+ @type handlebuffer: C{list}
+ @ivar handlebuffer: the list of open files, in the order of most recently
+ accessed, with the most recently accessed at the end of the list
+ (only if there is a limit on the number of open files, otherwise None)
+ @type ranges: C{list} of C{list} of (C{long}, C{long}, C{long}, C{string})
+ @ivar ranges: for each file, the list of files to be written when
+ writing to that file (may not be the actual file, i.e if it is
+ disabled)
+ @type begins: C{list} of C{long}
+ @ivar begins: the offset within the download that each non-disabled file
+ begins at
+ @type bufferdir: C{string}
+ @ivar bufferdir: the buffer directory
+ @type reset_file_status: C{method}
+ @ivar reset_file_status: a shortcut to the _reset_ranges method
+
+ """
+
def __init__(self, files, piece_lengths, doneflag, config,
disabled_files = None):
- # can raise IOError and ValueError
+ """Initializes the Storage.
+
+ Initializes some variables, and calculates defaults for others,
+ such as which pieces are contained by which file.
+
+ @type files: C{list} of C{tuple} of (C{string}, C{long})
+ @param files: the files list from the info of the metainfo dictionary
+ @type piece_lengths: C{list} of C{long}
+ @param piece_lengths: the list of piece lengths
+ @type doneflag: unknown
+ @param doneflag: unknown
+ @type config: C{dictionary}
+ @param config: the configuration information
+ @type disabled_files: C{list} of C{boolean}
+ @param disabled_files: list of true for the files that are disabled
+ (optional, default is no files disabled)
+ @raise IOError: unknown
+ @raise ValueError: unknown
+
+ """
+
self.files = files
self.piece_lengths = piece_lengths
self.doneflag = doneflag
@@ -107,6 +223,7 @@
l = length
else:
l = 0
+ self.make_directories(file)
h = open(file, 'wb+')
h.flush()
h.close()
@@ -127,6 +244,15 @@
if os.name == 'nt':
def _lock_file(self, name, f):
+ """Lock a file on Windows.
+
+ @type name: C{string}
+ @param name: The file name to lock (only used to get the file size)
+ @type f: C{file}
+ @param f: a file handle for the file to lock
+
+ """
+
import msvcrt
for p in range(0, min(self.sizes[name],MAXLOCKRANGE), MAXLOCKSIZE):
f.seek(p)
@@ -134,6 +260,15 @@
min(MAXLOCKSIZE,self.sizes[name]-p))
def _unlock_file(self, name, f):
+ """Unlock a file on Windows.
+
+ @type name: C{string}
+ @param name: The file name to unlock (only used to get the file size)
+ @type f: C{file}
+ @param f: a file handle for the file to unlock
+
+ """
+
import msvcrt
for p in range(0, min(self.sizes[name],MAXLOCKRANGE), MAXLOCKSIZE):
f.seek(p)
@@ -142,21 +277,53 @@
elif os.name == 'posix':
def _lock_file(self, name, f):
+ """Lock a file on Linux.
+
+ @type name: C{string}
+ @param name: The file name to lock (only used to get the file size)
+ @type f: C{file}
+ @param f: a file handle for the file to lock
+
+ """
+
import fcntl
fcntl.flock(f.fileno(), fcntl.LOCK_EX)
def _unlock_file(self, name, f):
+ """Unlock a file on Linux.
+
+ @type name: C{string}
+ @param name: The file name to unlock (only used to get the file size)
+ @type f: C{file}
+ @param f: a file handle for the file to unlock
+
+ """
+
import fcntl
fcntl.flock(f.fileno(), fcntl.LOCK_UN)
else:
def _lock_file(self, name, f):
+ """Dummy function to not Lock a file on other systems."""
pass
+
def _unlock_file(self, name, f):
+ """Dummy function to not unlock a file on other systems."""
pass
def was_preallocated(self, pos, length):
+ """Check if a download location was pre-allocated.
+
+ @type pos: C{long}
+ @param pos: the location to start checking
+ @type length: C{long}
+ @param length: the amount of the download to check
+ @rtype: C{boolean}
+ @return: whether the download location was pre-allocated
+
+ """
+
for file, begin, end in self._intervals(pos, length):
if self.tops.get(file, 0) < end:
return False
@@ -164,17 +331,44 @@
def _sync(self, file):
+ """Synchronize a file to disk.
+
+ Closes the open file so that it is synchronize to disk, the next time
+ it is referenced it will be reopened.
+
+ @type file: C{string}
+ @param file: the name of the file to sync
+
+ """
+
self._close(file)
if self.handlebuffer:
self.handlebuffer.remove(file)
def sync(self):
- # may raise IOError or OSError
+ """Synchronize all read/write files to disk.
+
+ @raise IOError: unknown
+ @raise OSError: unknown
+
+ """
+
for file in self.whandles.keys():
self._sync(file)
def set_readonly(self, f=None):
+ """Set a file (or all files) to be read-only.
+
+ Synchronizes the file (or all read/write files if none is given) to
+ disk. The next time the file is accessed it will be reopened.
+
+ @type f: C{int}
+ @param f: the number of the file to set read-only
+ (optional, default is to set all read/write files read-only)
+
+ """
+
if f is None:
self.sync()
return
@@ -184,10 +378,30 @@
def get_total_length(self):
+ """Get the total length of the download.
+
+ @rtype: C{long}
+ @return: the total length of the download
+
+ """
+
return self.total_length
def _open(self, file, mode):
+ """Open a file with the given mode.
+
+ @type file: C{string}
+ @param file: the file name to open
+ @type mode: C{string}
+ @param mode: the mode to open the file in
+ (r = read, w = write, a = append, add b for binary)
+ @rtype: C{file handle}
+ @return: the file handle to access the file with
+ @raise IOError: if the file has been modified since it was last read
+
+ """
+
if self.mtimes.has_key(file):
try:
if self.handlebuffer is not None:
@@ -203,6 +417,7 @@
+strftime(' != (%x %X) ?',localtime(getmtime(file))) )
raise IOError('modified during download')
try:
+ self.make_directories(file)
return open(file, mode)
except:
if DEBUG:
@@ -211,6 +426,13 @@
def _close(self, file):
+ """Close the file
+
+ @type file: C{string}
+ @param file: the name of the file to close
+
+ """
+
f = self.handles[file]
del self.handles[file]
if self.whandles.has_key(file):
@@ -227,6 +449,13 @@
def _close_file(self, file):
+ """Close the file and release the handle.
+
+ @type file: C{string}
+ @param file: the name of the file to close
+
+ """
+
if not self.handles.has_key(file):
return
self._close(file)
@@ -235,6 +464,18 @@
def _get_file_handle(self, file, for_write):
+ """Get a new or existing flie handle for the file.
+
+ @type file: C{string}
+ @param file: the name of the file to get a handle for
+ @type for_write: C{boolean}
+ @param for_write: whether to open the file for writing
+ @rtype: C{file handle}
+ @return: the file handle that can be used for the file
+ @raise IOError: if the file can not be opened
+
+ """
+
if self.handles.has_key(file):
if for_write and not self.whandles.has_key(file):
self._close(file)
@@ -280,12 +521,32 @@
def _reset_ranges(self):
+ """Re-initialize the ranges from the working copies."""
self.ranges = []
for l in self.working_ranges:
self.ranges.extend(l)
- self.begins = [i[0] for i in self.ranges]
+ self.begins = [i[0] for i in self.ranges]
+ if DEBUG:
+ print str(self.ranges)
+ print str(self.begins)
def _intervals(self, pos, amount):
+ """Get the files that are within the range.
+
+ Finds all the files that occur within a given range in the download,
+ and return a list of them. Includes the range of the file that is
+ inside the range, which will be the start (0) and end (length) of the
+ file unless it goes past the beginning or end of the range.
+
+ @type pos: C{long}
+ @param pos: the start of the range within the download
+ @type amount: C{long}
+ @param amount: the length of the range
+ @rtype: C{list} of C{tuple} of (C{string}, C{long}, C{long})
+ @return: the list of files and their start and end offsets in the range
+
+ """
+
r = []
stop = pos + amount
p = bisect(self.begins, pos) - 1
@@ -293,14 +554,29 @@
begin, end, offset, file = self.ranges[p]
if begin >= stop:
break
- r.append(( file,
- offset + max(pos, begin) - begin,
- offset + min(end, stop) - begin ))
+ if end > pos:
+ r.append(( file,
+ offset + max(pos, begin) - begin,
+ offset + min(end, stop) - begin ))
p += 1
return r
def read(self, pos, amount, flush_first = False):
+ """Read data from the download.
+
+ @type pos: C{long}
+ @param pos: the offset in the download to start reading from
+ @type amount: C{long}
+ @param amount: the length of the data to read
+ @type flush_first: C{boolean}
+ @param flush_first: whether to flush the files before reading the data
+ (optional, default is not to flush)
+ @rtype: L{DebTorrent.piecebuffer.SingleBuffer}
+ @return: the data that was read
+
+ """
+
r = PieceBuffer()
for file, pos, end in self._intervals(pos, amount):
if DEBUG:
@@ -322,6 +598,15 @@
return r
def write(self, pos, s):
+ """Write data to the download.
+
+ @type pos: C{long}
+ @param pos: the offset in the download to start writing at
+ @type s: unknown
+ @param s: data to write
+
+ """
+
# might raise an IOError
total = 0
for file, begin, end in self._intervals(pos, len(s)):
@@ -334,7 +619,20 @@
self.lock.release()
total += end - begin
+ def make_directories(self, file):
+ """Create missing parent directories for a file.
+
+ @type file: C{string}
+ @param file: the file name to create directories for
+
+ """
+
+ file = split(file)[0]
+ if file != '' and not exists(file):
+ makedirs(file)
+
def top_off(self):
+ """Extend all files to their appropriate length."""
for begin, end, offset, file in self.ranges:
l = offset + end - begin
if l > self.tops.get(file, 0):
@@ -345,6 +643,7 @@
self.lock.release()
def flush(self):
+ """Flush all files to disk."""
# may raise IOError or OSError
for file in self.whandles.keys():
self.lock.acquire()
@@ -352,6 +651,7 @@
self.lock.release()
def close(self):
+ """Close all open files."""
for file, f in self.handles.items():
try:
self.unlock_file(file, f)
@@ -367,6 +667,28 @@
def _get_disabled_ranges(self, f):
+ """Calculate the file ranges for the disabled file.
+
+ Calculates, based on the file lengths and piece lengths, the ranges
+ to write for the file. There are three lists calculated.
+
+ The working range is the list of files and file offsets to write if
+ the file is enabled.
+
+ The shared pieces is a list of piece numbers that the file shares
+ with other files.
+
+ The disabled range is the list of files and file offsets to write if
+ the file is disabled.
+
+ @type f: C{int}
+ @param f: the index of the file
+ @rtype: C{tuple}
+ @return: a tuple containing the working range, shared pieces, and
+ disabled range
+
+ """
+
if not self.file_ranges[f]:
return ((),(),())
r = self.disabled_ranges[f]
@@ -394,9 +716,23 @@
def set_bufferdir(self, dir):
+ """Sets the buffer directory.
+
+ @type dir: C{string}
+ @param dir: the new buffer directory
+
+ """
+
self.bufferdir = dir
def enable_file(self, f):
+ """Enable a file for writing.
+
+ @type f: C{int}
+ @param f: the index of the file to enable
+
+ """
+
if not self.disabled[f]:
return
self.disabled[f] = False
@@ -405,6 +741,7 @@
return
file = r[3]
if not exists(file):
+ self.make_directories(file)
h = open(file, 'wb+')
h.flush()
h.close()
@@ -415,13 +752,19 @@
self.working_ranges[f] = [r]
def disable_file(self, f):
+ """Disable a file from writing.
+
+ @type f: C{int}
+ @param f: the index of the file to disable
+
+ """
if self.disabled[f]:
return
self.disabled[f] = True
r = self._get_disabled_ranges(f)
if not r:
return
- for file, begin, end in r[2]:
+ for begin, end, offset, file in r[2]:
if not os.path.isdir(self.bufferdir):
os.makedirs(self.bufferdir)
if not exists(file):
@@ -432,48 +775,69 @@
self.tops[file] = getsize(file)
if not self.mtimes.has_key(file):
self.mtimes[file] = getmtime(file)
- self.working_ranges[f] = r[0]
+ self.working_ranges[f] = r[2]
reset_file_status = _reset_ranges
def get_piece_update_list(self, f):
+ """Get the list of pieces the file shares with other files.
+
+ @type f: C{int}
+ @param f: the index of the file to disable
+ @rtype: C{list} of C{int}
+ @return: the list of piece indexes
+
+ """
+
return self._get_disabled_ranges(f)[1]
def delete_file(self, f):
+ """Delete the file.
+
+ @type f: C{int}
+ @param f: the index of the file to delete
+
+ """
try:
os.remove(self.files[f][0])
except:
pass
- '''
- Pickled data format:
-
- d['files'] = [ file #, size, mtime {, file #, size, mtime...} ]
- file # in torrent, and the size and last modification
- time for those files. Missing files are either empty
- or disabled.
- d['partial files'] = [ name, size, mtime... ]
- Names, sizes and last modification times of files containing
- partial piece data. Filenames go by the following convention:
- {file #, 0-based}{nothing, "b" or "e"}
- eg: "0e" "3" "4b" "4e"
- Where "b" specifies the partial data for the first piece in
- the file, "e" the last piece, and no letter signifying that
- the file is disabled but is smaller than one piece, and that
- all the data is cached inside so adjacent files may be
- verified.
- '''
def pickle(self):
+ """Create a dictionary representing the current state of the download.
+
+ Pickled data format::
+
+ d['files'] = [ file #, size, mtime {, file #, size, mtime...} ]
+ file # in torrent, and the size and last modification
+ time for those files. Missing files are either empty
+ or disabled.
+ d['partial files'] = [ name, size, mtime... ]
+ Names, sizes and last modification times of files containing
+ partial piece data. Filenames go by the following convention:
+ {file #, 0-based}{nothing, "b" or "e"}
+ eg: "0e" "3" "4b" "4e"
+ Where "b" specifies the partial data for the first piece in
+ the file, "e" the last piece, and no letter signifying that
+ the file is disabled but is smaller than one piece, and that
+ all the data is cached inside so adjacent files may be
+ verified.
+
+ @rtype: C{dictionary}
+ @return: the pickled current status of the download
+
+ """
+
files = []
pfiles = []
for i in xrange(len(self.files)):
if not self.files[i][1]: # length == 0
continue
if self.disabled[i]:
- for file, start, end in self._get_disabled_ranges(i)[2]:
+ for start, end, offset, file in self._get_disabled_ranges(i)[2]:
pfiles.extend([basename(file),getsize(file),int(getmtime(file))])
continue
file = self.files[i][0]
@@ -482,7 +846,17 @@
def unpickle(self, data):
- # assume all previously-disabled files have already been disabled
+ """Extract the current status of the download from a pickled dictionary.
+
+ Assumes all previously-disabled files have already been disabled.
+
+ @type data: C{dictionary}
+ @param data: the pickled current status of the download
+ @rtype: C{list} of C{int}
+ @return: a list of the currently enabled pieces
+
+ """
+
try:
files = {}
pfiles = {}
@@ -515,6 +889,19 @@
print valid_pieces.keys()
def test(old, size, mtime):
+ """Test that the file has not changed since the status save.
+
+ @type old: C{tuple} of (C{long}, C{long})
+ @param old: the previous size and modification time of the file
+ @type size: C{long}
+ @param size: the current size of the file
+ @type mtime: C{long}
+ @param mtime: the current modification time of the file
+ @rtype: C{boolean}
+ @return: whether the file has been changed
+
+ """
+
oldsize, oldmtime = old
if size != oldsize:
return False
Modified: debtorrent/branches/http-listen/DebTorrent/BT1/__init__.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/BT1/__init__.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/BT1/__init__.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/BT1/__init__.py Tue May 29 05:58:07 2007
@@ -1,6 +1,11 @@
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
-# placeholder
+"""More specific sub-modules for the debtorrent protocol.
+
+This package contains ome more specific sub-modules used by the
+L{DebTorrent} package.
+
+"""
Modified: debtorrent/branches/http-listen/DebTorrent/BT1/btformats.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/BT1/btformats.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/BT1/btformats.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/BT1/btformats.py Tue May 29 05:58:07 2007
@@ -1,17 +1,41 @@
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Functions for verifying debtorrent metainfo.
+
+These functions all work on bdecoded debtorrent metainfo, and are used
+to verify their conformance with the protocol.
+
+ at type reg: C{regex}
+ at var reg: a compiled regex for verifying the security of path names
+ at type ints: C{tuple} of C{types}
+ at var ints: the types that are acceptable for integer values
+
+"""
from types import StringType, LongType, IntType, ListType, DictType
from re import compile
+from bisect import bisect
reg = compile(r'^[^/\\.~][^/\\]*$')
ints = (LongType, IntType)
def check_info(info):
+ """Checks the info dictionary for conformance.
+
+ Verifies that the info dictionary of the metainfo conforms to the
+ debtorrent protocol.
+
+ @type info: C{dictionary}
+ @param info: the info field from the metainfo dictionary
+ @raise ValueError: if the info doesn't conform
+
+ """
+
if type(info) != DictType:
raise ValueError, 'bad metainfo - not a dictionary'
pieces = info.get('pieces')
@@ -20,20 +44,28 @@
piecelengths = info.get('piece lengths')
if type(piecelengths) != ListType:
raise ValueError
+ total_length = 0L
+ piece_bounds = [0L]
for length in piecelengths:
if type(length) not in ints or length <= 0:
raise ValueError, 'bad metainfo - bad piece length'
+ total_length += length
+ piece_bounds.append(total_length)
if info.has_key('files') == info.has_key('length'):
raise ValueError, 'single/multiple file mix'
files = info.get('files')
if type(files) != ListType:
raise ValueError
+ total_length = 0L
for f in files:
if type(f) != DictType:
raise ValueError, 'bad metainfo - bad file value'
length = f.get('length')
if type(length) not in ints or length < 0:
raise ValueError, 'bad metainfo - bad length'
+ total_length += length
+ if piece_bounds[bisect(piece_bounds,total_length)-1] != total_length:
+ raise ValueError, 'bad metainfo - file does not end on piece boundary'
path = f.get('path')
if type(path) != ListType or path == []:
raise ValueError, 'bad metainfo - bad path'
@@ -49,6 +81,17 @@
# raise ValueError, 'bad metainfo - duplicate path'
def check_message(message):
+ """Checks the metainfo dictionary for conformance.
+
+ Verifies that the metainfo dictionary conforms to the
+ debtorrent protocol.
+
+ @type message: C{dictionary}
+ @param message: the bdecoded metainfo dictionary
+ @raise ValueError: if the metainfo doesn't conform
+
+ """
+
if type(message) != DictType:
raise ValueError
check_info(message.get('info'))
@@ -61,6 +104,17 @@
raise ValueError, 'name %s disallowed for security reasons' % name
def check_peers(message):
+ """Checks the peers dictionary returned by a tracker for conformance.
+
+ Verifies that the peers dictionary returned by a tracker conforms to the
+ debtorrent protocol.
+
+ @type message: C{dictionary}
+ @param message: the bdecoded peers dictionary returned by a tracker
+ @raise ValueError: if the info doesn't conform
+
+ """
+
if type(message) != DictType:
raise ValueError
if message.has_key('failure reason'):
Modified: debtorrent/branches/http-listen/DebTorrent/BTcrypto.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/BTcrypto.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/BTcrypto.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/BTcrypto.py Tue May 29 05:58:07 2007
@@ -2,8 +2,21 @@
# based on code by Uoti Urpala
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+
+"""Encrypted communication support.
+
+ at type KEY_LENGTH: C{int}
+ at var KEY_LENGTH: the length of the keys to generate
+ at type DH_PRIME: C{long}
+ at var DH_PRIME: a very large prime number
+ at type PAD_MAX: unknown
+ at var PAD_MAX: unknown
+ at type DH_BYTES: C{int}
+ at var DH_BYTES: the number of bytes to use for key lengths
+
+"""
from __future__ import generators # for python 2.2
from random import randrange,randint,seed
@@ -32,15 +45,77 @@
DH_BYTES = 96
def bytetonum(x):
+ """Convert a long number in a string to a number.
+
+ @type x: C{string}
+ @param x: the data to convert
+ @rtype: C{long}
+ @return: the converted data
+
+ """
+
return long(x.encode('hex'), 16)
def numtobyte(x):
+ """Convert a very large number to a string.
+
+ @type x: C{long}
+ @param x: the number to convert
+ @rtype: C{string}
+ @return: the string representation of the number
+
+ """
+
x = hex(x).lstrip('0x').rstrip('Ll')
x = '0'*(192 - len(x)) + x
return x.decode('hex')
class Crypto:
+ """
+
+ @type initiator: C{boolean}
+ @ivar initiator: whether the connection was initiated locally
+ @type disable_crypto: C{boolean}
+ @ivar disable_crypto: whether crypto has been disabled
+ @type privkey: C{long}
+ @ivar privkey: randomly generated private key
+ @type pubkey: C{string}
+ @ivar pubkey: public key corresponding to the private key
+ @type keylength: C{int}
+ @ivar keylength: the key length to use
+ @type _VC_pattern: unknown
+ @ivar _VC_pattern: unknown
+ @type S: unknown
+ @ivar S: unknown
+ @type block3a: unknown
+ @ivar block3a: unknown
+ @type block3bkey: unknown
+ @ivar block3bkey: unknown
+ @type block3b: unknown
+ @ivar block3b: unknown
+ @type encrypt: C{method}
+ @ivar encrypt: the method to call to encrypt data
+ @type decrypt: C{method}
+ @ivar decrypt: the method to call to decrypt data
+ @type _read: C{method}
+ @ivar _read: the method to call to read decrypted data
+ @type _write: C{method}
+ @ivar _write: the method to call to write encrypted data
+
+ """
+
def __init__(self, initiator, disable_crypto = False):
+ """Initialize the instance.
+
+ @type initiator: C{boolean}
+ @param initiator: whether the connection was initiated locally
+ @type disable_crypto: C{boolean}
+ @param disable_crypto: whether crypto has been disabled
+ (optional, default is False)
+ @raise NotImplementedError: if encryption is not installed
+
+ """
+
self.initiator = initiator
self.disable_crypto = disable_crypto
if not disable_crypto and not CRYPTO_OK:
@@ -51,17 +126,42 @@
self._VC_pattern = None
def received_key(self, k):
+ """Process a received key.
+
+ @type k: C{string}
+ @param k: the key that was received
+
+ """
+
self.S = numtobyte(pow(bytetonum(k), self.privkey, DH_PRIME))
self.block3a = sha('req1'+self.S).digest()
self.block3bkey = sha('req3'+self.S).digest()
self.block3b = None
def _gen_block3b(self, SKEY):
+ """
+
+ @type SKEY: C{string}
+ @param SKEY: unknown
+
+ """
+
a = sha('req2'+SKEY).digest()
return ''.join([ chr(ord(a[i])^ord(self.block3bkey[i]))
for i in xrange(20) ])
def test_skey(self, s, SKEY):
+ """Check that the encoding matches the encoded value.
+
+ @type s: C{string}
+ @param s: unknown
+ @type SKEY: C{string}
+ @param SKEY: unknown
+ @rtype: C{boolean}
+ @return: whether the encoding of SKEY matches s
+
+ """
+
block3b = self._gen_block3b(SKEY)
if block3b != s:
return False
@@ -71,6 +171,13 @@
return True
def set_skey(self, SKEY):
+ """
+
+ @type SKEY: C{string}
+ @param SKEY: unknown
+
+ """
+
if not self.block3b:
self.block3b = self._gen_block3b(SKEY)
crypta = ARC4.new(sha('keyA'+self.S+SKEY).digest())
@@ -85,22 +192,60 @@
self.decrypt('x'*1024)
def VC_pattern(self):
+ """Unknown.
+
+ @rtype: unknown
+ @return: unknown
+
+ """
if not self._VC_pattern:
self._VC_pattern = self.decrypt('\x00'*8)
return self._VC_pattern
def read(self, s):
+ """Decrypt and pass on the decrypted value.
+
+ @type s: C{string}
+ @param s: the string to decrypt
+
+ """
+
self._read(self.decrypt(s))
def write(self, s):
+ """Encrypt and pass on the encrypted value.
+
+ @type s: C{string}
+ @param s: the string to encrypt
+
+ """
+
self._write(self.encrypt(s))
def setrawaccess(self, _read, _write):
+ """Setup the methods to call for reading and writing.
+
+ @type _read: C{method}
+ @param _read: the method to call for reading decrypted data
+ @type _write: C{method}
+ @param _write: the method to call for writing encrypted data
+
+ """
+
self._read = _read
self._write = _write
def padding(self):
+ """Generate a random amount of random padding.
+
+ Generates random bytes, with a random length from 16 to L{PAD_MAX}.
+
+ @rtype: C{string}
+ @return: the randomly generated padding
+
+ """
+
return urandom(randrange(PAD_MAX-16)+16)
Modified: debtorrent/branches/http-listen/DebTorrent/ConfigDir.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/ConfigDir.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/ConfigDir.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/ConfigDir.py Tue May 29 05:58:07 2007
@@ -1,8 +1,21 @@
#written by John Hoffman
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+
+"""Manage configuration and cache files.
+
+ at type DIRNAME: C{string}
+ at var DIRNAME: the directory name to use for storing config files
+ at type hexchars: C{string}
+ at var hexchars: the 16 hex characters, in order
+ at type hexmap: C{list}
+ at var hexmap: a mapping from characters to the hex repesentation of the character
+ at type revmap: C{dictionary}
+ at var revmap: the reverse of L{hexmap}
+
+"""
from inifile import ini_write, ini_read
from bencode import bencode, bdecode
@@ -29,16 +42,45 @@
revmap[x] = chr(i)
def tohex(s):
+ """Convert a string to hex representation.
+
+ @type s: C{string}
+ @param s: the string to convert
+ @rtype: C{string}
+ @return: the converted string
+
+ """
+
r = []
for c in s:
r.append(hexmap[ord(c)])
return ''.join(r)
def unhex(s):
+ """Convert a hex representation back to a string.
+
+ @type s: C{string}
+ @param s: the hex representation of a string
+ @rtype: C{string}
+ @return: the original string
+
+ """
+
r = [ revmap[s[x:x+2]] for x in xrange(0, len(s), 2) ]
return ''.join(r)
-def copyfile(oldpath, newpath): # simple file copy, all in RAM
+def copyfile(oldpath, newpath):
+ """Simple file copy, all in RAM.
+
+ @type oldpath: C{string}
+ @param oldpath: the file name to copy from
+ @type newpath: C{string}
+ @param newpath: the file name to copy to
+ @rtype: C{boolean}
+ @return: whether the copy was successful
+
+ """
+
try:
f = open(oldpath,'rb')
r = f.read()
@@ -64,10 +106,46 @@
class ConfigDir:
+ """Manage configuration and cache files.
+
+ @type config_type: C{string}
+ @ivar config_type: the extension to include in the saved files' names
+ @type dir_root: C{string}
+ @ivar dir_root: the root directory to save config files in
+ @type dir_torrentcache: C{string}
+ @ivar dir_torrentcache: the directory to save torrent files in
+ @type dir_datacache: C{string}
+ @ivar dir_datacache: the directory to save stopped torrent's state in
+ @type dir_piececache: C{string}
+ @ivar dir_piececache: the directory to store temporary piece files in
+ @type configfile: C{string}
+ @ivar configfile: the file name for the saved configuration data
+ @type statefile: C{string}
+ @ivar statefile: the file name for the saved state
+ @type TorrentDataBuffer: C{dictionary}
+ @ivar TorrentDataBuffer: unknown
+ @type config: C{dictionary}
+ @ivar config: the current configuration variables
+
+ @group Config Handling: setDefaults, checkConfig, loadConfig, saveConfig, getConfig
+ @group State: getState, saveState
+ @group Torrent Files: getTorrents, getTorrentVariations, getTorrent, writeTorrent
+ @group Torrent Data: getTorrentData, writeTorrentData, deleteTorrentData, getPieceDir
+ @group Expire Cache: deleteOldCacheData, deleteOldTorrents
+
+ """
###### INITIALIZATION TASKS ######
def __init__(self, config_type = None):
+ """Initialize the instance, create directories and file names.
+
+ @type config_type: C{string}
+ @param config_type: the extension to include in the saved files' names
+ (optional, default is to use no extension)
+
+ """
+
self.config_type = config_type
if config_type:
config_ext = '.'+config_type
@@ -75,6 +153,15 @@
config_ext = ''
def check_sysvars(x):
+ """Check a system variable to see if it expands to something.
+
+ @type x: C{string}
+ @param x: the system variable to check
+ @rtype: C{string}
+ @return: the expanded variable, or None if it doesn't expand
+
+ """
+
y = os.path.expandvars(x)
if y != x and os.path.isdir(y):
return y
@@ -116,15 +203,39 @@
###### CONFIG HANDLING ######
def setDefaults(self, defaults, ignore=[]):
+ """Set the default values to use for the configuration.
+
+ @type defaults: C{dictionary}
+ @param defaults: the default config values
+ @type ignore: C{list}
+ @param ignore: the keys in the defaults to ignore
+ (optional, default is to ignore none of them)
+
+ """
+
self.config = defaultargs(defaults)
for k in ignore:
if self.config.has_key(k):
del self.config[k]
def checkConfig(self):
+ """Check if a config file already exists.
+
+ @rtype: C{boolean}
+ @return: whether the config file exists
+
+ """
+
return os.path.exists(self.configfile)
def loadConfig(self):
+ """Load a configuration from a config file.
+
+ @rtype: C{dictionary}
+ @return: the loaded configuration variables
+
+ """
+
try:
r = ini_read(self.configfile)['']
except:
@@ -148,6 +259,16 @@
return self.config
def saveConfig(self, new_config = None):
+ """Sets and writes to the file the new configuration.
+
+ @type new_config: C{dictionary}
+ @param new_config: the configuration to set and write
+ (optional, default is to use the previously set one)
+ @rtype: boolean
+ @return: whether writing to the file was successful
+
+ """
+
if new_config:
for k,v in new_config.items():
if self.config.has_key(k):
@@ -161,12 +282,27 @@
return False
def getConfig(self):
+ """Get the current configuration variables.
+
+ @rtype: C{dictionary}
+ @return: the current configuration variables
+
+ """
+
return self.config
###### STATE HANDLING ######
def getState(self):
+ """Get the state from the state file.
+
+ @rtype: unknown
+ @return: the previosuly saved state, or None if there was no previously
+ saved state
+
+ """
+
try:
f = open(self.statefile,'rb')
r = f.read()
@@ -183,6 +319,15 @@
return r
def saveState(self, state):
+ """Saves the state to the state file.
+
+ @type state: unknown
+ @param state: the state to save
+ @rtype: boolean
+ @return: whether the saving was successful
+
+ """
+
try:
f = open(self.statefile,'wb')
f.write(bencode(state))
@@ -199,6 +344,13 @@
###### TORRENT HANDLING ######
def getTorrents(self):
+ """Get a list of the torrents that have cache data.
+
+ @rtype: C{list} of C{string}
+ @return: the torrent hashes found
+
+ """
+
d = {}
for f in os.listdir(self.dir_torrentcache):
f = os.path.basename(f)
@@ -210,6 +362,15 @@
return d.keys()
def getTorrentVariations(self, t):
+ """Get the torrent variations in the cache data for a given hash.
+
+ @type t: C{string}
+ @param t: the torrent hash to check for
+ @rtype: C{list} of C{int}
+ @return: the variations of the hash found
+
+ """
+
t = tohex(t)
d = []
for f in os.listdir(self.dir_torrentcache):
@@ -224,6 +385,17 @@
return d
def getTorrent(self, t, v = -1):
+ """Get the torrent data for the hash.
+
+ @type t: C{string}
+ @param t: the torrent hash to lookup
+ @type v: C{int}
+ @param v: the variation to get (optional, default is the largest)
+ @rtype: C{dictionary}
+ @return: the torrent metainfo found
+
+ """
+
t = tohex(t)
if v == -1:
v = max(self.getTorrentVariations(t)) # potential exception
@@ -241,6 +413,20 @@
return r
def writeTorrent(self, data, t, v = -1):
+ """Save the torrent data.
+
+ @type data: C{dictionary}
+ @param data: the torrent metainfo
+ @type t: C{string}
+ @param t: the hash of the torrent metainfo
+ @type v: C{int}
+ @param v: the variation to save as, or None for no variation
+ (optional, default is the next after the largest)
+ @rtype: C{int}
+ @return: the variation used, or None if the write failed
+
+ """
+
t = tohex(t)
if v == -1:
try:
@@ -264,6 +450,15 @@
###### TORRENT DATA HANDLING ######
def getTorrentData(self, t):
+ """Retrieve cached data for a torrent.
+
+ @type t: C{string}
+ @param t: the info hash to retrieve cached data for
+ @rtype: C{dictionary}
+ @return: the bdecoded cached data
+
+ """
+
if self.TorrentDataBuffer.has_key(t):
return self.TorrentDataBuffer[t]
t = os.path.join(self.dir_datacache,tohex(t))
@@ -282,6 +477,17 @@
return r
def writeTorrentData(self, t, data):
+ """Write cached data for a torrent.
+
+ @type t: C{string}
+ @param t: the info hash to write cached data for
+ @type data: C{dictionary}
+ @param data: the data to cache
+ @rtype: C{boolean}
+ @return: whether the write was successful
+
+ """
+
self.TorrentDataBuffer[t] = data
try:
f = open(os.path.join(self.dir_datacache,tohex(t)),'wb')
@@ -298,18 +504,46 @@
return success
def deleteTorrentData(self, t):
+ """Delete the cached data for a torrent.
+
+ @type t: C{string}
+ @param t: the info hash to delete the cached data of
+
+ """
+
try:
os.remove(os.path.join(self.dir_datacache,tohex(t)))
except:
pass
def getPieceDir(self, t):
+ """Get the directory to save temporary pieces for a torrent.
+
+ @type t: C{string}
+ @param t: the info hash to get the piece cache data of
+ @rtype: C{string}
+ @return: the directory to save temporary pieces in
+
+ """
+
return os.path.join(self.dir_piececache,tohex(t))
###### EXPIRATION HANDLING ######
def deleteOldCacheData(self, days, still_active = [], delete_torrents = False):
+ """Delete old cache data after a period of time.
+
+ @type days: C{int}
+ @param days: the number of days to delete cached data after
+ @type still_active: C{list} of C{string}
+ @param still_active: the hashes of torrents that are still running
+ (optional, default is to delete all torrent's cached data)
+ @type delete_torrents: C{boolean}
+ @param delete_torrents: whether to delete the torrent files as well
+
+ """
+
if not days:
return
exptime = time() - (days*24*3600)
@@ -380,4 +614,14 @@
def deleteOldTorrents(self, days, still_active = []):
+ """Delete old cached data and torrents after a period of time.
+
+ @type days: C{int}
+ @param days: the number of days to delete cached data after
+ @type still_active: C{list} of C{string}
+ @param still_active: the hashes of torrents that are still running
+ (optional, default is to delete all torrent's cached data)
+
+ """
+
self.deleteOldCacheData(days, still_active, True)
Modified: debtorrent/branches/http-listen/DebTorrent/ConnChoice.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/ConnChoice.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/ConnChoice.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/ConnChoice.py Tue May 29 05:58:07 2007
@@ -1,7 +1,17 @@
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Sets the connection choices that are available.
+
+ at type connChoices: C{list} of C{dictionary}
+ at var connChoiceList: Details for each type of connection. Includes limits
+ for each type on the upload rate and number of connections.
+ at type connChoiceList: C{list} of C{string}
+ at var connChoiceList: the names of the connections that are available
+
+"""
connChoices=(
{'name':'automatic',
Modified: debtorrent/branches/http-listen/DebTorrent/CurrentRateMeasure.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/CurrentRateMeasure.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/CurrentRateMeasure.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/CurrentRateMeasure.py Tue May 29 05:58:07 2007
@@ -1,13 +1,42 @@
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Measuring rates of download and upload."""
from clock import clock
class Measure:
+ """The measurement of one rate.
+
+ @type max_rate_period: C{float}
+ @ivar max_rate_period: maximum amount of time to guess the current rate
+ estimate represents
+ @type ratesince: C{float}
+ @ivar ratesince: the oldest time the rate estimate is for
+ @type last: C{float}
+ @ivar last: the last time the rate was updated
+ @type rate: C{float}
+ @ivar rate: the latest calculated rate
+ @type total: C{long}
+ @ivar total: the total amount that went in to calculating the rate
+
+ """
+
def __init__(self, max_rate_period, fudge = 1):
+ """Initialize the measurement.
+
+ @type max_rate_period: C{float}
+ @param max_rate_period: maximum amount of time to guess the current
+ rate estimate represents
+ @type fudge: C{int}
+ @param fudge: time equivalent of writing to kernel-level TCP buffer,
+ for rate adjustment (optional, defaults to 1)
+
+ """
+
self.max_rate_period = max_rate_period
self.ratesince = clock() - fudge
self.last = self.ratesince
@@ -15,6 +44,13 @@
self.total = 0l
def update_rate(self, amount):
+ """Update the rate with new data.
+
+ @type amount: C{long}
+ @param amount: the new data to add into the rate calculation
+
+ """
+
self.total += amount
t = clock()
self.rate = (self.rate * (self.last - self.ratesince) +
@@ -24,17 +60,48 @@
self.ratesince = t - self.max_rate_period
def get_rate(self):
+ """Get the current rate measurement.
+
+ @rtype: C{float}
+ @return: the current rate
+
+ """
+
self.update_rate(0)
return self.rate
def get_rate_noupdate(self):
+ """Get the current rate measurement without updating it.
+
+ @rtype: C{float}
+ @return: the current rate
+
+ """
+
return self.rate
def time_until_rate(self, newrate):
+ """Calculate how long until the rate drops to the target.
+
+ @type newrate: C{float}
+ @param newrate: the target rate
+ @rtype: C{float}
+ @return: the number of seconds until the rate decreases to the target
+ rate, or 0 if it's already there (or below it)
+
+ """
+
if self.rate <= newrate:
return 0
t = clock() - self.ratesince
return ((self.rate * t) / newrate) - t
def get_total(self):
+ """Get the total amount used to calculate the rate..
+
+ @rtype: C{float}
+ @return: the total amount
+
+ """
+
return self.total
Modified: debtorrent/branches/http-listen/DebTorrent/__init__.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/__init__.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/__init__.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/__init__.py Tue May 29 05:58:07 2007
@@ -1,10 +1,22 @@
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+"""The main package to implement the debtorrent protocol.
+
+This package, and it's subpackage L{BT1}, contains all the modules needed
+to implement the DebTorrent protocol.
+
+ at type product_name: C{string}
+ at var product_name: the name given for the package
+ at type version_short: C{string}
+ at var version_short: the short version number
+
+"""
+
product_name = 'DebTorrent'
-version_short = 'T-0.1.0'
+version_short = 'T-0.1.1'
version = version_short+' ('+product_name+')'
report_email = 'debtorrent-devel at lists.alioth.debian.org'
Modified: debtorrent/branches/http-listen/DebTorrent/bencode.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/bencode.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/bencode.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/bencode.py Tue May 29 05:58:07 2007
@@ -1,8 +1,24 @@
# Written by Petru Paler, Uoti Urpala, Ross Cohen and John Hoffman
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+
+"""Functions for bencoding and bdecoding data.
+
+ at type decode_func: C{dictionary} of C{function}
+ at var decode_func: a dictionary of function calls to be made, based on data,
+ the keys are the first character of the data and the value is the
+ function to use to decode that data
+ at type bencached_marker: C{list}
+ at var bencached_marker: mutable type to ensure class origination
+ at type encode_func: C{dictionary} of C{function}
+ at var encode_func: a dictionary of function calls to be made, based on data,
+ the keys are the type of the data and the value is the
+ function to use to encode that data
+ at type BencachedType: C{type}
+ at var BencachedType: the L{Bencached} type
+"""
from types import IntType, LongType, StringType, ListType, TupleType, DictType
try:
@@ -16,6 +32,18 @@
from cStringIO import StringIO
def decode_int(x, f):
+ """Bdecode an integer.
+
+ @type x: C{string}
+ @param x: the data to decode
+ @type f: C{int}
+ @param f: the offset in the data to start at
+ @rtype: C{int}, C{int}
+ @return: the bdecoded integer, and the offset to read next
+ @raise ValueError: if the data is improperly encoded
+
+ """
+
f += 1
newf = x.index('e', f)
try:
@@ -30,6 +58,18 @@
return (n, newf+1)
def decode_string(x, f):
+ """Bdecode a string.
+
+ @type x: C{string}
+ @param x: the data to decode
+ @type f: C{int}
+ @param f: the offset in the data to start at
+ @rtype: C{string}, C{int}
+ @return: the bdecoded string, and the offset to read next
+ @raise ValueError: if the data is improperly encoded
+
+ """
+
colon = x.index(':', f)
try:
n = int(x[f:colon])
@@ -41,10 +81,32 @@
return (x[colon:colon+n], colon+n)
def decode_unicode(x, f):
+ """Bdecode a unicode string.
+
+ @type x: C{string}
+ @param x: the data to decode
+ @type f: C{int}
+ @param f: the offset in the data to start at
+ @rtype: C{int}, C{int}
+ @return: the bdecoded unicode string, and the offset to read next
+
+ """
+
s, f = decode_string(x, f+1)
return (s.decode('UTF-8'),f)
def decode_list(x, f):
+ """Bdecode a list.
+
+ @type x: C{string}
+ @param x: the data to decode
+ @type f: C{int}
+ @param f: the offset in the data to start at
+ @rtype: C{list}, C{int}
+ @return: the bdecoded list, and the offset to read next
+
+ """
+
r, f = [], f+1
while x[f] != 'e':
v, f = decode_func[x[f]](x, f)
@@ -52,6 +114,18 @@
return (r, f + 1)
def decode_dict(x, f):
+ """Bdecode a dictionary.
+
+ @type x: C{string}
+ @param x: the data to decode
+ @type f: C{int}
+ @param f: the offset in the data to start at
+ @rtype: C{dictionary}, C{int}
+ @return: the bdecoded dictionary, and the offset to read next
+ @raise ValueError: if the data is improperly encoded
+
+ """
+
r, f = {}, f+1
lastkey = None
while x[f] != 'e':
@@ -79,6 +153,18 @@
#decode_func['u'] = decode_unicode
def bdecode(x, sloppy = 0):
+ """Bdecode a string of data.
+
+ @type x: C{string}
+ @param x: the data to decode
+ @type sloppy: C{boolean}
+ @param sloppy: whether to allow errors in the decoding
+ @rtype: unknown
+ @return: the bdecoded data
+ @raise ValueError: if the data is improperly encoded
+
+ """
+
try:
r, l = decode_func[x[0]](x, 0)
# except (IndexError, KeyError):
@@ -89,6 +175,7 @@
return r
def test_bdecode():
+ """A test routine for the bdecoding functions."""
try:
bdecode('0:0:')
assert 0
@@ -233,36 +320,122 @@
bencached_marker = []
class Bencached:
+ """Dummy data structure for storing bencoded data in memory.
+
+ @type marker: C{list}
+ @ivar marker: mutable type to make sure the data was encoded by this class
+ @type bencoded: C{string}
+ @ivar bencoded: the bencoded data stored in a string
+
+ """
+
def __init__(self, s):
+ """
+
+ @type s: C{string}
+ @param s: the new bencoded data to store
+
+ """
+
self.marker = bencached_marker
self.bencoded = s
BencachedType = type(Bencached('')) # insufficient, but good as a filter
def encode_bencached(x,r):
+ """Bencode L{Bencached} data.
+
+ @type x: L{Bencached}
+ @param x: the data to encode
+ @type r: C{list}
+ @param r: the currently bencoded data, to which the bencoding of x
+ will be appended
+
+ """
+
assert x.marker == bencached_marker
r.append(x.bencoded)
def encode_int(x,r):
+ """Bencode an integer.
+
+ @type x: C{int}
+ @param x: the data to encode
+ @type r: C{list}
+ @param r: the currently bencoded data, to which the bencoding of x
+ will be appended
+
+ """
+
r.extend(('i',str(x),'e'))
def encode_bool(x,r):
+ """Bencode a boolean.
+
+ @type x: C{boolean}
+ @param x: the data to encode
+ @type r: C{list}
+ @param r: the currently bencoded data, to which the bencoding of x
+ will be appended
+
+ """
+
encode_int(int(x),r)
def encode_string(x,r):
+ """Bencode a string.
+
+ @type x: C{string}
+ @param x: the data to encode
+ @type r: C{list}
+ @param r: the currently bencoded data, to which the bencoding of x
+ will be appended
+
+ """
+
r.extend((str(len(x)),':',x))
def encode_unicode(x,r):
+ """Bencode a unicode string.
+
+ @type x: C{unicode}
+ @param x: the data to encode
+ @type r: C{list}
+ @param r: the currently bencoded data, to which the bencoding of x
+ will be appended
+
+ """
+
#r.append('u')
encode_string(x.encode('UTF-8'),r)
def encode_list(x,r):
- r.append('l')
- for e in x:
- encode_func[type(e)](e, r)
- r.append('e')
+ """Bencode a list.
+
+ @type x: C{list}
+ @param x: the data to encode
+ @type r: C{list}
+ @param r: the currently bencoded data, to which the bencoding of x
+ will be appended
+
+ """
+
+ r.append('l')
+ for e in x:
+ encode_func[type(e)](e, r)
+ r.append('e')
def encode_dict(x,r):
+ """Bencode a dictionary.
+
+ @type x: C{dictionary}
+ @param x: the data to encode
+ @type r: C{list}
+ @param r: the currently bencoded data, to which the bencoding of x
+ will be appended
+
+ """
+
r.append('d')
ilist = x.items()
ilist.sort()
@@ -285,6 +458,15 @@
encode_func[UnicodeType] = encode_unicode
def bencode(x):
+ """Bencode some data.
+
+ @type x: unknown
+ @param x: the data to encode
+ @rtype: string
+ @return: the bencoded data
+ @raise ValueError: if the data contains a type that cannot be encoded
+
+ """
r = []
try:
encode_func[type(x)](x, r)
@@ -294,6 +476,7 @@
return ''.join(r)
def test_bencode():
+ """A test routine for the bencoding functions."""
assert bencode(4) == 'i4e'
assert bencode(0) == 'i0e'
assert bencode(-10) == 'i-10e'
Modified: debtorrent/branches/http-listen/DebTorrent/bitfield.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/bitfield.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/bitfield.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/bitfield.py Tue May 29 05:58:07 2007
@@ -1,8 +1,21 @@
# Written by Bram Cohen, Uoti Urpala, and John Hoffman
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+
+"""Manage creation and modification of bitfields.
+
+ at type lookup_table: C{list} of C{tuple} of C{boolean}
+ at var lookup_table: A lookup table for converting characters to a tuple of
+ booleans. Each tuple has 8 booleans, which are true for all the bits
+ in the character that are set.
+ at type reverse_lookup_table: C{dictionary}
+ at var reverse_lookup_table: A reverse mapping of lookup_table. The keys are
+ the tuples of 8 booleans, which map to values that are the characters
+ the tuples were created from.
+
+"""
try:
True
@@ -18,6 +31,15 @@
negsum = lambda a: reduce(lambda x,y: x+(not y), a, 0)
def _int_to_booleans(x):
+ """Convert an integer to a list of booleans.
+
+ @type x: C{int}
+ @param x: the integer to convert
+ @rtype: C{tuple} of C{list} of C{boolean}
+ @return: the list of booleans
+
+ """
+
r = []
for i in range(8):
r.append(bool(x & 0x80))
@@ -33,7 +55,32 @@
class Bitfield:
+ """ A bitfield, a indicating the pieces a peer has.
+
+ @type length: C{int}
+ @ivar length: the length of the bitfield
+ @type array: C{list} of C{boolean}
+ @ivar array: the bitfield
+ @type numfalse: C{int}
+ @ivar numfalse: the number of false entries in the bitfield
+
+ """
+
def __init__(self, length = None, bitstring = None, copyfrom = None):
+ """
+
+ @type length: C{int}
+ @param length: the length of the bitfield to create
+ (optional, if missing use length of copyfrom)
+ @type bitstring: C{string}
+ @param bitstring: the bitfield string to initialize the bitfield from
+ (optional, default is to initialize all to false)
+ @type copyfrom: L{Bitfield}
+ @param copyfrom: another bitfield to make a copy of
+ (optional, default is to create a new empty one)
+
+ """
+
if copyfrom is not None:
self.length = copyfrom.length
self.array = copyfrom.array[:]
@@ -61,17 +108,49 @@
self.numfalse = length
def __setitem__(self, index, val):
+ """Set one of the bitfield entries.
+
+ @type index: C{int}
+ @param index: the index to set
+ @type val: C{boolean}
+ @param val: the value to set to
+
+ """
+
val = bool(val)
self.numfalse += self.array[index]-val
self.array[index] = val
def __getitem__(self, index):
+ """Get one of the bitfield entries.
+
+ @type index: C{int}
+ @param index: the index to get
+ @rtype: C{boolean}
+ @return: the value of the bitfield entry
+
+ """
+
return self.array[index]
def __len__(self):
+ """Get the length of the bitfield.
+
+ @rtype:C{int}
+ @return: the length of the bitfield
+
+ """
+
return self.length
def tostring(self):
+ """Convert the bitfield to a string
+
+ @rtype: C{string}
+ @return: the bitfield represented as a string
+
+ """
+
booleans = self.array
t = reverse_lookup_table
s = len(booleans) % 8
@@ -81,10 +160,18 @@
return ''.join(r)
def complete(self):
+ """Check if the bitfield is all true.
+
+ @rtype: C{boolean}
+ @return: whether the bitfield is complete (all true)
+
+ """
+
return not self.numfalse
def test_bitfield():
+ """Test the bitfield implementation."""
try:
x = Bitfield(7, 'ab')
assert False
Modified: debtorrent/branches/http-listen/DebTorrent/clock.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/clock.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/clock.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/clock.py Tue May 29 05:58:07 2007
@@ -1,8 +1,25 @@
# Written by John Hoffman
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Implement wall clock time for Unix systems.
+
+This module implements a clock() function that will return a non-decreasing
+time, regardless of the system it is called on. This is necessary for Unix
+systems, whose clock() function instead returns the current processor time.
+
+ at type _MAXFORWARD: C{int}
+ at var _MAXFORWARD: the maximum number of seconds to allow the clock
+ to move forward
+ at type _FUDGE: C{int}
+ at var _FUDGE: the fudged time change to use if the clock moved forward more
+ than L{_MAXFORWARD}, or if the clock moved back
+ at type _RTIME: L{RelativeTime}
+ at var _RTIME: the RelativeTime instance to use
+
+"""
from time import *
import sys
@@ -11,11 +28,35 @@
_FUDGE = 1
class RelativeTime:
+ """Calculate relative time on Unix systems.
+
+ @type time: C{float}
+ @ivar time: the last time value measured
+ @type offset: C{float}
+ @ivar offset: the offset to use from the current time values due to
+ any changes made in the clock while the program was running
+
+ """
+
def __init__(self):
+ """Initialize the time values."""
self.time = time()
self.offset = 0
def get_time(self):
+ """Calculate a non-decreasing time.
+
+ Uses the time() function to calculate non-decreasing time values.
+ Checks to make sure the time values are non-decreasing, and also
+ don't change by more than L{_MAXFORWARD} seconds within a reading.
+ These could occur if the system clock was changed during the running
+ of the program.
+
+ @rtype: C{float}
+ @return: the current time
+
+ """
+
t = time() + self.offset
if t < self.time or t > self.time + _MAXFORWARD:
self.time += _FUDGE
@@ -27,4 +68,14 @@
if sys.platform != 'win32':
_RTIME = RelativeTime()
def clock():
+ """Override the clock() function for Unix systems.
+
+ This function will return a non-decreasing measure of the current
+ time. This is only used on Unix systems. On Windows systems, the
+ clock() function from the C{time} module will be used.
+
+ @rtype: C{float}
+ @return: the relative time from the L{RelativeTime} instance
+
+ """
return _RTIME.get_time()
Modified: debtorrent/branches/http-listen/DebTorrent/download_bt1.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/download_bt1.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/download_bt1.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/download_bt1.py Tue May 29 05:58:07 2007
@@ -1,8 +1,19 @@
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+
+"""Manage a single download.
+
+ at type DEBUG: C{boolean}
+ at var DEBUG: whether to print debugging information
+ at type defaults: C{list} of C{tuple}
+ at var defaults: the default configuration variables, including descriptions
+ at type argslistheader: C{string}
+ at var argslistheader: the header to print before the default config
+
+"""
from zurllib import urlopen
from urlparse import urlparse
@@ -189,12 +200,46 @@
def _failfunc(x):
+ """Default function to use for printing error messages.
+
+ @type x: C{string}
+ @param x: the error message to print
+
+ """
+
print x
-# old-style downloader
def download(params, filefunc, statusfunc, finfunc, errorfunc, doneflag, cols,
pathFunc = None, presets = {}, exchandler = None,
failed = _failfunc, paramfunc = None):
+ """The old-style downloader (no longer used).
+
+ @type params: unknown
+ @param params: unknown
+ @type filefunc: unknown
+ @param filefunc: unknown
+ @type statusfunc: unknown
+ @param statusfunc: unknown
+ @type finfunc: unknown
+ @param finfunc: unknown
+ @type errorfunc: unknown
+ @param errorfunc: unknown
+ @type doneflag: unknown
+ @param doneflag: unknown
+ @type cols: unknown
+ @param cols: unknown
+ @type pathFunc: unknown
+ @param pathFunc: unknown
+ @type presets: unknown
+ @param presets: unknown
+ @type exchandler: unknown
+ @param exchandler: unknown
+ @type failed: unknown
+ @param failed: unknown
+ @type paramfunc: unknown
+ @param paramfunc: unknown
+
+ """
try:
config = parse_params(params, presets)
@@ -263,6 +308,18 @@
def parse_params(params, presets = {}):
+ """Parse the command-line parameters.
+
+ @type params: C{list} of C{string}
+ @param params: the command-line parameters
+ @type presets: C{dictionary}
+ @param presets: the preset values to use (optional)
+ @rtype: C{dictionary}
+ @return: the configuration variables
+ @raise ValueError: if the parameters are not properly specified
+
+ """
+
if len(params) == 0:
return None
config, args = parseargs(params, defaults, 0, 1, presets = presets)
@@ -283,11 +340,46 @@
def get_usage(defaults = defaults, cols = 100, presets = {}):
+ """Print the usage information for the program.
+
+ @type defaults: C{list} of C{tuple}
+ @param defaults: the default configuration variables
+ (optional, default is to use L{defaults})
+ @type cols: C{int}
+ @param cols: the width of the print out (optional, default is 100)
+ @type presets: C{dictionary}
+ @param presets: the preset values to use (optional)
+
+ """
+
return (argslistheader + formatDefinitions(defaults, cols, presets))
def get_response(file, url, errorfunc):
- response = get_packages(file, url, errorfunc)
+ """Get the response data from a metainfo or Packages file.
+
+ First checks to see if the data is in the Packages file format, and
+ returns the extracted response data if it is. Otherwise, assumes it is
+ a metainfo file and tries to get the response data from it.
+
+ @type file: C{string}
+ @param file: the file name to use, or None to indicate that the url is
+ to be used
+ @type url: C{string}
+ @param url: the URL to download the metainfo file from
+ @type status_to_download: C{int}
+ @param status_to_download: determines which packages to download based on
+ /var/lib/dpkg/status (0 = disabled [download all or use --priority],
+ 1 = download updated versions of installed packages,
+ 2 = download all installed packages)
+ @type errorfunc: C{function}
+ @param errorfunc: the function to use to print any error messages
+ @rtype: C{dictionary}
+ @return: the metainfo data
+
+ """
+
+ response = get_packages(file, url)
if response:
try:
check_message(response)
@@ -342,7 +434,24 @@
return response
-def get_packages(file, url, errorfunc):
+def get_packages(file, url):
+ """Extract the response data from a Packages file.
+
+ @type file: C{string}
+ @param file: the file name to use, or None to indicate that the url is
+ to be used
+ @type url: C{string}
+ @param url: the URL to download the metainfo file from
+ @type status_to_download: C{int}
+ @param status_to_download: determines which packages to download based on
+ /var/lib/dpkg/status (0 = disabled [download all or use --priority],
+ 1 = download updated versions of installed packages,
+ 2 = download all installed packages)
+ @rtype: C{dictionary}
+ @return: the metainfo data
+
+ """
+
encoding = None
# if params.has_key('filesystem_encoding'):
# encoding = params['filesystem_encoding']
@@ -432,9 +541,151 @@
class BT1Download:
+ """Manage a single download.
+
+ @type statusfunc: C{method}
+ @ivar statusfunc: the method to call to print status updates
+ @type finfunc: C{method}
+ @ivar finfunc: the method to call when the download is completed
+ @type errorfunc: C{method}
+ @ivar errorfunc: the method to call when an error occurs
+ @type excfunc: C{method}
+ @ivar excfunc: the method to call when an exception occurs
+ @type doneflag: C{threading.Event}
+ @ivar doneflag: the flag that indicates when the program is to be shutdown
+ @type config: C{dictionary}
+ @ivar config: the configuration variables
+ @type response: C{dictionary}
+ @ivar response: the response data from the metainfo file
+ @type infohash: C{string}
+ @ivar infohash: the hash of the info from the response data
+ @type myid: C{string}
+ @ivar myid: the peer ID to use
+ @type rawserver: L{Rawserver.Rawserver}
+ @ivar rawserver: the server controlling the program
+ @type port: C{int}
+ @ivar port: the port being listened to
+ @type info: C{dictionary}
+ @ivar info: the info data from the response data
+ @type pieces: C{list} of C{string}
+ @ivar pieces: the hashes of the pieces
+ @type piece_lengths: C{list} of C{int}
+ @ivar piece_lengths: the lengths of the pieces
+ @type len_pieces: C{int}
+ @ivar len_pieces: the number of pieces
+ @type status_priority: C{dictionary}
+ @ivar status_priority: the file priorities from the dpkg/status file
+ @type argslistheader: C{string}
+ @ivar argslistheader: the header to print before the default config
+ @type unpauseflag: C{threading.Event}
+ @ivar unpauseflag: the flag to unset to pause the download
+ @type downloader: L{BT1.Downloader.Downloader}
+ @ivar downloader: the Downloader instance
+ @type storagewrapper: L{BT1.StorageWrapper.StorageWrapper}
+ @ivar storagewrapper: the StorageWrapper instance
+ @type fileselector: L{BT1.FileSelector.FileSelector}
+ @ivar fileselector: the FileSelector instance
+ @type super_seeding_active: C{boolean}
+ @ivar super_seeding_active: whether the download is in super-seed mode
+ @type filedatflag: C{threading.Event}
+ @ivar filedatflag: unknown
+ @type spewflag: C{threading.Event}
+ @ivar spewflag: unknown
+ @type superseedflag: C{threading.Event}
+ @ivar superseedflag: unknown
+ @type whenpaused: unknown
+ @ivar whenpaused: unknown
+ @type finflag: C{threading.Event}
+ @ivar finflag: unknown
+ @type rerequest: unknown
+ @ivar rerequest: unknown
+ @type tcp_ack_fudge: C{float}
+ @ivar tcp_ack_fudge: the fraction of TCP ACK download overhead to add to
+ upload rate calculations
+ @type selector_enabled: C{boolean}
+ @ivar selector_enabled: whether to enable the file selector and fast resume function
+ @type appdataobj: L{ConfigDir.ConfigDir}
+ @ivar appdataobj: the configuration and cache directory manager
+ @type excflag: C{threading.Event}
+ @ivar excflag: unknown
+ @type failed: unknown
+ @ivar failed: unknown
+ @type checking: unknown
+ @ivar checking: unknown
+ @type started: unknown
+ @ivar started: unknown
+ @type picker: L{BT1.PiecePicker.PiecePicker}
+ @ivar picker: the PiecePicker instance
+ @type choker: L{BT1.Choker.Choker}
+ @ivar choker: the Choker instance
+ @type filename: C{string}
+ @ivar filename: the save location
+ @type files: C{list} of (C{string}, C{long})
+ @ivar files: the full file names and lengths of all the files in the download
+ @type datalength: C{long}
+ @ivar datalength: the total length of the download
+ @type priority: unknown
+ @ivar priority: unknown
+ @type storage: unknown
+ @ivar storage: unknown
+ @type upmeasure: unknown
+ @ivar upmeasure: unknown
+ @type downmeasure: unknown
+ @ivar downmeasure: unknown
+ @type ratelimiter: unknown
+ @ivar ratelimiter: unknown
+ @type ratemeasure: unknown
+ @ivar ratemeasure: unknown
+ @type ratemeasure_datarejected: unknown
+ @ivar ratemeasure_datarejected: unknown
+ @type connecter: unknown
+ @ivar connecter: unknown
+ @type encoder: unknown
+ @ivar encoder: unknown
+ @type encoder_ban: unknown
+ @ivar encoder_ban: unknown
+ @type httpdownloader: unknown
+ @ivar httpdownloader: unknown
+ @type statistics: unknown
+ @ivar statistics: unknown
+
+ """
+
def __init__(self, statusfunc, finfunc, errorfunc, excfunc, doneflag,
config, response, infohash, id, rawserver, port,
appdataobj = None):
+ """Initialize the instance.
+
+ @type statusfunc: C{method}
+ @param statusfunc: the method to call to print status updates
+ @type finfunc: C{method}
+ @param finfunc: the method to call when the download is completed
+ @type errorfunc: C{method}
+ @param errorfunc: the method to call when an error occurs
+ @type excfunc: C{method}
+ @param excfunc: the method to call when an exception occurs
+ @type doneflag: C{threading.Event}
+ @param doneflag: the flag that indicates when the program is to be shutdown
+ @type config: C{dictionary}
+ @param config: the configuration variables
+ @type response: C{dictionary}
+ @param response: the response data from the metainfo file
+ @type infohash: C{string}
+ @param infohash: the hash of the info from the response data
+ @type id: C{string}
+ @param id: the peer ID to use
+ @type rawserver: L{Rawserver.Rawserver}
+ @param rawserver: the server controlling the program
+ @type port: C{int}
+ @param port: the port being listened to
+ @type status_priority: C{dictionary}
+ @param status_priority: the file priorities, keys are file names,
+ values are the priority to use (optional, defaults to download all)
+ @type appdataobj: L{ConfigDir.ConfigDir}
+ @param appdataobj: the configuration and cache directory manager
+
+ """
+
self.statusfunc = statusfunc
self.finfunc = finfunc
self.errorfunc = errorfunc
@@ -487,6 +738,17 @@
def checkSaveLocation(self, loc):
+ """Check whether the download location exists.
+
+ For multiple files, returns true if a single one exists.
+
+ @type loc: C{string}
+ @param loc: the save location to check
+ @rtype: C{boolean}
+ @return: whether the location exists.
+
+ """
+
if self.info.has_key('length'):
return path.exists(loc)
for x in self.info['files']:
@@ -496,8 +758,31 @@
def saveAs(self, filefunc, pathfunc = None):
+ """Initialize the location to save the download to.
+
+ @type filefunc: C{method}
+ @param filefunc: the method to call to get the location to save the
+ download
+ @type pathfunc: C{method}
+ @param pathfunc: the method to call to alert the UI to any possible
+ change in the download location
+ @rtype: C{string}
+ @return: the location to save to
+
+ """
+
try:
def make(f, forcedir = False):
+ """Create the parent directories of a file.
+
+ @type f: C{string}
+ @param f: the file name
+ @type forcedir: C{boolean}
+ @param forcedir: set to True if f is a directory name
+ (optional, defaults to False)
+
+ """
+
if not forcedir:
f = path.split(f)[0]
if f != '' and not path.exists(f):
@@ -551,7 +836,8 @@
for i in x['path']:
n = path.join(n, i)
files.append((n, x['length']))
- make(n)
+ #Move directory creation to Storage
+ #make(n)
except OSError, e:
self.errorfunc("Couldn't allocate dir - " + str(e))
return None
@@ -564,10 +850,24 @@
def getFilename(self):
+ """Get the download location.
+
+ @rtype: C{string}
+ @return: the download location
+
+ """
+
return self.filename
def _finished(self):
+ """Finish the download.
+
+ Set the files to read-only, update the Choker for seeding, stop the
+ piece requester.
+
+ """
+
self.finflag.set()
try:
self.storage.set_readonly()
@@ -583,11 +883,27 @@
self.finfunc()
def _data_flunked(self, amount, index):
+ """Process a failed hash check on a piece.
+
+ @type amount: C{int}
+ @param amount: the amount of failed data
+ @type index: C{int}
+ @param index: the piece that failed
+
+ """
+
self.ratemeasure_datarejected(amount)
if not self.doneflag.isSet():
self.errorfunc('piece %d failed hash check, re-downloading it' % index)
def _failed(self, reason):
+ """Stop the failed download.
+
+ @type reason: C{string}
+ @param reason: the reason for the failure
+
+ """
+
self.failed = True
self.doneflag.set()
if reason is not None:
@@ -595,6 +911,22 @@
def initFiles(self, old_style = False, statusfunc = None):
+ """Initialize the files for the download.
+
+ Initialize the priorities, then create the Storage, StorageWrapper,
+ and FileSelector. Then initialize the StorageWrapper.
+
+ @type old_style: C{boolean}
+ @param old_style: whether to use the old-style StorageWrapper
+ initialization (optional, defaults to False)
+ @type statusfunc: C{method}
+ @param statusfunc: the method to use to diplay status updates
+ (optional, defaults to using L{statusfunc}
+ @rtype: C{boolean}
+ @return: whether the initialization was successful
+
+ """
+
if self.doneflag.isSet():
return None
if not statusfunc:
@@ -659,6 +991,9 @@
data = data.get('resume data')
if data:
self.fileselector.unpickle(data)
+
+ if self.priority:
+ self.fileselector.initialize_priorities_now(self.priority)
self.checking = True
if old_style:
@@ -667,20 +1002,63 @@
def getCachedTorrentData(self):
+ """Get the cached torrent data from the cache directory.
+
+ @rtype: C{dictionary}
+ @return: the bdecoded cached data
+
+ """
+
return self.appdataobj.getTorrentData(self.infohash)
def _make_upload(self, connection, ratelimiter, totalup):
+ """Create a new Upload instance
+
+ @type connection: unknown
+ @param connection: the connection to upload on
+ @type ratelimiter: L{BT1.RateLimiter.RateLimiter}
+ @param ratelimiter: the RateLimiter instance to use
+ @type totalup: L{CurrentRateMeasure.Measure}
+ @param totalup: the Measure instance to use to calculate the total
+ upload rate
+ @rtype: L{BT1.Uploader.Upload}
+ @return: the new Upload instance
+
+ """
+
return Upload(connection, ratelimiter, totalup,
self.choker, self.storagewrapper, self.picker,
self.config)
def _kick_peer(self, connection):
+ """Disconnect a peer.
+
+ @type connection: unknown
+ @param connection: the connection of the peer to disconnect
+
+ """
+
def k(connection = connection):
+ """Close a connection.
+
+ @type connection: unknown
+ @param connection: the connection of the peer to disconnect
+ (optional, defaults to the _kick_peer connection)
+
+ """
+
connection.close()
self.rawserver.add_task(k,0)
def _ban_peer(self, ip):
+ """Ban a peer from the download.
+
+ @type ip: C{string}
+ @param ip: the IP address of the peer to ban
+
+ """
+
self.encoder_ban(ip)
def _received_raw_data(self, x):
Modified: debtorrent/branches/http-listen/DebTorrent/inifile.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/inifile.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/inifile.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/inifile.py Tue May 29 05:58:07 2007
@@ -4,28 +4,33 @@
# $Id$
-'''
-reads/writes a Windows-style INI file
-format:
-
- aa = "bb"
- cc = 11
-
- [eee]
- ff = "gg"
-
-decodes to:
-d = { '': {'aa':'bb','cc':'11'}, 'eee': {'ff':'gg'} }
-
-the encoder can also take this as input:
-
-d = { 'aa': 'bb, 'cc': 11, 'eee': {'ff':'gg'} }
+'''Functions to read/write a Windows-style INI file
+
+format::
+
+ aa = "bb"
+ cc = 11
+
+ [eee]
+ ff = "gg"
+
+decodes to::
+
+ d = { '': {'aa':'bb','cc':'11'}, 'eee': {'ff':'gg'} }
+
+the encoder can also take this as input::
+
+ d = { 'aa': 'bb, 'cc': 11, 'eee': {'ff':'gg'} }
though it will only decode in the above format. Keywords must be strings.
Values that are strings are written surrounded by quotes, and the decoding
-routine automatically strips any.
-Booleans are written as integers. Anything else aside from string/int/float
-may have unpredictable results.
+routine automatically strips any quotes from the values.
+Booleans are written as integers. Anything other than strings, integers,
+and floats may have unpredictable results.
+
+ at type DEBUG: C{boolean}
+ at var DEBUG: whether to print debugging information
+
'''
from cStringIO import StringIO
@@ -45,6 +50,20 @@
DEBUG = True
def ini_write(f, d, comment=''):
+ """Write the ini file.
+
+ @type f: C{string}
+ @param f: the file name to write
+ @type d: C{dictionary}
+ @param d: the data to write to the ini file
+ @type comment: C{string}
+ @param comment: a comment to write at the top of the file, hash marks
+ will be prefixed (optional, default is no comment)
+ @rtype: C{boolean}
+ @return: whether the write succeeded
+
+ """
+
try:
a = {'':{}}
for k,v in d.items():
@@ -107,11 +126,34 @@
if DEBUG:
def errfunc(lineno, line, err):
+ """Display an error message when reading the ini file fails.
+
+ @type lineno: C{int}
+ @param lineno: the line number that caused the error
+ @type line: C{string}
+ @param line: the line that caused the error
+ @type err: C{string}
+ @param err: the error that was generated
+
+ """
+
print '('+str(lineno)+') '+err+': '+line
else:
errfunc = lambda lineno, line, err: None
def ini_read(f, errfunc = errfunc):
+ """Read the ini file.
+
+ @type f: C{string}
+ @param f: the file name to read
+ @type errfunc: C{function}
+ @param errfunc: the function to call when an error occurs
+ (optional, default is to do nothing, unless debugging is enabled in
+ which case the error is printed to standard output)
+ @rtype: C{dictionary}
+ @return: the data read from the ini file
+
+ """
try:
r = open(f,'r')
ll = r.readlines()
Modified: debtorrent/branches/http-listen/DebTorrent/parsedir.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/DebTorrent/parsedir.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/DebTorrent/parsedir.py (original)
+++ debtorrent/branches/http-listen/DebTorrent/parsedir.py Tue May 29 05:58:07 2007
@@ -5,7 +5,7 @@
# $Id$
from bencode import bencode, bdecode
-from BT1.btformats import check_info
+from download_bt1 import get_response
from os.path import exists, isfile
from sha import sha
import sys, os
@@ -22,7 +22,8 @@
print ":: "+x
def parsedir(directory, parsed, files, blocked,
- exts = ['.dtorrent'], return_metainfo = False, errfunc = _errfunc):
+ exts = ['dtorrent','Packages'],
+ return_metainfo = False, errfunc = _errfunc):
if NOISY:
errfunc('checking dir')
dirs_to_check = [directory]
@@ -36,7 +37,7 @@
newtorrent = None
for ext in exts:
if f.endswith(ext):
- newtorrent = ext[1:]
+ newtorrent = ext[:]
break
if newtorrent:
newtorrents = True
@@ -91,9 +92,7 @@
if NOISY:
errfunc('adding '+p)
try:
- ff = open(p, 'rb')
- d = bdecode(ff.read())
- check_info(d['info'])
+ d = get_response(p, '', errfunc)
h = sha(bencode(d['info'])).digest()
new_file[1] = h
if new_parsed.has_key(h):
@@ -133,10 +132,6 @@
errfunc('**warning** '+p+' has errors')
new_blocked[p] = 1
continue
- try:
- ff.close()
- except:
- pass
if NOISY:
errfunc('... successful')
new_parsed[h] = a
Modified: debtorrent/branches/http-listen/README.txt
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/README.txt?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/README.txt (original)
+++ debtorrent/branches/http-listen/README.txt Tue May 29 05:58:07 2007
@@ -1,65 +1,105 @@
To start testing DebTorrent, do the following:
-1. Download the source from the Alioth project:
+1. Download the source from the Alioth project:
- https://alioth.debian.org/frs/?group_id=31109
+ https://alioth.debian.org/frs/?group_id=31109
-2. Unpack it:
+2. Unpack it:
- tar -xzvf debtorrent_<version>.tar.gz
+ tar -xzvf debtorrent-<version>.tar.gz
-3. Change into the debtorrent directory:
+3. Change into the debtorrent directory:
- cd debtorrent-<version>
+ cd debtorrent-<version>
-4. Choose an archive to download:
+4. Copy into dtorrents/ the package lists in /var/lib/apt/lists/ that you
+ want to download from:
- a) Choose a .dtorrent file from the dtorrents directory
+ cp /var/lib/apt/lists/ftp.us.debian.org_debian_dists_etch_main_binary-i386_Packages \
+ dtorrents/ftp.us.debian.org_debian_dists_etch_main_binary-i386_Packages
- You can view the contents of the file using the btshowmetainfo.py command
+ Or, if you want to download from all of the package lists:
+
+ cp /var/lib/apt/lists/*_Packages dtorrents/
+
+5. Update your package lists (as root):
+
+ apt-get update
+
+6. Start the download:
+
+ ./btlaunchmany.py dtorrents/ --saveas downloads/ --status_to_download <x>
+
+ Set the <x> value to:
+
+ 0 - to download all of the packages in each Packages file (a mirror
+ of the archives, could be VERY large)
+
+ 1 - to download only the packages that are installed, according to
+ /var/lib/dpkg/status, and the version available is newer than the
+ installed one (this is the smallest download, but may do nothing if
+ your system is up to date)
- To determine if the dtorrent has peers/seeds, go to
-
- http://dttracker.debian.net:6969
-
- and look for the info hash shown by btshowmetainfo.
+ 2 - to download all the packages that are installed, according to
+ /var/lib/dpkg/status, regardless of the version
+
+ You can use some of the options to btlaunchmany (to see them, run it
+ without arguments) but most are untested.
+
+7. Wait for the downloads to complete:
- b) Or, create a .dtorrent file using btmakemetafile from a Packages file:
+ If you have any problems downloading, go to
+
+ http://dttracker.debian.net:6969
+
+ to check if your download is active. You can find which one is yours by
+ running btshowmetainfo on a Packages file to display the info hash of
+ the file:
+
+ btshowmetainfo.py ftp.us.debian.org_debian_dists_etch_main_binary-i386_Packages | grep "info hash"
+
+ If it is active, but you still can't download, or if you get errors from
+ running any of the programs, please post the problem/error to:
+
+ debtorrent-devel at lists.alioth.debian.org
- btmakemetafile.py http://dttracker.debian.net:6969/announce \
- /var/lib/apt/lists/<mirror>_debian_dists_<suite>_<section>_binary-<arch>_Packages \
- --target test.dtorrent
+8. Add the download directories to the top of your /etc/apt/sources.list,
+ for example:
- c) Or, choose a Packages file from /var/lib/apt/lists/
-
- d) Or, find one on the web, for example:
-
- http://ftp.us.debian.org/debian/dists/stable/contrib/binary-i386/Packages.gz
-
- The recommended Packages file to use is from the stable suite (as it
- doesn't change much), and section contrib (as it's the smallest).
+ deb file:/tmp/debtorrent/downloads/ftp.us.debian.org_debian_dists_etch_main_binary-amd64 ./
+ deb file:/tmp/debtorrent/downloads/ftp.us.debian.org_debian_dists_etch_contrib_binary-amd64 ./
+ etc...
+
+ There should be one entry for every directory in your downloads/ directory
+ that completed downloading. Do NOT add directories for downloads that did
+ not complete, as the pre-allocated empty files in them will cause MD5 sum
+ errors which will prevent apt from installing the packages.
+
+ The lines must be added to the top of the sources.list in order for apt
+ to get the files from there first rather than downloading them from the
+ mirror listed later in the file.
-5. Start the download, depending on which archive was chosen, do the following:
+9. Link the Packages files into the download directories, for example:
- a) btdownloadheadless.py dtorrents/<file>.dtorrent
+ ln -s dtorrents/ftp.us.debian.org_debian_dists_etch_main_binary-i386_Packages \
+ downloads/ftp.us.debian.org_debian_dists_etch_main_binary-i386/Packages
+
+ This is necessary for the next step to work, that is for apt to recognize
+ that the download directories are in fact package repositories.
- b) btdownloadheadless.py test.dtorrent
+10. Update your package lists (as root) to see the newly downloaded files:
- c) btdownloadheadless.py /var/lib/apt/lists/<mirror>_debian_dists_<suite>_<section>_binary-<arch>_Packages
+ apt-get update
+
+11. Install or upgrade packages normally (as root):
- d) btdownloadheadless.py http://ftp.us.debian.org/debian/dists/stable/contrib/binary-i386/Packages.gz
-
- You can use some of the options to btdownloadheadless (to see them, run it
- without arguments) but most are untested.
+ apt-get install ...
+ apt-get upgrade
+ apt-get dist-upgrade
-If you have any problems downloading, go to
-
- http://dttracker.debian.net:6969
-
-to check if your torrent is active. If it is, but you still can't download,
-or if you get errors from running any of the programs, please post the
-problem/error to debtorrent-devel at lists.alioth.debian.org
+For future upgrades/installs, repeat steps 5-11, omitting step 9 (1-4 and 9
+are not needed as they were only for initialization).
Known Bugs
@@ -68,8 +108,8 @@
On starting a download, the following error is generated:
Traceback (most recent call last):
- File "/home/user/workspace/debtorrent/DebTorrent/inifile.py", line 113, in ini_read
+ File "DebTorrent/inifile.py", line 113, in ini_read
r = open(f,'r')
-IOError: [Errno 2] No such file or directory: '/home/user/.DebTorrent/config.downloadheadless.ini'
+IOError: [Errno 2] No such file or directory: '/home/user/.DebTorrent/config.launchmany.ini'
The error has no effect on the download.
Modified: debtorrent/branches/http-listen/TODO
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/TODO?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/TODO (original)
+++ debtorrent/branches/http-listen/TODO Tue May 29 05:58:07 2007
@@ -68,35 +68,6 @@
matches.
-Modify pre-allocation to not create directories
-
-Currently, the pre-allocation type of allocation will create all the
-directories in the download, even if the files in the directory are not being
-downloaded. This could be modified to create directories on demand to reduce
-filesystem bloat.
-
-
-Add parsing of currently installed packages to determine what to download
-
-As a temporary measure, before full apt proxy support (dt://) is possible,
-debtorrent could parse the currently installed packages list in
-/var/lib/dpkg/status and use that list to determine which files to download.
-Then, configuration options (installed-only, installed-and-updated,
-updated-only) would tell debtorrent which files to download. updated-only would
-check the new version with dpkg --compare-versions to determine if the new
-version should be downloaded. The debtorrent directory could then be added as
-a file:/ source to apt, and new packages to install would not be found and use
-the backup method (http://...). Then, an upgrade path would be:
-1) apt-get update
-2) run debtorrent with the new Packages files
-3) wait for the download of updated packages to complete
-4) run apt-get upgrade/dist-upgrade
-Or the process could be automated by a cron script that gives debtorrent new
-Packages files every week/day/hour. Might need to parse the
-/var/cache/apt/archive directory when run the first time to hardlink in files
-that have already been downloaded.
-
-
Consider Sources
Sources files contain 2 or 3 files for every source package, but no SHA1 sums
Modified: debtorrent/branches/http-listen/btcompletedir.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/btcompletedir.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/btcompletedir.py (original)
+++ debtorrent/branches/http-listen/btcompletedir.py Tue May 29 05:58:07 2007
@@ -1,10 +1,16 @@
#!/usr/bin/env python
-
+#
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Make a metainfo (.dtorrent) file for every file in a directory.
+
+Creates a new metainfo file for every file in a given directory.
+
+"""
from DebTorrent import PSYCO
if PSYCO.psyco:
@@ -22,9 +28,24 @@
from DebTorrent.parseargs import parseargs, formatDefinitions
def prog(amount):
+ """Display the current status of the file scan.
+
+ @type amount: C{int}
+ @param amount: the number of packages that have been found so far in the
+ current file
+
+ """
+
print '%d packages found\r' % amount,
def next_file(file):
+ """Print the name of the file being scanned.
+
+ @type file: C{string}
+ @param file: the file name
+
+ """
+
print "\nProcessing file: %s" % file
if len(argv) < 3:
Modified: debtorrent/branches/http-listen/btcopyannounce.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/btcopyannounce.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/btcopyannounce.py (original)
+++ debtorrent/branches/http-listen/btcopyannounce.py Tue May 29 05:58:07 2007
@@ -1,11 +1,18 @@
#!/usr/bin/env python
-
+#
# btreannounce.py written by Henry 'Pi' James and Bram Cohen
# multitracker extensions by John Hoffman
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Copy the announce information from one metainfo file to others.
+
+Copies all the announce information from an input metainfo file, and
+makes all the other metainfo files match it.
+
+"""
from sys import argv,exit
from os.path import split
@@ -13,6 +20,15 @@
def give_announce_list(l):
+ """Converts an announce list into human-readable output.
+
+ @type l: C{list}
+ @param l: the announce list to convert
+ @rtype: C{string}
+ @return: the human-readbale announce list
+
+ """
+
list = []
for tier in l:
for tracker in tier:
Modified: debtorrent/branches/http-listen/btdownloadheadless.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/btdownloadheadless.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/btdownloadheadless.py (original)
+++ debtorrent/branches/http-listen/btdownloadheadless.py Tue May 29 05:58:07 2007
@@ -1,10 +1,17 @@
#!/usr/bin/env python
-
+#
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
-
+#
# $Id$
+
+"""A single downloader for debtorrent.
+
+This script implements a command-line client for managing a single
+debtorrent download.
+
+"""
from DebTorrent import PSYCO
if PSYCO.psyco:
@@ -45,6 +52,18 @@
PROFILER = False
def hours(n):
+ """Formats seconds into a human-readable time.
+
+ Formats a given number of seconds into a human-readable time appropriate
+ for display to the user.
+
+ @type n: C{int}
+ @param n: the number of seconds
+ @rtype: C{string}
+ @return: a displayable representation of the number of seconds
+
+ """
+
if n == 0:
return 'complete!'
try:
@@ -60,7 +79,41 @@
return '%d min %02d sec' % (m, s)
class HeadlessDisplayer:
+ """Displays on the command-line the status of the download.
+
+ Implements a displayer for showing in text format the current status
+ of the download.
+
+ @type done: C{boolean}
+ @ivar done: whether the download has completed or not
+ @type file: C{string}
+ @ivar file: the download file name to display
+ @type percentDone: C{string}
+ @ivar percentDone: the current percentage of the download that is complete
+ @type timeEst: C{string}
+ @ivar timeEst: the estimated time remaining
+ @type downloadTo: C{string}
+ @ivar downloadTo: the location of the download show to the user
+ @type downRate: C{string}
+ @ivar downRate: the last download rate shown to the user
+ @type upRate: C{string}
+ @ivar upRate: the last upload rate shown to the user
+ @type shareRating: C{string}
+ @ivar shareRating: the last sharing information shown to the user
+ @type seedStatus: C{string}
+ @ivar seedStatus: information on the current status of seeds seen
+ @type peerStatus: C{string}
+ @ivar peerStatus: information on the current status of peers seen
+ @type errors: C{list} of C{string}
+ @ivar errors: a list of all the errors that have occurred, in the order
+ in which they occurred
+ @type last_update_time: C{float}
+ @ivar last_update_time: the time in seconds of the last update
+
+ """
+
def __init__(self):
+ """Initialize the status of the download."""
self.done = False
self.file = ''
self.percentDone = ''
@@ -75,6 +128,7 @@
self.last_update_time = -1
def finished(self):
+ """Concludes the download successfully."""
self.done = True
self.percentDone = '100'
self.timeEst = 'Download Succeeded!'
@@ -82,6 +136,7 @@
self.display()
def failed(self):
+ """Ends the download when an error has occurred."""
self.done = True
self.percentDone = '0'
self.timeEst = 'Download Failed!'
@@ -89,12 +144,34 @@
self.display()
def error(self, errormsg):
+ """Displays an error to the user."""
self.errors.append(errormsg)
self.display()
def display(self, dpflag = Event(), fractionDone = None, timeEst = None,
downRate = None, upRate = None, activity = None,
statistics = None, **kws):
+ """Displays the current status of the download.
+
+ Writes a status update to standard output.
+
+ @type dpflag: a threading event
+ @param dpflag: unknown
+ @type fractionDone: C{float}
+ @param fractionDone: the amount of the download that is complete [0-1]
+ @type timeEst: C{int}
+ @param timeEst: the number of seconds remaining to complete the download
+ @type downRate: C{int} or C{float}
+ @param downRate: the current download rate in bytes per second
+ @type upRate: C{int} or C{float}
+ @param upRate: the current upload rate in bytes per second
+ @type activity: C{string}
+ @param activity: the currently occurring activity (overwrites the timeEst)
+ @type statistics: L{DebTorrent.BT1.Statistics.Statistics}
+ @param statistics: various statistics of the current download
+
+ """
+
if self.last_update_time + 0.1 > clock() and fractionDone not in (0.0, 1.0) and activity is not None:
return
self.last_update_time = clock()
@@ -134,6 +211,25 @@
dpflag.set()
def chooseFile(self, default, size, saveas, dir):
+ """Sets up the save directory name.
+
+ Takes the default and user-provided save directory, saves them, and
+ returns the directory to be used for the download.
+
+ @type default: C{string}
+ @param default: the default save location
+ @type size: C{int}
+ @param size: the size of the download in bytes
+ @type saveas: C{string}
+ @param saveas: the user-provided directory to save the download to,
+ or an empty string indicating the default should be used
+ @type dir: unknown
+ @param dir: unknown
+ @rtype: C{string}
+ @return: the directory to save the download to
+
+ """
+
self.file = '%s (%.1f MB)' % (default, float(size) / (1 << 20))
if saveas != '':
default = saveas
@@ -141,9 +237,19 @@
return default
def newpath(self, path):
+ """Overwrite the previously saved download location."""
self.downloadTo = path
def run(params):
+ """Runs the downloader.
+
+ The main function used to create the displayer and start the download.
+
+ @type params: C{list} of C{strings}
+ @param params: a list of the command-line arguments given to the script
+
+ """
+
h = HeadlessDisplayer()
while 1:
configdir = ConfigDir('downloadheadless')
@@ -170,6 +276,7 @@
doneflag = Event()
def disp_exception(text):
+ """Prints exceptions to standard output."""
print text
rawserver = RawServer(doneflag, config['timeout_check_interval'],
config['timeout'], ipv6_enable = config['ipv6_enabled'],
@@ -250,11 +357,8 @@
import profile, pstats
p = profile.Profile()
p.runcall(run, argv[1:])
- log = open('profile_data.'+strftime('%y%m%d%H%M%S')+'.txt','a')
- normalstdout = sys.stdout
- sys.stdout = log
+ p.dump_stats('btdownloadheadless.pstat')
# pstats.Stats(p).strip_dirs().sort_stats('cumulative').print_stats()
pstats.Stats(p).strip_dirs().sort_stats('time').print_stats()
- sys.stdout = normalstdout
else:
run(argv[1:])
Modified: debtorrent/branches/http-listen/btlaunchmany.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/btlaunchmany.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/btlaunchmany.py (original)
+++ debtorrent/branches/http-listen/btlaunchmany.py Tue May 29 05:58:07 2007
@@ -1,10 +1,17 @@
#!/usr/bin/env python
-
+#
# Written by John Hoffman
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""A multiple downloader for debtorrent.
+
+This script implements a command-line client for managing multiple
+debtorrent downloads.
+
+"""
from DebTorrent import PSYCO
if PSYCO.psyco:
@@ -31,7 +38,21 @@
True = 1
False = 0
+PROFILER = False
+
def hours(n):
+ """Formats seconds into a human-readable time.
+
+ Formats a given number of seconds into a human-readable time appropriate
+ for display to the user.
+
+ @type n: C{int}
+ @param n: the number of seconds
+ @rtype: C{string}
+ @return: a displayable representation of the number of seconds
+
+ """
+
if n == 0:
return 'complete!'
try:
@@ -48,9 +69,28 @@
Exceptions = []
+"""The exceptions that have occurred."""
class HeadlessDisplayer:
+ """Displays on the command-line the status of the downloads.
+
+ Implements a displayer for showing in text format the current status
+ of the downloads.
+
+ """
+
def display(self, data):
+ """Displays the current status of the downloads.
+
+ Writes a status update to standard output.
+
+ @type data: C{list} of C{tuple}
+ @param data: a tuple per download, containing the current status of the download
+ @rtype: C{boolean}
+ @return: False
+
+ """
+
print ''
if not data:
self.message('no torrents')
@@ -63,17 +103,37 @@
return False
def message(self, s):
+ """Prints a message to standard output.
+
+ @type s: C{string}
+ @param s: the message to print to standard output
+
+ """
+
print "### "+s
def exception(self, s):
+ """Saves a new exception and alerts the user.
+
+ @type s: C{string}
+ @param s: the exception that occurred
+
+ """
+
Exceptions.append(s)
self.message('SYSTEM ERROR - EXCEPTION GENERATED')
-if __name__ == '__main__':
- if argv[1:] == ['--version']:
- print version
- exit(0)
+def run(params):
+ """Runs the downloader.
+
+ The main function used to create the displayer and start the download.
+
+ @type params: C{list} of C{strings}
+ @param params: a list of the command-line arguments given to the script
+
+ """
+
defaults.extend( [
( 'parse_dir_interval', 60,
"how often to rescan the torrent directory, in seconds" ),
@@ -96,7 +156,7 @@
print "<directory> - directory to look for .dtorrent files (semi-recursive)"
print get_usage(defaults, 80, configdefaults)
exit(1)
- config, args = parseargs(argv[1:], defaults, 1, 1, configdefaults)
+ config, args = parseargs(params, defaults, 1, 1, configdefaults)
if config['save_options']:
configdir.saveConfig(config)
configdir.deleteOldCacheData(config['expire_cache_data'])
@@ -111,4 +171,19 @@
if Exceptions:
print '\nEXCEPTION:'
print Exceptions[0]
- print 'please report this to '+report_email
+ print 'please report this to '+report_email
+
+if __name__ == '__main__':
+ if argv[1:] == ['--version']:
+ print version
+ exit(0)
+
+ if PROFILER:
+ import profile, pstats
+ p = profile.Profile()
+ p.runcall(run, argv[1:])
+ p.dump_stats('btlaunchmany.pstat')
+# pstats.Stats(p).strip_dirs().sort_stats('cumulative').print_stats()
+ pstats.Stats(p).strip_dirs().sort_stats('time').print_stats()
+ else:
+ run(argv[1:])
Modified: debtorrent/branches/http-listen/btmakemetafile.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/btmakemetafile.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/btmakemetafile.py (original)
+++ debtorrent/branches/http-listen/btmakemetafile.py Tue May 29 05:58:07 2007
@@ -1,11 +1,18 @@
#!/usr/bin/env python
-
+#
# Written by Bram Cohen
# multitracker extensions by John Hoffman
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Creates a metainfo file from a Packages file.
+
+Converts a Packages file into a metainfo file containing all the information
+needed to download all the packages in the Packages file.
+
+"""
from DebTorrent import PSYCO
if PSYCO.psyco:
@@ -24,6 +31,14 @@
def prog(amount):
+ """Display the current status of the file scan.
+
+ @type amount: C{int}
+ @param amount: the number of packages that have been found so far in the
+ file
+
+ """
+
print '%d packages found\r' % amount,
if len(argv) < 3:
Modified: debtorrent/branches/http-listen/btreannounce.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/btreannounce.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/btreannounce.py (original)
+++ debtorrent/branches/http-listen/btreannounce.py Tue May 29 05:58:07 2007
@@ -1,11 +1,13 @@
#!/usr/bin/env python
-
+#
# Written by Henry 'Pi' James and Bram Cohen
# multitracker extensions by John Hoffman
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Change the announce info in a metainfo file."""
from sys import argv,exit
from os.path import split
Modified: debtorrent/branches/http-listen/btrename.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/btrename.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/btrename.py (original)
+++ debtorrent/branches/http-listen/btrename.py Tue May 29 05:58:07 2007
@@ -1,10 +1,17 @@
#!/usr/bin/env python
-
+#
# Written by Henry 'Pi' James
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Change the name field in a metainfo file.
+
+Rename the directory that will be created (suggested) when the metainfo file's
+download is started.
+
+"""
from sys import *
from os.path import *
Modified: debtorrent/branches/http-listen/btsethttpseeds.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/btsethttpseeds.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/btsethttpseeds.py (original)
+++ debtorrent/branches/http-listen/btsethttpseeds.py Tue May 29 05:58:07 2007
@@ -1,11 +1,13 @@
#!/usr/bin/env python
-
+#
# Written by Henry 'Pi' James and Bram Cohen
# multitracker extensions by John Hoffman
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Change the http-seeds info in the metainfo file."""
from sys import argv,exit
from os.path import split
Modified: debtorrent/branches/http-listen/btshowmetainfo.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/btshowmetainfo.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/btshowmetainfo.py (original)
+++ debtorrent/branches/http-listen/btshowmetainfo.py Tue May 29 05:58:07 2007
@@ -1,16 +1,19 @@
#!/usr/bin/env python
-
+#
# Written by Henry 'Pi' James and Loring Holden
# modified for multitracker display by John Hoffman
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Display a summary of the information in a metainfo or Packages file."""
from sys import *
from os.path import *
from sha import *
from DebTorrent.bencode import *
+from DebTorrent.download_bt1 import get_packages
NAME, EXT = splitext(basename(argv[0]))
VERSION = '20030621'
@@ -24,8 +27,12 @@
exit(2) # common exit code for syntax error
for metainfo_name in argv[1:]:
- metainfo_file = open(metainfo_name, 'rb')
- metainfo = bdecode(metainfo_file.read())
+ if len(metainfo_name) >= 8 and metainfo_name[-8:] == 'Packages':
+ (metainfo, temp) = get_packages(metainfo_name, '', 0)
+ else:
+ metainfo_file = open(metainfo_name, 'rb')
+ metainfo = bdecode(metainfo_file.read())
+
# print metainfo
info = metainfo['info']
info_hash = sha(bencode(info))
Modified: debtorrent/branches/http-listen/bttrack.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/bttrack.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/bttrack.py (original)
+++ debtorrent/branches/http-listen/bttrack.py Tue May 29 05:58:07 2007
@@ -1,10 +1,12 @@
#!/usr/bin/env python
-
+#
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""Start a debtorrent download tracker."""
from DebTorrent import PSYCO
if PSYCO.psyco:
@@ -23,15 +25,10 @@
if __name__ == '__main__':
if PROFILE:
import profile, pstats
- from time import strftime
- import sys
p = profile.Profile()
p.runcall(track, argv[1:])
- log = open('profile_data.'+strftime('%y%m%d%H%M%S')+'.txt','a')
- normalstdout = sys.stdout
- sys.stdout = log
+ p.dump_stats('bttrack.pstat')
# pstats.Stats(p).strip_dirs().sort_stats('cumulative').print_stats()
pstats.Stats(p).strip_dirs().sort_stats('time').print_stats()
- sys.stdout = normalstdout
else:
track(argv[1:])
Propchange: debtorrent/branches/http-listen/docs/
------------------------------------------------------------------------------
--- svn:ignore (added)
+++ svn:ignore Tue May 29 05:58:07 2007
@@ -1,0 +1,1 @@
+html
Modified: debtorrent/branches/http-listen/setup.py
URL: http://svn.debian.org/wsvn/debtorrent/debtorrent/branches/http-listen/setup.py?rev=71&op=diff
==============================================================================
--- debtorrent/branches/http-listen/setup.py (original)
+++ debtorrent/branches/http-listen/setup.py Tue May 29 05:58:07 2007
@@ -1,10 +1,18 @@
#!/usr/bin/env python
-
+#
# Written by Bram Cohen
# Modified by Cameron Dale
# see LICENSE.txt for license information
+#
+# $Id$
-# $Id$
+"""The DebTorrent program for downloading debian packages.
+
+This program contains several scripts and packages to implement the debtorrent
+protocol for downloading debian packages from an archive is a bittorrent-like
+way.
+
+"""
import sys
assert sys.version >= '2', "Install Python 2.0 or greater"
@@ -21,11 +29,9 @@
packages = ["DebTorrent","DebTorrent.BT1"],
- scripts = ["btdownloadgui.py", "btdownloadheadless.py",
+ scripts = ["btdownloadheadless.py",
"bttrack.py", "btmakemetafile.py", "btlaunchmany.py", "btcompletedir.py",
- "btdownloadcurses.py", "btcompletedirgui.py", "btlaunchmanycurses.py",
- "btmakemetafile.py", "btreannounce.py", "btrename.py", "btshowmetainfo.py",
- 'btmaketorrentgui.py', 'btcopyannounce.py', 'btsethttpseeds.py',
- 'bt-t-make.py',
+ "btreannounce.py", "btrename.py", "btshowmetainfo.py",
+ 'btcopyannounce.py', 'btsethttpseeds.py',
]
)
More information about the Debtorrent-commits
mailing list