[buildd-tools-devel] SFTP protocol implementation

[Roger Leigh]

Hi folks,

Just a short note to keep you in the loop about progress on
implementing the SFTP protocol.  This is really named the
"SSH File Transfer Protocol", and is documented in
http://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
sftp itself implements an earlier version 2 of the protocol
while this is version 6.  The end implementation should
implement and negotiate protocols 2-6 plus version-specific
and version-independent extensions.

The latest code for this is at
http://git.debian.org/?p=users/rleigh/schroot.git;a=tree;f=sftp;h=272e0482c62f9edfb1ea3ca3e85e2ab272d8a414;hb=f48181c472113c84806bb4a0000413ef3d2f1f82
(sorry, horrific URI, it's actually the client-server-protocol
branch at http://git.debian.org/?p=users/rleigh/schroot.git;a=summary
looking at the sftp directory).


The aim of this is to define a protocol so that schroot can
(initally optionally) run as a daemon instead of a single-shot
setuid- root executable.  The schroot executable would no longer be
setuid, and would instead communicate with the main daemon using this
protocol.  This would have several advantages, including the potential
 to support other virtualisation technologies than chroots and to
support Linux features which are currently not possible to take
advantage of, such as per-process namespaces.  Disadvantage: there
will be a performance hit since currently I/O is just based on the fds
passed by the shell and this will require that to go over the
client/server connection.  This is likely negligable for local users
unless you're streaming a *lot* of data over stdin/stdout.

The protocol is extensible, so schroot-specific requests can be
added in addition to the core set which deals exclusively with
file transfer.  The protocol is suited to the task because most
of the client-server communication would be transfer of file data
(well, pipe I/O, but the protocol implements read() and write()).

The file transfer capabilities are potentially of use to sbuild
as a deamon as well, since it provides the means to submit and
retrieve jobs (sent sources and receive built packages, logs and
other data).


While there's still a lot of work remaining, some parts are
now complete, and others on the way to completion.
Done:
- fundamental type serialisation (signed two's-complement integer
  logic needs checking) to byte stream
- string serialisation of fundamental types
- string serialisation of compound types
There are also testcases for this.  This is the core code implementing
the protocol wire format and conversion to and from native
types/structures/classes.

In progress:
- server request handlers.
  Wire vars in place, just need filling from packet buffer and then
  using.
- client request submission and reply handling.
  Submission is done, handling and error propagation needs adding.

To Do:
- event loops to process the client and server packet buffers and
  despatch handlers.  Will probably use polling for performance.
- Add per-file request queues to allow parallelisation of requests?
- POSIX API to wrap the low level client interface.
- Handle earlier protocol versions than 6 (add conditionals and
  possibly compound type variants).  Realistically, we only need
  to achieve sftp version 2 compatibility, but doing a complete
  job isn't hard (or required).
- Implement schroot-specific requests.
- Add client/server support to schroot.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.