A Quicklisp Debian package

Fri Sep 30 19:59:20 UTC 2011

Hi Paulo,

First of all, is there any particular reason why you've moved this discussion
off-list?  Assuming you did this unintentionally, I've brought the discussion
back on-list (so that we benefit from other people's input, if any).

Quoth Paulo Sequeira <psequeirag at gmail.com>:
> That's my understanding from various mails in the list[1][2]. 

I see.

> The CL Team has very limited availability so they're focusing on maintaining
> core CL infrastructure (implementations and some other critical packages),
> but is letting go of most packages unless a maintainer shows up for them.

Pretty unlikely now that we have Quicklisp, and completely unnecessary if the
quicklisp package(s) we are discussing succeed in making it possible for
administrators to install Quicklisp libraries site-wide (and provides an easy
way for unprivileged users to use them).

> Not really, but I'm not probably the right person to answer that. In any
> case, I've been thinking that depending on lisp-compiler would not really
> help too much, as there's no standard interface for loading & running lisp
> sources.

LOAD is part of the ANSI standard, but yes, important parts of it are
implementation dependent.  However, I don't think this will prevent the
Quicklisp quickstart file from LOADing correctly across implementations.  We
will also need a different invocation command for each lisp.  For example, I'd
be very surprised if --no-sysinit is used by any lisp other than sbcl.

> I initially suggested lisp-compiler because it seemed unfair to arbitrarily
> pick a specific implementation in order to install Quicklisp, but I guess the
> alternative would require just too much work at this time (suggestions
> welcomed).

Requiring lisp-compiler is definitely the right thing to do since we can't
possibly force users to install SBCL in order to install Quicklisp.  However,
it is going to require more work so we should leave this until we have a
working prototype package that works with SBCL.

> This is just great! Way much more than I was hoping to have at this time. You
> and Daniel really kick-started it.

Thanks.

> Agreed, just as would happen to a user that happens to have one of his builds
> broken because the sysadmin upgraded gcc.

Indeed.

> I'm not that versed on Debian's policy to tell. What's the idea behind this?
> To use that cl-quicklisp user to run sbcl whenever we want to install a
> library? Why not use root for that?

If you run sbcl as root, load the quickstart file (quicklisp.lisp) and then
evaluate (quicklisp-quickstart:install) you'll end up with Quicklisp installed
under /root/quicklisp/ and compiled files under /root/.cache/, which is not
what we want.

> I'm pretty sure that having something like /usr/share/cl-quicklisp/.cache
> goes against LSB and/or Debian policy; instead, I think that the expected
> place for that is something like /var/cache/cl-quicklisp. I feel it could be
> reasonable to ask Zack Beane to enable the customization of this path; we can
> see what he says.

No need to ask him yet.  It's probably not that difficult to work out ourselves
and I'm not sure it does go against LSB/Debian policy given that users of the
site-wide quicklisp libraries will never actually use these compiled files in
this site-wide scenario. (They will use their own compiled files under their
own ~/.cache directory).  In this case, they can be considered the by-product
of evaluating (quicklisp-quickstart:install) and discarded - we can even delete
the ~cl-quicklisp/.cache directory if necessary.

> Also, I'm noting that the "Common Lisp in Debian Manual" mandates lisp source
> files to be installed to /usr/share/common-lisp/source/destination-dir[3];
> should we consider using this path instead of /usr/share/cl-quicklisp? Not
> sure. Perhaps we can leave it as you have it right now (comments anyone?)

Personally I think we should make a clean break with the past and ditch the cl-
prefix completely.  Quicklisp changes everything and I think we should
introduce a new ql- prefix to reflect this fact.

That said, the language Common Lisp hasn't changed so I now propose the
following names for things:

 source package: quicklisp
 binary package: ql-quicklisp
    system user: ql-quicklisp
  ~ql-quicklisp: /usr/share/common-lisp/ql-quicklisp/

As I envisage it, ql-quicklisp will do little more than create the system user
ql-quicklisp and install Quicklisp under ~ql-quicklisp/.  Other ql-* packages
(e.g. ql-alexandria) will then depend on ql-quicklisp and consist primarily of
a postinst script which runs a lisp (as system user ql-quicklisp) and evalutes
the necessary (ql:quickload :package) form.  Quicklisp does the rest.

> Agreed. I'll need to look deeper into Quicklisp and CLC to clarify my ideas
> for the script. I'll see if I can research a little this weekend and post
> what I find.

The more I think about it, the more I think we should break with the past, use
Quicklisp to do as much as possible and keep things (in terms of our Debian
packages) as simple as possible.

Once the sources for the Quicklisp libraries are installed in a site-wide
location all we have to do is create symlinks to the various system definition
(.asd) files in a single directory, ~ql-quicklisp/installed-systems/ for
example, and publicise this directory in the docs.

Unprivileged users then simply have to add the line:

 (push #p"/usr/share/common-lisp/ql-quicklisp/installed-systems"
       asdf:*central-registry*)

to their init file for these systems to be ASDF:LOAD-SYSTEM enabled.

In short, ql-* packages use Quicklisp so the user doesn't have to.

> Well, working on a Debian package for Common Lisp is something I've been
> wanting to do. I'm very limited with the time I can put on this, but I think
> I can commit with 4 hours weekly, would that be ok?

Better than a kick in the teeth, as we optimistically say in the UK :)

Please find attached tarball containing two simple scripts and the Quicklisp
quickstart file.

The two ql-* scripts demonstrate what I have proposed our Debian ql-* packages
should basically do.

In order to test it yourself (and assuming you don't workas root):

 1. Unpack the tarball in your home directory.

 2. cd ~/ql-quicklisp

 3. sudo ql-quicklisp.sh

 4. sudo ql-alexandria.sh

 5. sbcl

    * (push #p"/usr/share/common-lisp/ql-quicklisp/installed-systems"
            asdf:*central-registry*)
    [...]
    * (asdf:load-system "alexandria")
    [...]

 6. Sit back and enjoy hot beverage of choice whilst contemplating the fact
    that you've just installed a quicklisp package site-wide and loaded it as
    an unprivileged user,

Of course, apart from the Debain packaging work, there's other stuff that needs
doing, like:

 - 'apt-get update' handling
 - a prerm script for 'apt-get remove ql-alexandria'

One thing you could do which would be very helpful, if you think this is a
workable idea, is confirm whether or not we are in fact violating any policy
requirements by taking this approach.

In the mean time, I will think about setting up a github organisation for all
the ql-* debian packages we will hopefully be churning out in the not too
distant future.

Sebastian

P.S. The attachment caused the message to require moderator approval (which
     will probably never happen) so I'm including my two files inline (below).
     You'll have to grab the Quicklisp quickstart file from
     http://beta.quicklisp.org/quicklisp.lisp and put in ~/ql-quicklisp/
     alongside the other two before you rin ~/ql-quicklisp/ql-quicklisp.sh.

~/ql-quicklisp/ql-quicklisp.sh
--8<---------------cut here---------------start------------->8---
#!/bin/sh
#
# Copyright © 2011 Sebastian Tennant <sdt at sebyte.me>
#
# [GPLv3 licence blurb goes here]

# create system user ql-quicklisp (suppressing the warning emitted if
# system user ql-quicklisp already exists)
adduser --quiet --system --home /usr/share/common-lisp/ql-quicklisp/ \
        --shell /bin/sh --disabled-login ql-quicklisp

[ -d ~ql-quicklisp ] || ( echo "Directory ~ql-quicklisp not found.  Aborting." && exit 1 )

# copy Quicklisp's quickstart file to ~ql-quicklisp
cp -f quicklisp.lisp ~ql-quicklisp/quicklisp-quickstart.lisp

# setup installed-systems files
touch ~ql-quicklisp/installed-systems.current
touch ~ql-quicklisp/installed-systems.new
touch ~ql-quicklisp/installed-systems.old

# setup installed-systems dir
mkdir -p ~ql-quicklisp/installed-systems

# enter ~ql-quicklisp
cd ~ql-quicklisp

# run sbcl as system user ql-quicklisp, load Quicklisp quickstart file
# and evaluate QUICKLISP-QUICKSTART:INSTALL to install Quicklisp
# proper
su -c '/usr/bin/sbcl --disable-ldb --lose-on-corruption --noinform \
                     --end-runtime-options \
                     --disable-debugger --no-sysinit --no-userinit --noprint \
                     --load quicklisp-quickstart.lisp \
                     --eval \(quicklisp-quickstart:install\) \
                     --quit > quicklisp-quickstart:install.log 2>&1' \
   ql-quicklisp

if [ $? -eq 0 ]; then
    echo "Site-wide Quicklisp installed under ~ql-quicklisp/quicklisp"
else
    echo "An error occurred.  See ~ql-quicklisp/quicklisp-quickstart:install.log for details"
fi;
--8<---------------cut here---------------end--------------->8---

~/ql-quicklisp/ql-alexandria.sh
--8<---------------cut here---------------start------------->8---
#!/bin/sh
#
# Copyright © 2011 Sebastian Tennant <sdt at sebyte.me>
#
# [GPLv3 licence blurb goes here]

[ -d ~ql-quicklisp ] || ( echo "Directory ~ql-quicklisp not found.  Aborting." && exit 1 )

# enter ~ql-quicklisp
cd ~ql-quicklisp

# run sbcl as system user ql-quicklisp, load Quicklisp quickstart file
# and evaluate QUICKLISP-QUICKSTART:INSTALL to install Quicklisp
# proper
su -c '/usr/bin/sbcl --disable-ldb --lose-on-corruption --noinform \
                     --end-runtime-options \
                     --disable-debugger --no-sysinit --no-userinit --noprint \
                     --load quicklisp/setup.lisp \
                     --eval \(quicklisp-client:quickload\ :alexandria\) \
                     --quit > quicklisp-client:quickload_alexandria.log 2>&1' \
   ql-quicklisp

[ $? -ne 0 ] && echo "An error occurred. See:

 ~ql-quicklisp/quicklisp:quickload_alexandria.log

for details."

# rotate installed systems files
rm -f installed-systems.new
mv -f installed-systems.current installed-systems.old
cat quicklisp/dists/quicklisp/installed/systems/*.txt | sort > installed-systems.current
grep -v -f installed-systems.old installed-systems.current > installed-systems.new

# create symbolic links to any new .asd files under installed-systems/ and
# report results
if [ -s installed-systems.new ]; then
    for system in $(cat installed-systems.new); do
	ln -s  ../quicklisp/$system installed-systems/
    done;
    echo "The following new ASDF systems are now available:

$(ls installed-systems)
"
else
    echo "alexandria is already installed.
"
fi;
--8<---------------cut here---------------end--------------->8---