[Popcon-developers] Debconf ideas

[Enrico Zini]

Hello,

I post here a small list of useful things to do collected during various
discussions at DebConf:

 - patch ld.so.cache to add an option to preserve the atimes of the
   libraries during indexing, and use it on the nightly cron job.  Like
   updatedb does.
 - when scanning a .py file, try to scan a .pyc file as well
 - allow a package to install a /usr/share/popcon/pkgname file that
   contains a list of files to watch in case they are found in
   nonstandard locations.  This would give packages such as web
   applications or openclipart a chance to participate.
 - I would like to process incoming votes as well.  A simple way to do
   that is to notify me when a vote enters the archive and when a vote
   is deleted from the archive.
   I wrote a script that allows you to do that easily (please find it
   attached to this mail): it implements a queue to which a user can
   write and other users can read, without missing any item and without
   reading items twice.  I can simply get notifications of new entries
   and deleted entries through this queue.


Ciao,

Enrico

-- 
GPG key: 1024D/797EBFAB 2000-12-05 Enrico Zini <enrico at debian.org>
-------------- next part --------------
#!/bin/sh -ue

# Copyright (C) 2007  Enrico Zini <enrico at debian.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

# How it works:
#
# * This script implements a queue to which lines can be written and read.
# * Lines are grouped together in files that change name every
#   {second/minute/hour, as configured}.
# * Files stay in the queue for a minimum of a given amount of time.
# * The currently appended file cannot be read.  This means that if entries are
#   grouped on one file per hour, only data older than at most one hour can be
#   read.
# * Readers use a 'bookmark file' to mark the last queue file read, and this
#   allows to always read new data, not miss data and not read data twice

help() {
	cat <<EOT
Usage: $0 cmd args...

Implements a queue that can be appended and accessed by many programs at the
same time.

Commands are:
	help
		print this help message

	init dir 'timestep' 'lifetime'
		set up a queue in the given directory, aggregating entries on a
		'timestep' basis.  'timestep' can be 'second', 'minute' or
		'hour'.  Keep in mind that only entries from up to 2 timesteps
		ago can be read from the queue.

		'lifetime' tells the minimum time data will stay in the queue.

		'lifetime' will be passed to 'date' and can have values like "3
		days" or "2 hours".

	add dir "line"
		adds the given line to the queue in the given directory
	
	read dir bookmarkfile
		read lines from the given directory to standard output,
		updating the bookmark in the given file

	cleanup dir
		clean elements in the queue older than the queue lifetime.
EOT
}

# Read the queue configuration into environment variables.
# After this is called, we will have:
#  TIMEFMT: the strftime string used to generate the queue file names
#  TIMESTEP: the configured time step for files in the queue
#  LIFETIME: the minimum lifetime of data in the queue
config() {
	case $1 in
		second)
			TIMEFMT="%Y%m%d-%H%M%S"
			;;
		minute)
			TIMEFMT="%Y%m%d-%H%M"
			;;
		hour)
			TIMEFMT="%Y%m%d-%H"
			;;
		*)
			echo "Unknown timestep: $1" >&2
			exit 1
			;;
	esac
}

# Read the queue configuration
readconf() {
	if [ ! -f "$DIR/conf" ]
	then
		echo "There is no queue in $DIR: should you use 'init' first?" >&2
		exit 1
	fi
	. "$DIR/conf"
	config "$TIMESTEP"
}

# Create a new queue
queueInit() {
	DIR="$1"
	TIMESTEP="$2"
	LIFETIME="$3"
	if [ -f "$DIR/conf" ]
	then
		echo "A queue already exists in $DIR" >&2
		exit 1
	fi
	config "$TIMESTEP"
	mkdir "$DIR"
	echo "TIMESTEP='$TIMESTEP'" > "$DIR/conf"
	echo "LIFETIME='$LIFETIME'" >> "$DIR/conf"
}

# Add one line to the queue
queueAdd() {
	DIR="$1"
	LINE="$2"
	readconf
	echo "$LINE" >> "$DIR/`date +\"$TIMEFMT\"`"
}

# Read items that have been written from the queue after the last time we read
# it
queueRead() {
	DIR="$1"
	readconf
	BOOKMARK="$2"
	if [ -f "$BOOKMARK" ]
	then
		FIRST="`cat \"$BOOKMARK\"`"
	else
		FIRST="`date +\"$TIMEFMT\" -d '@0'`"
	fi
	LAST="`date +\"$TIMEFMT\" -d \"1 $TIMESTEP ago\"`"

	# We need a bit of perl to do string comparisons, because test would
	# not do them
	find "$DIR" -maxdepth 1 -type f -printf '%f\n' | sort | perl -ne "next if \$_ le \"$FIRST\"; next if \$_ gt \"$LAST\"; print" | while read NAME
	do
		cat "$DIR/$NAME"
	done
	echo "$LAST" > "$BOOKMARK"
}

# Cleanup items from the queue for which the lifetime has expired
queueCleanup() {
	DIR="$1"
	readconf
	FIRST="`date +\"$TIMEFMT\" -d \"$LIFETIME ago\"`"
	# We need a bit of perl to do string comparisons, because test would
	# not do them
	find "$DIR" -maxdepth 1 -type f -printf '%f\n' | sort | perl -ne "next if \$_ gt \"$FIRST\"; print" | while read NAME
	do
		rm "$DIR/$NAME"
	done
}

if [ $# -eq 0 ]
then
	CMD=help
else
	CMD="$1"
	shift
fi

case $CMD in
	init)
		if [ $# -lt 3 ]
		then
			help
			exit 1
		fi
		queueInit "$@"
		;;
	add)
		if [ $# -lt 2 ]
		then
			help
			exit 1
		fi
		queueAdd "$@"
		;;
	read)
		if [ $# -lt 2 ]
		then
			help
			exit 1
		fi
		queueRead "$@"
		;;
	cleanup)
		if [ $# -lt 1 ]
		then
			help
			exit 1
		fi
		queueCleanup "$@"
		;;
	help)
		help
		;;
	*)
		echo "Unknown option $CMD" >&1
		exit 1
esac
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.alioth.debian.org/pipermail/popcon-developers/attachments/20070618/6f7b4d41/attachment.pgp