[Collab-qa-commits] r1733 - in udd: . scripts udd
Charles Plessy
plessy at alioth.debian.org
Sat Apr 10 01:38:35 UTC 2010
Author: plessy
Date: 2010-04-10 01:38:35 +0000 (Sat, 10 Apr 2010)
New Revision: 1733
Added:
udd/scripts/fetch_bibref.sh
udd/udd/bibref_gatherer.py
Modified:
udd/config-org.yaml
Log:
Bibliographic reference gatherer, untested and still in development.
Somewhat discussed in the following thread and others:
http://lists.debian.org/msgid-search/20100206130204.GA27756@kunpuu.plessy.org
Modified: udd/config-org.yaml
===================================================================
--- udd/config-org.yaml 2010-03-27 09:00:41 UTC (rev 1732)
+++ udd/config-org.yaml 2010-04-10 01:38:35 UTC (rev 1733)
@@ -19,6 +19,7 @@
ddtp: module udd.ddtp_gatherer
ftpnew: module udd.ftpnew_gatherer
screenshots: module udd.screenshot_gatherer
+ bibref: module udd.bibref_gatherer
dehs: module udd.dehs_gatherer
ldap: module udd.ldap_gatherer
wannabuild: module udd.wannabuild_gatherer
@@ -557,6 +558,14 @@
table: screenshots
screenshots_json: /org/udd.debian.org/mirrors/screenshots/screenshots.json
+bibref:
+ type: bibref
+ update-command: /org/udd.debian.org/udd/scripts/fetch_bibref.sh
+ path: /org/udd.debian.org/mirrors/bibref
+ cache: /org/udd.debian.org/mirrors/cache
+ table: bibref
+ bibref_yaml: /org/udd.debian.org/mirrors/bibref/bibref.yaml
+
wannabuild:
type: wannabuild
wbdb: "dbname=wanna-build host=localhost port=5433 user=guest"
Added: udd/scripts/fetch_bibref.sh
===================================================================
--- udd/scripts/fetch_bibref.sh (rev 0)
+++ udd/scripts/fetch_bibref.sh 2010-04-10 01:38:35 UTC (rev 1733)
@@ -0,0 +1,5 @@
+#!/bin/sh
+TARGETDIR=/org/udd.debian.org/mirrors/bibref
+mkdir -p $TARGETDIR
+rm -rf $TARGETDIR/*
+wget -q http://upstream-metadata.debian.net/for_UDD/biblio.yaml -O ${TARGETDIR}/bibref.yaml
Property changes on: udd/scripts/fetch_bibref.sh
___________________________________________________________________
Added: svn:executable
+ *
Copied: udd/udd/bibref_gatherer.py (from rev 1732, udd/udd/screenshot_gatherer.py)
===================================================================
--- udd/udd/bibref_gatherer.py (rev 0)
+++ udd/udd/bibref_gatherer.py 2010-04-10 01:38:35 UTC (rev 1733)
@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+
+"""
+This script imports bibliographic references from upstream-metadata.debian.net.
+"""
+
+from gatherer import gatherer
+from sys import stderr, exit
+
+online=0
+
+def get_gatherer(connection, config, source):
+ return bibref_gatherer(connection, config, source)
+
+class screenshot_gatherer(gatherer):
+ """
+ Bibliographic references from upstream-metadata.debian.net.
+ """
+
+ def __init__(self, connection, config, source):
+ gatherer.__init__(self, connection, config, source)
+ self.assert_my_config('table')
+ my_config = self.my_config
+
+ cur = self.cursor()
+ query = "DELETE FROM %s" % my_config['table']
+ cur.execute(query)
+ query = """PREPARE bibref_insert (text, text, text) AS
+ INSERT INTO %s
+ (package, key, value)
+ VALUES ($1, $2, $3)""" % (my_config['table'])
+ cur.execute(query)
+
+ pkg = None
+
+ def run(self):
+ my_config = self.my_config
+ #start harassing the DB, preparing the final inserts and making place
+ #for the new data:
+ cur = self.cursor()
+
+ bibref_file = my_config['bibref_yaml']
+ fp = open(bibref_file, 'r')
+ result = fp.read()
+ fp.close()
+
+ for res in safe_load_all(result):
+ package, key, value = res
+ query = """EXECUTE bibref_insert
+ (%(package)s, %(key)s, %(value)s)"""
+ try:
+ cur.execute(query, res)
+ except UnicodeEncodeError, err:
+ print >>stderr, "Unable to inject data for package %s. %s" % (res['name'], err)
+ print >>stderr, "-->", res
+ cur.execute("DEALLOCATE bibref_insert")
+ cur.execute("ANALYZE %s" % my_config['table'])
+
+if __name__ == '__main__':
+ main()
+
+# vim:set et tabstop=2:
+
Property changes on: udd/udd/bibref_gatherer.py
___________________________________________________________________
Added: svn:mergeinfo
+
More information about the Collab-qa-commits
mailing list