[Collab-qa-commits] r1044 - udd/src/udd

[he at alioth.debian.org]

Author: he
Date: 2008-08-09 20:34:32 +0000 (Sat, 09 Aug 2008)
New Revision: 1044

Modified:
   udd/src/udd/carnivore_gatherer.py
Log:
Use sets in the carnivore gatherer to get rid of dupes in the data before
pushing them to the DB


Modified: udd/src/udd/carnivore_gatherer.py
===================================================================
--- udd/src/udd/carnivore_gatherer.py	2008-08-09 20:00:50 UTC (rev 1043)
+++ udd/src/udd/carnivore_gatherer.py	2008-08-09 20:34:32 UTC (rev 1044)
@@ -102,11 +102,11 @@
             match = re.compile('(\w+)@debian.org').search(content)
             record[info["name"]] = match.group(1) 
           elif info["content-type"] == "comma-separated":
-            record[info["name"]] = content.rstrip().split(", ")
+            record[info["name"]] = set(content.rstrip().split(", "))
           elif info["content-type"] == "multiple entries":
             if info["name"] not in record:
-              record[info["name"]] = []
-            record[info["name"]].append(content.rstrip())
+              record[info["name"]] = set()
+            record[info["name"]].add(content.rstrip())
 
 if __name__ == '__main__':
   main()