[Reproducible-commits] [misc] 01/01: reports: add a bin/get-latest-data to reduce manual work
Ximin Luo
infinity0 at debian.org
Mon May 16 14:33:16 UTC 2016
This is an automated email from the git hooks/post-receive script.
infinity0 pushed a commit to branch master
in repository misc.
commit 96e88ae811bcafa8f7584fa9cf41c297e95d1219
Author: Ximin Luo <infinity0 at debian.org>
Date: Mon May 16 16:32:32 2016 +0200
reports: add a bin/get-latest-data to reduce manual work
---
.gitignore | 1 -
reports/.gitignore | 2 ++
reports/README | 19 +++++-------------
reports/bin/get-latest-data | 47 +++++++++++++++++++++++++++++++++++++++++++++
reports/bin/review-bugs | 19 ------------------
reports/bugs | 1 +
reports/changelogs | 1 +
reports/reproducible.db | 1 +
8 files changed, 57 insertions(+), 34 deletions(-)
diff --git a/.gitignore b/.gitignore
index c4d5353..f1fe79e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -6,5 +6,4 @@ rbuilds
.idea
*.pyc
*.iml
-reproducible.db
reproducible.json
diff --git a/reports/.gitignore b/reports/.gitignore
new file mode 100644
index 0000000..c33f270
--- /dev/null
+++ b/reports/.gitignore
@@ -0,0 +1,2 @@
+/data
+/latest
diff --git a/reports/README b/reports/README
index 6e4a1d7..beab28d 100644
--- a/reports/README
+++ b/reports/README
@@ -15,13 +15,15 @@ Reports are from Sunday 00:00 UTC to Sunday 23:59 UTC.
Process
-------
-1. Look at packages that became reproducible this week.
+0. Get the previous week's data:
+
+ $ bin/get-latest-data
+
+1. Look at packages that became reproducible in that week.
There's a script that will use the history in the
reproducible.debian.net database to figure this out. Usage:
- $ rm -f reproducible.db # remove previously downloaded database
- $ wget https://reproducible.debian.net/reproducible.db
$ bin/newly-reproducible -w 77 # for week 77 of the reports…
The output needs to be manually verified. Some packages will
@@ -30,14 +32,6 @@ Process
might give some false positive: packages might have been
reproducible before, broken by a toolchain upload later fixed.
- To download all the changelogs, one can do:
-
- $ mkdir changelogs
- $ for url in $(bin/newly-reproducible | sed -n -e 's,.*<,,;s,>.*,,p' | sort -u); do
- p=${url%/*}; p=${p##*/};
- torsocks wget -O changelogs/$p $url;
- done
-
One can look at the history of the tests by typing:
$ bin/history ghc # defaults to unstable
@@ -49,9 +43,6 @@ Process
updated in the past 7 days, then use the `bts` tool to cache
them, and finally display them using `mutt`.
- Usage:
-
- $ rm -f bugs # remove previously generated bug list
$ bin/review-bugs -w 77 # for week 77 of the reports
3. Look at all uploads for the past week.
diff --git a/reports/bin/get-latest-data b/reports/bin/get-latest-data
new file mode 100755
index 0000000..03c6cf8
--- /dev/null
+++ b/reports/bin/get-latest-data
@@ -0,0 +1,47 @@
+#!/bin/sh
+set -e
+
+scriptdir="$(readlink -f "$(dirname "$0")")"
+week_1_end=1430604000 # May 3 2015, 00:00, Sunday
+
+mkdir -p data
+now="$(date +%s)"
+
+prev_week=$(((now - week_1_end) / (7*24*3600) + 1))
+
+while getopts 'w:' opt; do
+ case $opt in
+ w)
+ week=$OPTARG
+ ;;
+ esac
+done
+
+week="${week:-$prev_week}"
+week_end=$((week_1_end + (week - 1)*7*24*3600))
+week_start=$((week_end - 7*24*3600))
+week_path="week_${week}_ending_$(date -d@${week_end} +%Y-%m-%d)"
+
+echo >&2 "Getting data for the period $(date -d@${week_start} +%Y-%m-%d) to $(date -d@${week_end} +%Y-%m-%d)"
+
+#rm -rf "data/$week_path"
+mkdir -p "data/$week_path"
+(cd "data/$week_path"
+
+# Get latest db
+wget -q -N https://reproducible.debian.net/reproducible.db
+
+# Get latest changelogs
+mkdir -p changelogs
+for url in $("$scriptdir/newly-reproducible" -w "$week" | sed -n -e 's,.*<,,;s,>.*,,p' | sort -u); do
+ p=${url%/*}; p=${p##*/};
+ torsocks wget -q -O changelogs/$p $url || true
+done
+
+# Get latest bugs
+echo "SELECT DISTINCT bugs.id FROM bugs_usertags, bugs WHERE email = 'reproducible-builds at lists.alioth.debian.org' AND bugs.id = bugs_usertags.id AND bugs.last_modified > to_timestamp($week_start) AND bugs.last_modified < to_timestamp($week_end) ORDER BY bugs.id;" \
+ | ssh alioth.debian.org psql service=udd -t \
+ | awk '/[0-9]/ { print $1 }' > bugs
+)
+
+ln -sf "data/$week_path" latest
diff --git a/reports/bin/review-bugs b/reports/bin/review-bugs
index 228bb6c..2df73f6 100755
--- a/reports/bin/review-bugs
+++ b/reports/bin/review-bugs
@@ -4,25 +4,6 @@
# Copyright © 2015 Lunar <lunar at debian.org>
# Licensed under WTFPL — http://www.wtfpl.net/txt/copying/
-
-week_1=1430604000 # May 3 2015, 00:00
-
-while getopts 'w:' opt; do
- case $opt in
- w)
- week=$OPTARG
- ;;
- esac
-done
-
-query_date_end=$(date +%s)
-[ -n "$week" ] && query_date_end=$((week_1 + (week - 1)*7*24*3600))
-query_date_begin=$((query_date_end - 7*24*3600))
-
-if ! [ -f bugs ]; then
- echo "SELECT DISTINCT bugs.id FROM bugs_usertags, bugs WHERE email = 'reproducible-builds at lists.alioth.debian.org' AND bugs.id = bugs_usertags.id AND bugs.last_modified > to_timestamp($query_date_begin) AND bugs.last_modified < to_timestamp($query_date_end) ORDER BY bugs.id;" | ssh alioth.debian.org psql service=udd -t | awk '/[0-9]/ { print $1 }' > bugs
-fi
-
TOTAL="$(wc -l bugs | awk '{ print $1 }')"
for bug in $(cat bugs); do
diff --git a/reports/bugs b/reports/bugs
new file mode 120000
index 0000000..9116836
--- /dev/null
+++ b/reports/bugs
@@ -0,0 +1 @@
+latest/bugs
\ No newline at end of file
diff --git a/reports/changelogs b/reports/changelogs
new file mode 120000
index 0000000..24a5e61
--- /dev/null
+++ b/reports/changelogs
@@ -0,0 +1 @@
+latest/changelogs
\ No newline at end of file
diff --git a/reports/reproducible.db b/reports/reproducible.db
new file mode 120000
index 0000000..d05ea18
--- /dev/null
+++ b/reports/reproducible.db
@@ -0,0 +1 @@
+latest/reproducible.db
\ No newline at end of file
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/reproducible/misc.git
More information about the Reproducible-commits
mailing list