diff options
author | Luke Shumaker <lukeshu@parabola.nu> | 2018-06-21 22:01:54 -0400 |
---|---|---|
committer | Luke Shumaker <lukeshu@parabola.nu> | 2018-07-08 21:06:05 -0400 |
commit | 128b6bbe8ec8900d06b50df0217b59d7e2aa2b98 (patch) | |
tree | 37a12b61a003309eafd0070c85aa2da1ce53fc12 | |
parent | 08995618eb606523b7d05ac987b1243981290839 (diff) |
Don't parse .db files ourselves; use pyalpm instead
In a patchset that I recently submitted, Eli was concerned that I was
parsing .db files with bsdtar+awk, when the format of .db files isn't
"public"; the only guarantees made about it are that libalpm can parse it.
https://lists.archlinux.org/pipermail/arch-projects/2018-June/004932.html
I wasn't too concerned, because `ftpdir-cleanup` and `sourceballs` already
parse the .db files in the same way. Nonetheless, I think Eli is right: we
shouldn't be parsing these files ourselves.
So, add a `dbquery` function that uses pyalpm to parse the .db files:
- It takes as arguments Python 3 expressions;
1. one that that returns a bool deciding whether we want to print
information on a package, and
2. another that returns the string to print for a package.
Currently, all callers use "True" for the decider expression, as
ftpdir-cleanup and sourceballs operate on *every* package. However, I'm
including a way to filter packages because, I'm coming at this from the
context that I want to parse .db files in other places too.
- libalpm doesn't offer an easy way to say "parse this DB file for me";
instead, we must construct a configuration that has a syncdb pointing to
that file, which we then have it sync in to a temporary directory.
As a final note, when re-writing the bit of sourceballs to use dbquery
instead of AWK, I realized that it does not correctly handle licenses that
have a space in them (as of 2018-07-07 there are 67 packages in the Arch
repos that have license containing a space). I did not fix this bug; I
merely translated it from AWK to Python, as the program would also need to
be adjusted elsewhere.
-rwxr-xr-x | cron-jobs/ftpdir-cleanup | 2 | ||||
-rwxr-xr-x | cron-jobs/sourceballs | 14 | ||||
-rw-r--r-- | db-functions | 25 | ||||
-rw-r--r-- | test/Dockerfile | 2 |
4 files changed, 29 insertions, 14 deletions
diff --git a/cron-jobs/ftpdir-cleanup b/cron-jobs/ftpdir-cleanup index 9df5f99..77e49c8 100755 --- a/cron-jobs/ftpdir-cleanup +++ b/cron-jobs/ftpdir-cleanup @@ -44,7 +44,7 @@ for repo in "${PKGREPOS[@]}"; do fi done | sort > "${WORKDIR}/repo-${repo}-${arch}" # get a list of package files defined in the repo db - bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" | awk '/^%FILENAME%/{getline;print}' | sort > "${WORKDIR}/db-${repo}-${arch}" + dbquery "$repo" "$arch" True pkg.filename | sort > "${WORKDIR}/db-${repo}-${arch}" missing_pkgs=($(comm -13 "${WORKDIR}/repo-${repo}-${arch}" "${WORKDIR}/db-${repo}-${arch}")) if (( ${#missing_pkgs[@]} >= 1 )); then diff --git a/cron-jobs/sourceballs b/cron-jobs/sourceballs index 6be28ab..784b48b 100755 --- a/cron-jobs/sourceballs +++ b/cron-jobs/sourceballs @@ -24,18 +24,8 @@ for repo in "${PKGREPOS[@]}"; do if [[ ! -f ${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT} ]]; then continue fi - bsdtar -xOf "${FTP_BASE}/${repo}/os/${arch}/${repo}${DBEXT}" \ - | awk '/^%NAME%/ { getline b }; - /^%BASE%/ { getline b }; - /^%VERSION%/ { getline v }; - /^%LICENSE%/,/^$/ { - if ( !/^%LICENSE%/ ) { l=l" "$0 } - }; - /^%ARCH%/ { - getline a; - printf "%s %s %s %s\n", b, v, a, l; - l=""; - }' + dbquery "$repo" "$arch" True \ + 'f"{pkg.base or pkg.name} {pkg.version} {pkg.arch} {'\'' '\''.join(pkg.licenses)}"' done | sort -u > "${WORKDIR}/db-${repo}" done diff --git a/db-functions b/db-functions index 0491c22..f1d821a 100644 --- a/db-functions +++ b/db-functions @@ -294,6 +294,31 @@ getpkgfiles() { echo "${files[@]}" } +# usage: dbquery repo arch filter_expr output_expr +dbquery() { + local repo=$1 + local arch=$2 + local filter=$3 + local output=$4 + local dbfile="${FTP_BASE}/${repo}/os/${arch}/${repo}.db" + + python3 - "$dbfile" "$filter" "$output" <<-'EOT' + import os.path + import sys + import tempfile + import pyalpm + db_dir, db_file = os.path.split(os.path.abspath(sys.argv[1])) + with tempfile.TemporaryDirectory() as tmpdirname: + handle = pyalpm.Handle(tmpdirname, tmpdirname) + db = handle.register_syncdb(db_file[:-3], 0) + db.servers = ["file://{}".format(db_dir)] + db.update(False) + for pkg in db.search(".*"): + if eval(sys.argv[2], {}, {"pkg": pkg}): + print(eval(sys.argv[3], {}, {"pkg": pkg})) + EOT +} + check_pkgfile() { local pkgfile=$1 diff --git a/test/Dockerfile b/test/Dockerfile index 83c8449..0d01a75 100644 --- a/test/Dockerfile +++ b/test/Dockerfile @@ -1,5 +1,5 @@ FROM archlinux/base -RUN pacman -Syu --noconfirm --needed sudo fakeroot awk subversion make kcov bash-bats gettext grep +RUN pacman -Syu --noconfirm --needed sudo fakeroot awk subversion make kcov bash-bats gettext grep pyalpm RUN pacman-key --init RUN echo '%wheel ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/wheel RUN useradd -N -g users -G wheel -d /build -m tester |