| Age | Commit message (Collapse) | Author |
|---|
|
testing2x needs to:
1. Verify that the new version of the package is found in TESTING_REPO
2. Determine which of STABLE_REPOS the old version of the package is in
Currently, to do those things, it checks for the existence of PKGBUILD
files in SVN. That information is already stored in the DBEXT files; get
it from there, instead of talking to SVN.
"Ignore space change" might be useful when viewing this diff.
|
|
This should have the same result, but be a bit less confusing to read.
Also, have a slightly more specific error message on a missing PKGBUILD.
|
|
Don't try to parse PKGBUILD files from SVN; all of the information we need
is already in the DBEXT files. Several programs use [[ -f PKGBUILD ]] or
[[ -r PKGBUILD ]] on files from SVN; those checks can stay, just remove all
instances of actually trying to *parse* those files.
As an exception, don't modify parse_pkgbuilds.sh (which is called by
check_packages.py, which is called by cron-jobs/integrity-check).
Most of the attributes we need have been present in the DBEXT files for as
long as `repo-add` has been part of pacman:
attribute | git | git date | ver | ver date
----------+----------+------------+-------+-----------
FILENAME | aa1c0ba9 | 2006-11-20 | 3.0.0 | 2007-03-25
NAME | aa1c0ba9 | 2006-11-20 | 3.0.0 | 2007-03-25
VERSION | aa1c0ba9 | 2006-11-20 | 3.0.0 | 2007-03-25
ARCH | aa1c0ba9 | 2006-11-20 | 3.0.0 | 2007-03-25
BASE | 4b21504f | 2009-07-22 | 3.3.0 | 2009-08-02
However, `BASE` for split packages is slightly newer, and its presence
relies not only on the `repo-add` version, but also the `makepkg` version
used to build the package.
As of 2018-06-15, the oldest package in any of the Arch Linux repos is
community/gimp-refocus, built on 2013-07-22; I don't believe it is
necessary to handle packages from before that change was made (before
`BASE` was set).
"Ignore space change" might be useful when viewing this diff.
|
|
|
|
In a patchset that I recently submitted, Eli was concerned that I was
parsing .db files with bsdtar+awk, when the format of .db files isn't
"public"; the only guarantees made about it are that libalpm can parse it.
https://lists.archlinux.org/pipermail/arch-projects/2018-June/004932.html
I wasn't too concerned, because `ftpdir-cleanup` and `sourceballs` already
parse the .db files in the same way. Nonetheless, I think Eli is right: we
shouldn't be parsing these files ourselves.
So, add a `dbquery` function that uses pyalpm to parse the .db files:
- It takes as arguments Python 3 expressions;
1. one that that returns a bool deciding whether we want to print
information on a package, and
2. another that returns the string to print for a package.
Currently, all callers use "True" for the decider expression, as
ftpdir-cleanup and sourceballs operate on *every* package. However, I'm
including a way to filter packages because, I'm coming at this from the
context that I want to parse .db files in other places too.
- libalpm doesn't offer an easy way to say "parse this DB file for me";
instead, we must construct a configuration that has a syncdb pointing to
that file, which we then have it sync in to a temporary directory.
As a final note, when re-writing the bit of sourceballs to use dbquery
instead of AWK, I realized that it does not correctly handle licenses that
have a space in them (as of 2018-07-07 there are 67 packages in the Arch
repos that have license containing a space). I did not fix this bug; I
merely translated it from AWK to Python, as the program would also need to
be adjusted elsewhere.
|
|
- ftpdir-cleanup: ${pkg[0]} -> ${pkgs[0]}
This mistake was introduced in 7628525156110022fa70ad91e4bc13ee8a3cceb0
- test: common.bash: $pkgarches[@] -> ${pkgarches[@]}
This mistake was introduced in 7628525156110022fa70ad91e4bc13ee8a3cceb0
|
|
|
|
Use arrays via nameref, since makepkg does not support multidimensional
arrays, and assigning to/retrieving from an array variable with an
unknown name requires this.
Requires bash 4.3, requires that architectures never contain chars that
are invalid in a variable name -- which makepkg explicitly requires in
order to support architecture-dependent sources/dependencies, and also
incorporates into linting checks which forbid anything but [[:alnum:]_]
so that's all good.
|
|
pacman 5.1 enforces this restriction. OTOH it is a simpler setup to set
this as the user homedir directly in account creation (just like
makechrootpkg has always done) than to create an additional,
world-writable, directory.
dockerfile: don't use tmpfs for /build
|
|
The print_all_package_names function changed its behavior, adapt
|
|
|
|
This does correct handling of
- executing a program by symlink
- any weird characters in the full path
- I'm sure there's another case I thought about when I originally did
this.
|
|
Search for unquoted variables using the command:
grep -Prn --exclude-dir=.git '(?<!["=]|\[\[ |\[\[ -[zn] )\$(?!{?#|\(|\? )'
and ignore a bunch of false positives.
You may verify that the only differences between the lines removed and
lines added are double-quotes:
diff -u \
<(git show|sed -n 's/^-//p'|grep -v '^-- a/'|sed 's/"//g') \
<(git show|sed -n 's/^+//p'|grep -v '^++ b/'|sed 's/"//g')
|
|
transaction
This is a leftover change from the patch that became 0432cff; v2 of that
patch broke db-functions:getpkgfiles, but the testsuite didn't catch it.
This patch adds a testcase that catches this type of breakage.
|
|
|
|
|
|
- db-functions: getpkgfile: The .sig file extension should be part of
the filename parameter, not part of the message format string.
- db-functions: arch_repo_modify: Shouldn't use ${action}
string-interpolation in the message format string. Since the
entire message is a command, and we're using @Q to escape arguments
anyway, go ahead and just construct the entire command as a single
string that way, and feed it to '%s'.
|
|
Using the following command to find unquoted variables (and ignoring
more than a a few false positives),
grep -Prn --exclude-dir=.git '(?<!["=]|\[\[ |\[\[ -[zn] )\$(?!{?#|\(|\? )'
one is lead to find a few cleanups that are something other than "add
double-quotes". That's what these are. We'll leave dumb adding of
double-quotes for another commit.
Most of these are still fixing quoting issues, just with a better fix.
- parse_pkgbuilds.sh: Avoid having to escape quotes in `eval` strings
by using `declare -p`. Updates the logic copied from makepkg, with the
latest logic copied from makepkg. See
https://git.archlinux.org/pacman.git/commit/?id=9e52a36794552b77ecf26f7f34b226d096978f1e
- sourceballs: Avoid using ary=($string) to do field separation by
using `read` and test that multiple licenses actually work as
expected.
- sourceballs: Replace `[[ -z ${ary[*]} ]]` with test for the array
length
- db-functions: Replace mangling echo field separators using sed, with
printf formatters
- db-functions: Replace for/echo loop to print an array line by line,
with `printf '%s\n'`
- db-functions: set_repo_permissions: Line up error messages, quote
"$group"
- db-move: Replace `$(echo ${array[@]})` with `${array[*]}`
- testing2x: Use `"$@"` instead of `$*` when looping over an array
Also, not really quoting related but on the same line as a quoting
issue, optimize:
- db-functions: Replace
[[ -n "$(... | sort | uniq -D)" ]]
with
! ... | awk 'a[$0]++{exit 1}'
|
|
Instead, compare the value of $v to 'true'.
|
|
(By default, prefer `grep &>/dev/null`)
`grep -q` may exit as soon as it finds a match; this is a good optimization
for when the input is a file. However, if the input is the output of
another program, then that other program will receive SIGPIPE, and further
writes will fail. When this happens, it might (bsdtar does) print a
message about a "write error" to stderr. Which is going to confuse and
alarm the user.
In one of the cases (in common.bash, in the test suite), this had
already been mitigated by wrapping bsdtar in "echo "$(bsdtar ...)", as
Bash builtin echo doesn't complain if it gets SIGPIPE. However, that
means we're storing the entire output of bsdtar in memory, which is
silly. Additionally, the way it was implemented is also wrong;
because it was being used with `grep -qv` instead of just `grep -q`,
it *always* found a non-matching line (even something inconsequential
like `%NAME%`), and *never* triggered a test failure.
Looking at a few of these cases, it might also make sense to switch to
using `bsdtar tf` instead of `bsdtar xf` when checking membership, but
that's work for another day.
|
|
It is a method of notifying text-editors that a file is in Bash syntax
without giving it a propper shebang (which would be confusing, as it
would suggest that the file should be executable), as well as working
across virtually all text-editors (unlike "-*- Mode: Bash -*-" or
whatever).
|
|
|
|
|
|
pkgs should be passed as a single argument
|
|
- ftpdir-cleanup.bats: Glob expansion does not occur in [[ -f ]] tests.
The [[ ! -f .../${pkgname}-*${PKGEXT} ]] checks were checking that there
were no files containing a literal '*' for that part of their name.
Obviously, this isn't what was intended.
- sourceballs.bats: [ -r ] checks explode if the glob returns >1 file.
Avoid using them if the path being checked contains a glob.
|
|
I managed to stumble across a bug in BATS where the run() function
screwed with the global IFS. The bug has been fixed in git, but isn't
in a release yet.
https://github.com/sstephenson/bats/issues/89
Anyway, this bug breaks __getCheckSum(). Fortunately, we have avoided
tripping it so far because luck has it that we never call
__getCheckSum() after run() in the same test.
So, there's nothing actually broken here, but it makes me nervous. So
go ahead and modify __getCheckSum to not rely on IFS.
And, while we're at it: declare the result variable and set it as
separate commands. Doing both in the same command masks the exit code
of the subshell expansion. We don't explicitly check the exit code,
but BATS runs the test suite with `set -e`, so splitting it does mean
that BATS will now detect errors from sha1sum. We don't really expect
that to happen, but if BATS will give us error checking on it for
free, why not?
|
|
These are things that were (IMO) missed in 5afac1e. I found them using:
git grep -E '(plain|msg|msg2|warning|error|die) "[^"]*\$'
I went a little above-and-beyond for escaping strings for the error
messages in db-functions' arch_repo_add and arch_repo_remove. The
code should explain itself, but I wanted to point it out, as it's more than
the usual "slap %s in there, and move the ${...} to the right".
|
|
Use less duplicated logic, opening the way to more elegantly change
the interface for the repo-add/repo-remove scripts which are themselves
symlinked together and mostly have the same interface.
|
|
The current glob `*.pkg.tar.?z` is both less restrictive and more
restrictive than makepkg, as it accepts any valid unicode character.
To be more exact, it's almost completely orthogonal to the one in makepkg.
makepkg only accepts .tar.gz, .tar.bz2, .tar.xz, .tar.lzo, .tar.lrz, and
.tar.Z and most of those fail to match against a two-char compression type.
dbscripts accepts .pkg.tar.đź’©z which incidentally is what I think of
cherry-picking xz and gz as supported methods.
Since this can be anything makepkg.conf accepts, it needs to be able to
match all that, unless we decide to perform additional restrictions in
which case we should still explicitly list each allowed extension. Using
bash extended globbing allows us to do this relatively painlessly.
Document the fact that this has *always* been some sort of glob, and
update the two cases where this was (not!) being evaluated by bash
[[ ... ]], to use a not-elegant-at-all proxy function is_globfile() to
evaluate globs *before* testing if they exist.
|
|
This fully removes the use of find from the codebase, leads to a
micro-optimization in a couple cases, and ensures that $PKGEXT is
consistently treated as a shell globbing character (which is important
because it is used as one).
Of the eight instances in these files:
- One was unnecessary as `cat` can natively consume all files passed to
it and no directory traversal was in use.
- Two were unnecessary as they were hardcoded to read a single file....
- Another four were only being used to strip leading directory paths,
and can be replaced by globstar and ${filepath##*/}
- The final two were checking the modification time of the files, and
can be replaced with touch(1) and [[ -nt ]]. Although this introduces
an additional temporary file, this is not such a big deal.
|
|
Don't bother emitting errors. bash doesn't show globbing errors if it
cannot read a directory to try globbing there. And the former code never
aborted on errors anyway, as without `set -o pipefail` the sort command
swallowed the return code.
|
|
Commit b61a714 introduced a regression that in a db-functions function
called by db-update, config:PKGEXT is treated like a fixed string, instead
of as a glob. Because of inadequacy of the test-suite, this did not cause
a test failure.
https://lists.archlinux.org/pipermail/arch-projects/2018-February/004742.html
The broken function checks if the repo already contains a package that
matches the one being released. This did not trigger a test-suite failure
because right after the broken check there is another check that checks for
the exact filename of the new package file. In the test-suite, all
package files have the same extension, so the latter check always "saved"
us.
So, modify a relevant test case (a test case that verifies that we can't
release the same package twice) such that the second time we release the
package, we set PKGEXT to something different (.gz, instead of the normal
.xz). This way, we can be sure that when db-update rejects the operation,
it's because of the glob check, not the exact-filename check.
This is made slightly tricky because the __buildPackage() test helper
routine tries to cache builds, to speed up the tests. Without
modification, this means that it will just ignore the new PKGEXT. So,
modify it to only use the cache if it can successfully copy build artifacts
out, in a way that respects PKGEXT.
|
|
PKGEXT is a makepkg variable referring to a fixed filename suffix, but
we were also using it to mean a bash glob referring to candidate
filenames. This is wrong, so rename it to PKGEXTS which is more
descriptive of its purpose.
Exclude the testsuite from this change, as the testsuite actually uses
PKGEXT for its intended purpose. Fix the testsuite to consistently use
PKGEXT, as it hardcoded the file extension in several cases, and pin
its value to .pkg.tar.xz
|
|
Catch some cases that were missed in the previous run.
|
|
|
|
|
|
|
|
We dropped it long ago and this should be reflected here rather than
being changed on our servers only.
Signed-off-by: Florian Pritz <bluewind@xinu.at>
|
|
reuse getpkgfile() in getpkgfiles()
|
|
Fix numerous instances of POSIX `[ ... ]`, including reliance on ugly
deprecated constructs like POSIX `-a`. Since we require bash regardless,
it makes sense to take full advantage of it.
bash `[[ ... ]]` does not require quoting variables as the shell
natively recognizes them as variables rather than expanded strings.
Use shell arithmetic rather than test, when checking numerical values.
|
|
Rather than using ugly hacks like arr[${#arr[*]}]="foo", bash 3.1 has
the += operator.
Update strings to use the same operator while we are at it.
|
|
This meant only the first array element was checked in
check_repo_permission(). Although arguably this should never cause real
issues as something else would have to be broken if multiple
architectures have different permissions, we should catch this now
anyway.
|
|
libmakepkg messaging functions provide automatic access to gettext
(which we do not currently make use of) in addition to cleanly
separating data from message strings.
In order for this to work properly, pass argv correctly from die() to
libmakepkg's error()
|
|
Rather than doing a test(1) on the numerical value of the return code,
use the `if !` and `||` operators during control flow.
|
|
|
|
|
|
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
|
|
|
|
|
|
|
