diff options
Diffstat (limited to 'HACKING')
| -rw-r--r-- | HACKING/build-system.md | 4 | ||||
| -rw-r--r-- | HACKING/chroot-tools.md | 3 | ||||
| -rw-r--r-- | HACKING/contributing.md | 9 | ||||
| -rw-r--r-- | HACKING/librefetch.md | 25 | ||||
| -rw-r--r-- | HACKING/libs.md | 15 | ||||
| -rw-r--r-- | HACKING/licensing.md | 8 | ||||
| -rw-r--r-- | HACKING/mksource.md | 92 | ||||
| -rw-r--r-- | HACKING/releasing.md | 70 | ||||
| -rw-r--r-- | HACKING/testing.md | 30 | ||||
| -rw-r--r-- | HACKING/translations.md | 108 |
10 files changed, 339 insertions, 25 deletions
diff --git a/HACKING/build-system.md b/HACKING/build-system.md index 6c8381a..005fa44 100644 --- a/HACKING/build-system.md +++ b/HACKING/build-system.md @@ -4,9 +4,9 @@ It is documented in `build-aux/Makfile.README.txt`. It provides all of the standard targets and such; you tell it what to do by setting a series of `at.whatever` or `module.whatever` variables. -There are a couple of variables that get automatically set from git. +There are a couple of variables that get automatically set from Git. This happens by `include`ing a hidden makefile that sets them; if -`$(topsrcdir)/.git` exists, it knows to use git to generate it; +`$(topsrcdir)/.git` exists, it knows to use Git to generate it; otherwise it expects the file to exist: | variable | file | diff --git a/HACKING/chroot-tools.md b/HACKING/chroot-tools.md new file mode 100644 index 0000000..e50aa11 --- /dev/null +++ b/HACKING/chroot-tools.md @@ -0,0 +1,3 @@ +Unfortunately, `makechrootpkg.sh` is GPLv2 ONLY. This means that +everything that uses it must be held to GPLv2+ instead of GPLv3+. I'm +calling this anything that gets loaded into the same process as it. diff --git a/HACKING/contributing.md b/HACKING/contributing.md index 3a9d083..52982d7 100644 --- a/HACKING/contributing.md +++ b/HACKING/contributing.md @@ -8,19 +8,18 @@ it probably means the code was too complicated in the first place. Patches should be sent to <dev@lists.parabola.nu>; please put "[PATCH]" and "libretools" in the subject line. If you have commit access, but want me to look over it first, feel free to create a new -branch in git, and I will notice it. Try to avoid pushing to the +branch in Git, and I will notice it. Try to avoid pushing to the "master" branch unless it's a trivial change; it makes it easier to review things; though I *will* look over every commit before I do a release, so don't think you can sneak something in :) Be sure to make sure to follow the licensing requirements in -`HACKING/licensing.md` +[./licensing.md][] I'd love to discuss possible changes on IRC (I'm lukeshu), either on -irc.freenode.net#parabola or in personal messages. My account may be +irc.libera.chat#parabola or in personal messages. My account may be online even if I'm not; I will eventually see your message, I do a search for mentions of "luke" on #parabola every time I get on. Please write unit tests for new functionality. Or old functionality. -Please write unit tests! See `HACKING/testing.md` for details on -testing. +Please write unit tests! See [./testing.md][] for details on testing. diff --git a/HACKING/librefetch.md b/HACKING/librefetch.md new file mode 100644 index 0000000..52bdbab --- /dev/null +++ b/HACKING/librefetch.md @@ -0,0 +1,25 @@ +`librefetch` respects `SOURCE_DATE_EPOCH`[1] for reproducible builds, +and has had this behavior from the start (though the variable name +wasn't there from the start, that came later). But how can this be!? +If you asked h01ger (the guy heading up reproducible builds in Debian +and Fedora), tar needs to support `--clamp-mtime` to do this, and that +flag wasn't in any tar implementation when librefetch came into +existence. The flag not yet being in upstream GNU tar was for a long +time the biggest blocker to most of upstream Debian being +reproducible! + +While it is a nice flag that I'm surprised wasn't added years ago, +it's trivial to just adjust the input files first: + + find . -exec touch --no-dereference --date=DATE -- {} +` + +Of course, `--clamp-mtime`/`--mtime` is now in upstream GNU tar. So +librefetch supporting this is no longer impressive. But, librefetch +still isn't using `--mtime`. Why? Well, because it uses libarchive +`bsdtar`, not GNU tar, and bsdtar still doesn't have the flag. + +I wrote this document mostly because I expect that sometime in the +future I'll think to myself "I should simplify the code and just use +`--mtime`", not remembering the differences between tars. + +[1]: https://reproducible-builds.org/specs/source-date-epoch/ diff --git a/HACKING/libs.md b/HACKING/libs.md new file mode 100644 index 0000000..8bebaf6 --- /dev/null +++ b/HACKING/libs.md @@ -0,0 +1,15 @@ +Special stuff about hacking ih the /src/lib directory: + + - Everything should be GPLv2 AND GPLv3 compatible. No GPLv3 only. + - Name a file `libre${NAME}` if it should be executable directly, or + `${name}.sh` if it should only be available to be sourced. + - When printing a message that is internal to /src/lib, and not part + of the programm calling the library; prefix the print command with + `_l`. `_l()` is defined in `common.sh` (and `librelib`, since it + cannot use any libraries itself). + - When changing the message functions, be aware that some are + duplicated in: + * /src/chroot-tools/chcleanup + * /src/chroot-tools/distcc-tool + * /src/lib/librelib + And that they probably need to be updated as well. diff --git a/HACKING/licensing.md b/HACKING/licensing.md index 6d17e31..ec83689 100644 --- a/HACKING/licensing.md +++ b/HACKING/licensing.md @@ -6,14 +6,12 @@ New code should (please) be licensed GPLv2+. I say v2 instead of v3 because some code from Arch is GPLv2 (no "or any later version"), and having to worry about which programs can be combined is a huge pain. -Copyright statements should look like +Copyright statements for most code, should be of the form: # Copyright (C) YEARS NAME <EMAIL> -for most code, for 3rd-party code that has been imported, indent it a -bit: +For 3rd-party code, which has been imported, indent it a bit: # Copyright (C) YEARS NAME <EMAIL> -Always put a line with `# License:` specifying the license of that -file. +Always put a `# License:` line specifying the license of that file. diff --git a/HACKING/mksource.md b/HACKING/mksource.md new file mode 100644 index 0000000..01d759d --- /dev/null +++ b/HACKING/mksource.md @@ -0,0 +1,92 @@ +This is the internal libremakepkg program flow for a PKGBUILD with mksource(). + +<pre> +Given: + * a PKGBUILD with a mksource() function and a repo.parabola.nu URL in source() + * /etc/makepkg.d/librefetch.conf is sourced by makepkg.conf + * the checksum has be generated with: `librefetch -g`, and added to mksha256sums() + * mk*sums() is populated with a checksum for each mksource() (use: `librefetch -g`) + * *sums() is populated with 'SKIP' for each source() (add it before publishing the PKGBUILD) + +/usr/bin/libremakepkg + -> msg 'Starting pre-build activities...' + -> msg 'Downloading sources...' + -> /usr/lib/libretools/chroot/makechrootpkg.sh::download_sources() + +/usr/lib/libretools/chroot/makechrootpkg.sh::download_sources() + -> /usr/bin/makepkg --allsource (w/ special ENV and in-chroot makepkg.conf) + +/usr/bin/makepkg --allsource ($SOURCEONLY == 2): + -> msg "Making package:" + -> /usr/share/makepkg/source.sh::download_sources('allarch') + +/usr/share/makepkg/source.sh::download_sources(): + -> msg "Retrieving sources..." + -> (/usr/bin/librefetch is download agent) + +/usr/bin/librefetch::doit() + -> 'download' mode ($src == PKGBUILD::source() , $inmirror == true) + -> if found + -> return to /usr/bin/makepkg + -> if not found + -> 'create' mode + -> /tmp/<SPECIAL>/makepkg -p /tmp/<SPECIAL>/PKGBUILD + +/tmp/<SPECIAL>/makepkg (SOURCEONLY == 0): + -> msg "Making source:" + -> msg "Checking source dependencies..." + -> /tmp/<SPECIAL>/source.sh::download_sources('allarch') + +/tmp/<SPECIAL>/source.sh::download_sources(): + -> msg "Retrieving sources..." + -> (/usr/bin/librefetch is download agent) + +/usr/bin/librefetch::doit() + -> 'download' mode ($src == PKGBUILD::mksource() , $inmirror == false) + -> return to /tmp/<SPECIAL>/makepkg + +/tmp/<SPECIAL>/makepkg (post-return from /usr/bin/librefetch): + -> NOTE: all standard PKGBUILD functions are stubbed out, + with the only purpose to run the real PKGBUILD::mksource() + the standard functions in the real PKGBUILD will run later in the chroot + -> msg "Validating source files" + -> msg "Extracting sources..." + -> msg "Starting prepare()" + -> msg "Starting build()" + -> msg "Starting mksource()" + -> PKGBUILD::mksource() + -> msg "Starting check()" + -> msg "Starting package()" + -> msg "Creating package" + -> msg "Finished making:" + +NOTE: here we rewind back to /usr/lib/libretools/chroot/makechrootpkg.sh::download_sources() + to get/create the source-ball signature + +/usr/bin/librefetch::doit() + -> 'download' mode + -> if not found + -> 'create' mode ($src == PKGBUILD::source() , $inmirror == true) + -> /usr/bin/librefetch::create_signature() + +/usr/bin/librefetch::create_signature() + -> msg "Signing package..." + -> /usr/share/makepkg/integrity.sh::check_source_integrity() + -> /usr/share/makepkg/integrity/verify_checksum.sh + -> msg "Validating source files" + -> /usr/share/makepkg/integrity/verify_signature.sh + -> msg "Verifying source file signatures with gpg" + +NOTE: here we rewind back to /tmp/<SPECIAL>/makepkg + +/tmp/<SPECIAL>/makepkg: + -> msg "Source package created:" + -> exit + +NOTE: then we rewind back to /usr/lib/libretools/chroot/makechrootpkg.sh::download_sources() + which returns to /usr/bin/libremakepkg + +/usr/bin/libremakepkg: + -> msg "Starting to build the package" + -> NOTE: build package in the normal way +</pre> diff --git a/HACKING/releasing.md b/HACKING/releasing.md new file mode 100644 index 0000000..325d7be --- /dev/null +++ b/HACKING/releasing.md @@ -0,0 +1,70 @@ +libretools release process +========================== + +This is the release process that I (LukeShu) follow (up-to-date as of +2024-05-10): + +Go through these steps; if any step fails, then jump to the "error +handling" section. + + 1. Ensure that the tests pass; as described in [./testing.md][], I + run the tests as + + sudo echo && TMPDIR=/var/lib/archbuild/tmp make check + + where `/var/lib/archbuild/tmp` is a directory owned by my normal + (non-root) user on a btrfs volume. I do not bother to run the + tests on a non-btrfs volume, although I arguably should. + + 2. Look over the commits going into the release, and draft a release + announcement email. I try to follow the same general format for + each announcement; check the mailing list archives for that + format. + + 3. Create a local `vYYYYMMDD[.N]` tag in libretools.git. The ".N" + suffix is for if this isn't the first release of the day. + + 4. If the devtools-par.git commit has changed since the last + libretools release: Create a corresponding local + `libretools-YYYYMMDD[.N]` tag in devtools-par.git. + + 5. Run `make dist` in the libretools.git checkout. + + 6. Copy the resulting `libretools-*.tar.gz` file to + `makepkg.conf:SRCDEST` (on my machine, that's + `~/packages/srcdest/`). + + 7. Update `libre/libretools/PKGBUILD` in abslibre.git. Running + `updpkgsums` will prompt me for my GnuPG credentials to create a + `.sig` for that tarball. + + 8. Run `sudo libremakepkg`. + + 9. Install the built packages from `makepkg.conf:PKGDEST` (on my + machine, that's `~/packages/pkgdest/`) + + 10. Do any additional manual testing that is relevant to changes in + this particular release. + + 11. Run `librestage` and `librerelease`. Doing this *after* + installing them in step 9 at least verifies that the new + libretools release is able to release itself. + + 12. Push any Git refs (tags/branches) from `libretools.git`, + `devtools-par.git`, or `abslibre.git`. + + 13. Hit "send" the release announcement email. + +## Error handling + +If any of the normal steps fails and I need to edit what's in the +release, I: + + 1. Delete any local Git tags that I created. + + 2. Delete any files from `makepkg.conf:SRCDEST` (on my machine, + that's `~/packages/srcdest/`). + + 3. Make whatever commits I need to. + + 4. Start the process over. diff --git a/HACKING/testing.md b/HACKING/testing.md index 8dee485..bd3bcd1 100644 --- a/HACKING/testing.md +++ b/HACKING/testing.md @@ -1,18 +1,22 @@ Testing ======= -Please write unit tests for new things. Tests can be run with `make -check`, which just runs `./testenv roundup` in the `test/` directory. -Relatedly, you need the `roundup` (the `sh-roundup` package on -Parabola) tool to run the tests. `./testenv` can be given -`--no-network` and/or `--no-sudo` to dissable tests that require those -things. Make can be made to pass those things in by setting -`TESTENVFLAGS`. If you don't dissable either, I *strongly* recommend -setting TMPDIR to somewhere on a btrfs partition before running the -tests; otherwise the chroot tests will take forever. I mean, they +Please write unit tests for new things. Tests can be run with `make check`, +which just runs `./testenv bats cases` in the `test/` directory. You will need +the `bats` tool (the `bats` Parabola package) in order to run the tests. `bats` +can be given `--filter-tags !network,!sudo,!btrfs` to disable tests which +require those things. `make check` can be made to pass those flags in by +setting `BATS_FLAGS`. If `!btrfs` is not given, then TMPDIR *must* be on btrfs +partition (or else the tests that need btrfs will fail). If neither `!network` +nor `!sudo` is given, then I *strongly* recommend setting TMPDIR to somewhere on +a btrfs partition; otherwise the chroot tests will take forever. I mean, they already take long on btrfs, but without it... _dang_. -I also recommend having the `haveged` daemon running. That's good -general advice, but also: some of the tests make GPG keys, this -"should" take on the order of 1 second, but can take several minutes -if you don't have `haveged` running. +> I (LukeShu) run the tests with: +> +> sudo echo && TMPDIR=/var/lib/archbuild/tmp make check +> +> The `sudo echo` is so that I don't get prompted for my password in +> the middle of the run. The `TMPDIR` is to point it at a btrfs +> volume with sufficient free space--my normal `/tmp` or `$TMPDIR` are +> too small for the tests. diff --git a/HACKING/translations.md b/HACKING/translations.md new file mode 100644 index 0000000..1b84f01 --- /dev/null +++ b/HACKING/translations.md @@ -0,0 +1,108 @@ +Translations for programs are provided in files with the `.po` suffix. These are +created by copying PO-template (`.pot`) files, and filling in the missing +"msgstr" values. + +To add a translation, you'll first (1) need to create the `.pot` files if you +don't already have them, and then (2) copy them and (3) fill them in. + + +## 1. Create the `.pot` files + +If you are working from a release source tarball, you can skip this step (the +`.pot` files are included in the tarball); if you are working from Git, read on. + +libretools' .pot files are not tracked in Git; they are created by running +`make`. You'll need a checkout of both 'devtools-par' and 'libretools' to run +`make` if building from Git. The 'devtools-par' and 'libretools' source root +directories should have the same parent directory. + + $ git clone https://git.parabola.nu/packages/libretools.git/ + $ git clone https://git.parabola.nu/packages/devtools-par.git/ + $ cd libretools + $ make po/files.generate + $ cd po/ + $ ls + gitget.pot + librelib.pot + libretools.pot + +NOTE: If you have modified the libretools source code, that may cause errors +during translation; especially if you have modified any of the translatable +strings. The failure will be evident by the error message: + + msgcmp: found N fatal errors + +If you see this error, refer to the "Updating translations" section below. + + +## 2. Create the `.po` Files + +Create a folder under `po/` with the two-letter language code[^1] you want to +add. Then, copy the `.pot` files to that folder, changing the file extension to +`.po`. Finally, link the `subdir.mk` file as the Makefile for that directory. +For example: + + $ mkdir es + $ for file in *.pot; do cp $file es/${file%t}; done + $ ln -s ../subdir.mk es/Makefile + +[^1]: See the langauge code table: <http://www.lingoes.net/en/translator/langcode.htm> + + +## 3. Fill the `.po` Files + +From there you can open each .po file in your favorite text editor, and fill in +the `msgstr` values with translations of the associated `msgid` keys. + + +## 4. Synchronizing/Updating Translations + +Normally, the translations are prepared as part of the dist make target +(the same command used to generate the release source-ball). Eg: + + $ make dist + +However, if you have modified the libretools source code significantly or +modified any translatable strings, it is likely that the `make` command will +fail, due to the translation files being mis-aligned with the latest changes. +You will need to run `msgmerge` to merge the newly generated .pot templates +into the existing .po files, as indicated by the `msgcmp` command traces in +the build log. + +Eg: If the following error message appeers after this log trace: + + msgcmp --use-untranslated --use-fuzzy po/es/libretools.po po/libretools.pot + .... + msgcmp: found 42 fatal errors + +then run: + + $ msgmerge po/es/libretools.po po/libretools.pot + +Notice any red (fuzzy) lines. Those are relatively minor mis-matches, which the +tool was able to reconcile, but may need correction. Also notice any green, +commented-out, 'msgid'/'msgstr' line pairs. Those are unresolved mis-matches, +which definitely need correction. They may correspond to a new 'msgid'/'msgstr' +line pair in the `msgmerge` output, with the new text as 'msgid', and an empty +'msgstr'; but the changes were too significant for the tool to determine the +correlation. The standard output of `msgmerge` is the raw text for a new +replacemnt .po file. Next, direct the output to a file, make the corrections +to the replacemnt .po file, using the `msgmerge` output as a guide; then replace +the failed .po file with the new corrected file. + + $ msgmerge po/es/libretools.po po/libretools.pot > po/es/libretools.po-merged + $ mv po/es/libretools.po{-merged,} + +Alternatively, a merge tool such as `meld`, is especially helpful for this task. +A merge tool should highlight the same merge conflicts, shown in the the +`msgmerge` output, as a side-by-side comparison, allowing you to merge the +changes from the generated '-merged' file, into the tracked .po file, in-place. +In order to minimize the possible merge conflict next time, it is best to merge +all of the changes, especially the comments with line numbers. + + $ meld po/es/libretools.po{-merged,} + +Once `make clean && make` is able to complete successfully, commit the new .po +file to the VCS. + + $ git commit -m 'sync translations' po/es/libretools.po |