From db125088520d739b19607057e8635a9ddb28fe6e Mon Sep 17 00:00:00 2001 From: Luke Shumaker Date: Fri, 24 May 2013 13:50:34 -0400 Subject: clean up doc/; normalize filenames, clean up createworkdir(.md) --- doc/treepkg.md | 130 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 doc/treepkg.md (limited to 'doc/treepkg.md') diff --git a/doc/treepkg.md b/doc/treepkg.md new file mode 100644 index 0000000..7f7ece1 --- /dev/null +++ b/doc/treepkg.md @@ -0,0 +1,130 @@ +# TreePKG + +`treepkg` is a tool for recursively building packages from an ABS tree. It has +been tested while building packages for the mips64el port and it has proven to +be useful. + +It was written having in mind the experience of `fullpkg`, which converted to +`fullpkg-ng` and then splitted into `fullpkg-build` and `fullpkg-find`. It's +aim is to simplify algorithms implemented on the fullpkg family, while solving +some design issues that made fullpkg miss some packages sometimes. + + +## Requirements + +`treepkg` needs the help of `toru-path` for "indexing" an ABS tree. `toru-path` +stores a tokyocabinet database of "pkgname" => "path" pairs, where pkgname is +replaced by the "pkgbase", "pkgname", and "provides" fields of a PKGBUILD, +followed by the path of the current PKGBUILD. + +This information is then used by `treepkg` to know where to find the PKGBUILD +of a package. The fullpkg family needed to guess this by traversing the full +ABS tree, and therefore it was unable to find pkgnames that differ from +pkgbase. + +So, to use `treepkg` you need to run `toru-path` after the ABS tree update. + +> Split PKGBUILDs make it difficult to extract metadata if it's stored inside +> package() functions. This will happen with the provides field and `treepkg` +> won't find that linux-libre-headers provides linux-headers, for instance. + +## How does it work + +`treepkg` must be run from the path of the PKGBUILD you're going to build (this +may change over time). This will be DEPTH=0 and it will create a temporary +directory to work on in the format /tmp/pkgbase-treepkg-random-string. Inside +this directory, it will copy the needed PKGBUILDs prefixed with their depth +number. The first package will always be copied to 000\_pkgbase. + +From then on, treepkg sources the PKGBUILD and runs itself over all pkgnames on +the depends and makedepends array, only if it detects their versions aren't +already built or deprecated by newer ones, using the `is_built` utility. + +While processing this info, it will increase the depth of the packages. It'll +also write a CSV file with the knowledge it acquires from the ABS tree (useful +for debugging). This file is called BUILDORDER. + +When this process ends (no more needed dependencies are found), the temporary +work dir is traversed in inverse order (from higher depths to 0) running the +FULLBUILDCMD from libretools.conf and then the HOOKLOCALRELEASE variable, which +*must* provide a way to `repo-add` the packages to an available repository, so +the next build will find and install the newer version using pacman. + +For instance, having this as the first pacman repository (on /etc/pacman.conf): + + [stage3] + Server = /var/cache/pacman/pkg + +And this on /etc/makepkg.conf: + + PKGDEST=/var/cache/pacman/pkg + +Your HOOKLOCALRELEASE script should look like this: + + # Get needed vars + source /etc/makepkg.conf + source /etc/libretools.conf + source PKGBUILD + + unset build package check + + fullver=$(full_version ${epoch:-0} ${pkgver} ${pkgrel}) + pkgs=() + + # Generate canonical package paths + msg "Adding packages to [stage3]..." + for name in ${pkgname[@]}; do + msg2 "${name} ${fullver}" + pkgs+=("${PKGDEST}/${name}-${fullver}-*.pkg.tar.*") + done + + # Add the packages to a local + repo-add ${PKGDEST}/stage3.db.tar.gz ${pkgs[@]} + + # Stage the packages for later releasing + librestage $1 + +> Note the first HOOKLOCALRELEASE argument is the remote repository name (core, +> extra, etc.) + +There's a special case when a dependency depends on another that was put on +a depth level lower than itself. In this case the build order will be wrongly +assumed and you may end up with broken packages. + +To explain it with an example: + + ghostscript (0) - fontconfig (1) + \ cups (1) - fontconfig (ignored) + +The second time fontconfig appears, it will be ignored. In this case cups will +build against an fontconfig version that will be outdated by the fontconfig +version at depth 1. In this cases, `treepkg` will detect it cached the +dependency on a lower depth, and will "bury" it to a depth higher than the +current one. Thus this will become the build path: + + ghostscript (0) - fontconfig (buried) + \ cups (1) - fontconfig (2) + +> Note: currently, `treepkg` doesn't perform recursive burying, so if you hit +> a really long build tree with some circular dependencies you may find +> packages buried several times and queued to build before their actuals deps. + +## Tips + +`treepkg` accepts two arguments: 1) the temporary work dir and 2) the next +depth level it should use (if current equals 0, it'll pass 1). You don't need +to pass this arguments when running it manually, they're used internally to +automatically construct the build path. + +But if a build failed, `treepkg` will cancel itself immediately informing you +where the leftovers files were left. If you pass this path to `treepkg` as the +first argument, it will resume the build, skipping to the last package being +packaged. + +You can take the opportunity given by this to modify the build path or the +PKGBUILDs, without having to run `treepkg` twice. For instance you can remove +a package from the build order, or move it manually, or update the PKGBUILD +that made `treepkg` fail in the first place. You can force a skipped package +(after building it manually) by using `touch built_ok` on the PKGBUILD dir. + +You don't probably want to mess with the second argument though. -- cgit v1.2.2