summaryrefslogtreecommitdiff
path: root/doc/treepkg.md
blob: 1e30e33298aa58ae2ed55e5c9256c754b360dbf4 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
# TreePKG

`treepkg` is a tool for recursively building packages from an ABS tree. It has
been tested while building packages for the mips64el port and it has proven to
be useful.

It was written having in mind the experience of `fullpkg`, which converted to
`fullpkg-ng` and then splitted into `fullpkg-build` and `fullpkg-find`. It's
aim is to simplify algorithms implemented on the fullpkg family, while solving
some design issues that made fullpkg miss some packages sometimes.


## Requirements

`treepkg` needs the help of `toru-path` for "indexing" an ABS tree. `toru-path`
stores a tokyocabinet database of "pkgname" => "path" pairs, where pkgname is
replaced by the "pkgbase", "pkgname", and "provides" fields of a PKGBUILD,
followed by the path of the current PKGBUILD.

This information is then used by `treepkg` to know where to find the PKGBUILD
of a package. The fullpkg family needed to guess this by traversing the full
ABS tree, and therefore it was unable to find pkgnames that differ from
pkgbase.

So, to use `treepkg` you need to run `toru-path` after the ABS tree update.

> Split PKGBUILDs make it difficult to extract metadata if it's stored inside
> package() functions. This will happen with the provides field and `treepkg`
> won't find that linux-libre-headers provides linux-headers, for instance.

## How does it work

`treepkg` must be run from the path of the PKGBUILD you're going to build (this
may change over time). This will be DEPTH=0 and it will create a temporary
directory to work on in the format /tmp/pkgbase-treepkg-random-string. Inside
this directory, it will copy the needed PKGBUILDs prefixed with their depth
number. The first package will always be copied to 000\_pkgbase.

From then on, treepkg sources the PKGBUILD and runs itself over all pkgnames on
the depends and makedepends array, only if it detects their versions aren't
already built or deprecated by newer ones, using the `is_built` utility.

While processing this info, it will increase the depth of the packages. It'll
also write a CSV file with the knowledge it acquires from the ABS tree (useful
for debugging). This file is called BUILDORDER.

When this process ends (no more needed dependencies are found), the temporary
work dir is traversed in inverse order (from higher depths to 0) running the
FULLBUILDCMD from libretools.conf and then the HOOKLOCALRELEASE variable, which
*must* provide a way to `repo-add` the packages to an available repository, so
the next build will find and install the newer version using pacman.

For instance, having this as the first pacman repository (on /etc/pacman.conf):

    [stage3]
    Server = /var/cache/pacman/pkg

And this on /etc/makepkg.conf:

    PKGDEST=/var/cache/pacman/pkg

Your HOOKLOCALRELEASE script should look like this:

    # Get needed vars
    source /etc/makepkg.conf
    source /etc/libretools.conf
    source PKGBUILD

    unset build package check

    fullver=$(full_version ${epoch:-0} ${pkgver} ${pkgrel})
    pkgs=()

    # Generate canonical package paths
    msg "Adding packages to [stage3]..."
    for name in ${pkgname[@]}; do
        msg2 "${name} ${fullver}"
        pkgs+=("${PKGDEST}/${name}-${fullver}-*.pkg.tar.*")
    done

    # Add the packages to a local
    repo-add ${PKGDEST}/stage3.db.tar.gz ${pkgs[@]}

    # Stage the packages for later releasing
    librestage $1

> Note the first HOOKLOCALRELEASE argument is the remote repository name (core,
> extra, etc.)

There's a special case when a dependency depends on another that was put on
a depth level lower than itself. In this case the build order will be wrongly
assumed and you may end up with broken packages.

To explain it with an example:

    ghostscript (0) - fontconfig (1)
                    \ cups (1)       - fontconfig (ignored)

The second time fontconfig appears, it will be ignored. In this case cups will
build against an fontconfig version that will be outdated by the fontconfig
version at depth 1. In this cases, `treepkg` will detect it cached the
dependency on a lower depth, and will "bury" it to a depth higher than the
current one. Thus this will become the build path:

    ghostscript (0) - fontconfig (buried)
                    \ cups (1)            - fontconfig (2)

> Note: currently, `treepkg` doesn't perform recursive burying, so if you hit
> a really long build tree with some circular dependencies you may find
> packages buried several times and queued to build before their actuals deps.

## Tips

`treepkg` accepts two arguments: 1) the temporary work dir and 2) the next
depth level it should use (if current equals 0, it'll pass 1). You don't need
to pass this arguments when running it manually, they're used internally to
automatically construct the build path.

But if a build failed, `treepkg` will cancel itself immediately informing you
where the leftovers files were left. If you pass this path to `treepkg` as the
first argument, it will resume the build, skipping to the last package being
packaged.

You can take the opportunity given by this to modify the build path or the
PKGBUILDs, without having to run `treepkg` twice. For instance you can remove
a package from the build order, or move it manually, or update the PKGBUILD
that made `treepkg` fail in the first place. You can force a skipped package
(after building it manually) by using `touch built_ok` on the PKGBUILD dir.

You don't probably want to mess with the second argument though.