From e10e919aa7d9b9a957f47886ce4ab93139bbabd8 Mon Sep 17 00:00:00 2001 From: Andreas Grapentin Date: Wed, 18 Apr 2018 11:22:41 +0200 Subject: finished README --- README | 215 ++++++++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 145 insertions(+), 70 deletions(-) diff --git a/README b/README index 7a051b8..ddefe0a 100644 --- a/README +++ b/README @@ -2,28 +2,49 @@ parabola-riscv64-bootstrap ========================== + β€œDon't cry because it's over, smile because it happened.” + ― Dr. Seuss + 1. Introduction --------------- This project is an attempt to bootstrap a self-contained parabola GNU/Linux-libre system for the riscv64 architecture. The scripts are created -with the goal to be as generic as possible, to make future porting effort -easier. Feel free to adapt them to target any architecture you want by -modifying the variables in create.sh +with the goal to be as architectur agnostic as possible, to make future porting +efforts easier. -The build process is split into several stages, the rationale of which is -outlined in section 2 below. +The build process is split into four stages, the rationale of which is outlined +in section 2 below. To initiate a complete build of all stages, run: $> sudo ./create -To discard fragments from an earlier run and start from a clean slate, run: - $> sudo rm -rf build && sudo ./create +The builds can be configured to keep going if the build of a single package +fails, by creating the file `build/.KEEP_GOING`. Otherwise, the build will stop +once an error is encountered. This is useful for getting as much work done as +possible unattended, but will make debugging harder in the later stages, +because temporary build fragments and filesystem trees will be overwritten by +the next package. + +The complete console output of a package build process can be found in the +corresponding .MAKEPKGLOG file in the build directory of that package. + +1.1. System Requirements +------------------------ -You might have to unmount the directories mounted by the stages as noted in the -sections below. +The scripts require, among probably other things, to be running on a fairly +POSIX-conforming GNU/Linux system, and in particulary need the following tools +to be present and functional: -1.1. A note to the reader + * decently up-to-date GNU build toolchain (gcc, glibc, binutils) + * most of the things in base-devel + * pacman, makepkg + +I have tried to make the script smart enough to check for required bits and +pieces where needed, and to report when anything is missing ahead of time, but +some requirements may be missing. + +1.2. A note to the reader ------------------------- The scripts assume to be running on a parabola GNU/Linux-libre system, and may @@ -35,14 +56,15 @@ pay close attention to any output, and be prepared to fix and modify patches and scripts. Also, if you found this project useful, and want to chat about anything, you -can email me at . +can email me at , or find me as in +#parabola and others on irc.freenode.org. -1.2. Current state of the project +1.3. Current state of the project --------------------------------- -The stage3 native base-devel makepkg chroot is finished, and stage4 is primed -to begin with native compilation of a first batch of release packages in the -chroot. +All four stages are complete and this repository is now closed. A pointer where +to find future development efforts for the parabola RISC-V port will be added +here in due time. 2. Build Stages --------------- @@ -51,20 +73,38 @@ The following subsections outline the reasoning behind the separate bootstrap stages. More details about *how* things are done may be gathered from reading the inline comments of the respective scripts. -2.1. Stage 1 ------------- +From Stage 2 onwards, the scripts use DEPTREEs to determine what packages need +to be built. these files are located in the build/stageX/ directories and can +give an insight into what packages are going to be built next and what +dependencies are still missing. + +Since risc-v is a fairly new ISA, some packages were packaged with config.sub +and config.guess files that are too old to recognize the target triplet. This +requires config.sub and config.guess files to be refreshed with newer versions +from upstream. This is done automatically for stage 2 and onwards, if +REGEN_CONFIG_FRAGMENTS is set to yes (the default) in create.sh or the +environment. + +Packages with the `any' architecture are reused from upstream arch for this +bootstrap, since they should work for any architecture and do not need to be +re-(or cross-)compiled. -The first stage creates a cross-compile toolchain for the target triplet -defined in $CHOST, consisting of binutils, linux-libre-api-headers, gcc and -glibc. This toolchain has been pre-packaged for riscv64-unknown-linux-gnu on -parabola GNU/Linux-libre and may be installed by running: +Additionally, all checkdepends are ignored, and the check() phase of the builds +is skipped for sanity reasons. - $> pacman -S riscv64-unknown-linux-gnu-gcc +Note that the stages use upstream arch and parabola PKGBUILDs and attempt to +apply custom patches to resolve unbuildable or missing dependencies, and fix +risc-v specific build issues. Look for these patches in src/stageX/patches and +be prepared to fix and adapt them for future porting efforts. + +2.1. Stage 1 +------------ -In any case, the toolchain will be bootstrapped by the stage1.sh script, unless -it is already installed. The scripts will check for $CHOST-ar and $CHOST-gcc in -$PATH to determine whether binutils and gcc are installed, and will then -proceed to look for the following files in $CHOST-gcc's sysroot: +The first stage creates and installs a cross-compile toolchain for the target +triplet defined in $CHOST, consisting of binutils, linux-libre-api-headers, gcc +and glibc. The scripts will check for $CHOST-ar and $CHOST-gcc in $PATH to +determine whether binutils and gcc are installed, and will then proceed to look +for the following files in $CHOST-gcc's sysroot: $sysroot/lib/libc.so.6 # for $CHOST-glibc $sysroot/include/linux/kernel.h # for $CHOST-linux-libre-api-headers @@ -90,58 +130,93 @@ src/stage1/toolchain-pkgbuilds/*. 2.2. Stage 2 ------------ -Stage 2 uses the toolchain created in Stage 1 to cross-compile the packages of -the base-devel group of packages plus transitive runtime dependencies, in order -to bootstrap a functional cross-makepkg librechroot. +Stage 2 uses the toolchain created in Stage 1 to cross-compile a subset of the +packages of the base-devel group plus transitive runtime dependencies. The script creates an empty skeleton chroot, into which the cross-compiled -packages are installed, and creates a sane makepkg.conf and a patched -makepkg.sh to work in the prepared chroot root directory. To make the sysroot -of the compiler available to builds in the chroot and vice-versa, the /usr -directory of the chroot is mounted into the sysroot. At the end of Stage 2, -this directory is unmounted automatically. - -The transitive dependency tree of the base-devel package group is slightly -modified in a way that resolves any cyclic dependencies, and preserves a -somewhat sane build-order of the packages. Some packages that are too painful -to cross-compile are skipped entirely. - -To build the packages, the dependency tree is traversed and packages are -rebuilt using upstream PKGBUILDs and custom patches to cross-compile the -packages for the configured target architecture. Packages with the `any' -architecture are simply reused for this stage, since they should work for any -architecture and do not need to be cross-compiled. - -As a final note, since the upstream PKGBUILDS change frequently, and the -patches are unlikely to be maintained once the initial bootstrap is done and -stable, they will probably cease to function in the near future. Exercise -caution. +packages are going to be installed, and creates a sane makepkg.conf and a +patched makepkg.sh to work in the prepared chroot root directory. To make the +sysroot of the compiler available to builds in the chroot and vice-versa, the +/usr directory of the chroot is mounted into the sysroot. At the end of Stage +2, or in case of an error, this directory is unmounted automatically. + +To build the packages, the DEPTREE is traversed and packages are cross-compiled +using upstream PKGBUILDs and custom patches, and the compiled packages are +installed into the chroot immediately. + +Note that this process is a bit fragile and dependent on arbitrary +particularities of the host system, and thus might fail for subtle reasons, +like missing, or superfluous build-time installed packages on the host. +Exercise caution and common sense. 2.3. Stage 3 ------------ Stage 3 uses the cross-compiled makepkg chroot created in Stage 2 to natively -compile the base-devel group of packages again, but without using a -cross-compiler and without the need for mandatory package patches. Many -packages `just work' when compiling them using the upstream PKGBUILDs, but some -still need to be altered, and build time dependencies need to be removed when -they are too painful to compile. - -This stage requires more packages to complete, since the host system can not -provide things like git for vcs sources. Everything is taken from the makepkg -chroot. - -The process is similar to stage2, a clean librechroot is created using the -packages built in Stage 2, a modified libremakepkg.sh is created to inclued a -update hook for config.sub and config.guess scripts, which are unfortunately -too old for most packages to detect the architecture correctly, a dependency -tree is made, and then traversed. - -The patches in this stage are fewer and apply less pressure to the packages, -so, while computationally more expensive, this stage is probably a bit easier -to maintain and adapt. +recompile the base-devel group of packages. This stage requires to build more +packages, since a reduced set of make-time dependencies need to be present in +the makepkg chroot, as well as runtime dependencies. Additionally, running the +cross-compiled native compiler instead of the cross compiler takes longer. As a +result, stage 3 is expected to takes much longer than stage 2. + +However, since now the process is isolated from the host systems installed +packages, since everything is built cleanly in a chroot, the process is much +more stable and less prone to hard to diagnose problems with the host system. + +The scripts create a clean librechroot from the cross-compiled packages +produced in stage 2. A modified libremakepkg script is created to perform +config fragment regeneration and to skip the check() phase, and then packages +are built in order. + +Since no cross-compilation is needed from stage 3 onwards, fewer packages need +patching, and the patches are typically smaller and apply less pressure. As a +consequence, the patches probably need less maintenance in the future, and less +work to adapt to a different architecture. + +Note that the cross-compiled packages from stage 3 can be a bit derpy at times, +hence the stage 3 build scripts prioritize building bash and make natively, to +avoid some very weird known issues down the line. 2.4. Stage 4 ------------ -tbd. +Stage 4 does a final recompile of the packages of the base-devel group, similar +to stage 3, with the difference that more make-time dependencies are enabled, +and the packages of the base group are added to the DEPTREE. + +Stage 4 relies entirely on the packages natively compiled in stage 3, and no +cross-compiled packages are present in the build chroot at any time. This +results in reliable builds and reproducible build failures. However, since the +number of packages to be built in stage 4 is close to 750, expect the builds to +take a long time (days / weeks, not including work required to fix broken +patches and builds). + +The result of stage 4 is a repository of packages that should allow to +bootstrap and boot a risc-v virtual machine, with the packages required to +build the entirety of the arch / parabola package repository. At this point, I +consider the bootstrap process done. + +3. Final Words +============== + +I would like to thank the awesome abaumann from the archlinux32 project for +pointers on how to bootstrap a PKGBUILD based system for a new architecture, +and for his work on the bootstrap32 project, which helped a lot in getting this +project started: + + https://github.com/archlinux32/bootstrap32 + +Further, I would like to thank the amazing people in #riscv and #fedora-riscv +on irc.freenode.org, especially rwmjones, davidlt and sorear for all the help +getting packages to build and work on risc-v, and for providing the source +packages of all fedora-riscv packages, including all patches and build recipes, +which have been of immense help: + + https://fedorapeople.org/groups/risc-v/SRPMS/ + +Lastly, it was refreshing to see yor names pop up over and over again on the +risc-v ports, patches and pull requests out there, and it made me realize how +much work it really is to get a port like this off the ground. I could not have +made it this far without your work. + + ― Andreas Grapentin, 2018-04-18 -- cgit v1.2.2