# -*- Mode: markdown -*- Autothing 3: The smart way to write GNU Makefiles ================================================= Autothing is a thing that does things automatically. Ok, more helpfully: Autothing is a pair of .mk Makefile fragments (`Makefile.head.mk` and `Makefile.tail.mk`) that you can `include` from your Makefiles to make them easier to write; specifically, it makes it _easy_ to write non-recursive Makefiles--and ones that are similar to plain recursive Makefiles, at that! To many people, talking about GNU Make directly is a non-starter because it means giving up the many other features that things like GNU Automake provide. Other projects like GNU Automake were created to plaster over differences between make(1) implementations; however, this isn't all that Automake provides, it also makes it easy to do complex things that users want, or the GNU Coding Standards require. That's silly; the implementation of these features should be orthogonal to plastering over the differences between Make implementations. So, in addition to the Automake core, Automake is distributed with several "modules" that implement similar feature sets to what Automake provides. Autothing does depend on GNU Make; other make(1) implementations will not work. However, if you are open to adding GNU Make as a dependency, then Autothing should obviate the need for GNU Automake, while also making your Makefiles better. Non-recursive? -------------- (For those of you who aren't up on Makefile jargon) When you have a project that spans multiple directories, you'll probably want to split up the Makefile, having the appropriate parts in each sub-directory. There are a number of strategies you can use to approach this. One of the more prevelant strategies (so much so that GNU make includes special support for it) is to write "recursive Makefiles"; that is, have Make rules that include commands like other-directory/libfoo.so: $(MAKE) -C other-directory libfoo.so or other-directory/libfoo.so cd other-directory && $(MAKE) libfoo.so This approach is popular because it is both very easy to implement, and is supported by a wide variety of Make implementations. But, it also introduces a wide variety of issues; so much so that a rather famous paper was written about it: "Recursive Make Considered Harmful" (Miller, 1997). For all of the arguments against it, and all of the alternative approaches, recusive Makefiles are hard to beat because they are just so easy to write, and the alternatives... aren't. UNTIL NOW! Instead of having rules that spawn a separate Make process in another directory for targets in that directory, Autothing lets you provide a list of directories that include targets that targets in this directory might depend on, and Autothing will automagically include the Makefile in that other directory into *this* instance of the Make program. Peter Miller (1997) "Recursive Make Considered Harmful" An example Makefile / Introduction ---------------------------------- Write your Makefiles of the form: # Initialize basic information about how your project is structured. topsrcdir ?= ... topoutdir ?= ... # Include the Autothing entry point include $(topsrcdir)/build-aux/Makefile.head.mk # Now write your Makefile very similarly to how you normally # would. Just make sure that outputs are relative to $(outdir) # and inputs relative to $(srcdir). $(outdir)/%.o: $(srcdir)/%.c: $(CC) -c -o $@ $< $(outdir)/hello: $(outdir)/hello.o # If any of the dependencies of files here are outputs of a # Makefile in another directory, list those directories here. at.subdirs = ... # This part is kind of a pain: define a list of ouput targets that # this Makefile produces. at.targets = $(outdir)/%.o $(outdir)/hello # Include the Autothing exit point include $(topsrcdir)/build-aux/Makefile.tail.mk This is similar to, but not quite, the comfortable way that you probably already write your Makefiles. It is recommended that Autothing lives inside of the "build-aux" directory in the top level of your project sources; "build-aux" is a standard directory for auxiliary build programs and tools. What does Autothing do for me? ------------------------------ There are two fundamental things that Autothing provides: 1. Variable namespacing 2. Tools for dealing with paths The first is important because globals are bad for composability. The second is important because GNU Make is too dumb to know that `foo/bar/../baz` == `foo/baz`. Then, there's something that maybe doesn't belong, but I didn't have the heart to cut it out: 3. A module (plugin) system, which allows for modules to provide additional feature sets. The module system is "important" because there are very often common bits that you want to be included in every Makefile, and this gives some structure to that. Let's step through each of those features. ## Variable namespacing When you write a Makefile, you quite likely use (global) variables. When you have a project that uses multiple Makefiles, each Makefile might have the same variable names, but with different values (especially if converting from recursive Make). You could be very disciplined and carefully name your variables so that they don't conflict. This is difficult and error prone normally, but becomes neigh-on-impossible if you are converting a large-ish project from recursive Make. So, Autothing provides a solution. If you provide Autothing with a list of targets defined in your Makefile (via the `at.targets` variable), Autothing will make any variables you defined local to that Makefile; they will be present when making targets listed in `at.targets`, but will be hidden from other Makfiles that get included. Any variables defined before `Makefile.head.mk` is included are treated as truly global; all Makefiles included will have access to them. ## Tools for dealing with paths As stated above, GNU Make is too dumb to realize that `foo/bar/../baz` == `foo/baz`; so one has to be reasonably careful about path normalization. For dealing with path normalization problems that arise because of the way Autothing inclusions work, several global functions are provided for dealing with paths. `$(call at.is_subdir,a,b)` returns whether `b` is a sub-directory of `a` (including `a` as a sub-directory of itself). `at.is_strict_subdir` does the same, but does not treat `a` as a sub-directory of itself. (These function names mimic the terms "subset" and "strict subset" in mathematics.) These use an empty string for "false" and a non-empty string for "true". `$(call at.path,files...)` is a generic path-normalization routine. The outputs of the other `at.*` functions are already normalized, and do not need to be passed through this. Files immediately inside of `$(srcdir)` or `$(outdir)` (without another directory name after the variable) are already normalized, and do not need to be passed through this function either. However, it is always safe to pass a path through this function, so if in doubt, call `at.path`. `$(call at.relbase,dir,files...)` and its cousin `at.relto` take a directory and a list of files, and transform each of the filenames to be relative to the directory, if the file is inside of the directory. Where they differ is that if the file is not inside of the directory; `at.relbase` transforms it into an absolute path, while `at.relto` prepends as many `../` segments as necessary to make it relative to the directory. (These function names mimic the `--relative-base` and `--relative-to` flags of the `realpath` utility that is part of GNU coreutils.) If `$(srcdir)` and `$(outdir)` are the same, then `$(call at.out2src,files...)` is a no-op, but otherwise it takes a (possibly relative) path in `$(outdir)`, and transforms it to the equivalent filename in `$(srcdir)`. `$(call at.addprefix,dir,files...)` takes a directory and a list of filenames, and looks at each filename; if it is an absolute path, it passes it through (well, "only" normalizes it); if the filename is a relative path, it is joined with the given base directory. ## Modules to provide feature sets The module system serves two purposes 1. Allow your developers to share logic between Makefiles in multiple directories. 2. Allow your developers to import "standard" modules implementing common feature sets, so they don't have to. Distributed along with autothing are some "standard" modules that provide commonly desired functionality from Makefiles; tricky little things that your developers shouldn't have to implement themselves for every project; the things that GNU Automake would take care of if you used Automake (a piece of software that Autothing hopes to replace). The module system is conceptually quite simple: have 4 directories for `.mk` makefile snippets that get included at certain points: Makefile.once.head/*.mk Makefile.each.head/*.mk a/Makefile Makefile.each.tail/*.mk Makefile.each.head/*.mk b/Makefile Makefile.each.tail/*.mk Makefile.each.head/*.mk c/Makefile Makefile.each.tail/*.mk Makefile.once.tail/*.mk Deciding which of the 4 directories to put your snippets in... you'll figure it out pretty quickly once you start playing with it. Beyond these 4 directories, Autothing itself imposes no structure, but there are some conventions that are followed by the distributed along with Autothing, and I recommend that your developers follow. Each of the `.mk` files is name `NN-MODULE.mk` where NN is a number (to affect the order that the module files are evaluated in, in case of dependencies between them), and MODULE is the module name. Each module has "public" variables prefixed with `MODULE.`, and "private" variables prefixed with `_MODULE.` (again, "MODULE" being the module name). For example, the "groups" parameter of the "files" module is configured via the `files.groups` variable. Within this convention, Autothing presents itself as a pseudo-module named "at"; that is, public Autothing variables are prefixed with `at.`. If you follow these conventions, then the "mod" module distributed along with Autothing can display information about the modules that a project uses, and documentation on each module. Running the command `make at-modules` (implemented by the "mod" module) will produce a list of the modules present in a project, and short descriptions of them: $ make at-modules Autothing modules used in this project: - dist `dist` target for distribution tarballs (more) - files Keeping track of groups of files (more) - gitfiles Automatically populate files.src.src from git (more) - gnuconf GNU standard configuration variables (more) - mod Display information about Autothing modules (more) - nested Easy nested .PHONY targets (more) - quote Macros to quote tricky strings (more) - texinfo The GNU documentation system (more) - var Depend on the values of variables (more) - write-atomic `write-atomic` auxiliary build script (more) - write-ifchanged `write-ifchanged` auxiliary build script (more) The "(more)" at the end of a line indicates that there is further documentation for that module, which can be produced by running the command `make at-modules/MODULE_NAME`. See the output of `make at-modules/mod` for instructions on how to produce this further documentation for modules you develop. Besides the "mod" module, the set modules distributed along with Autothing primarily exists to provide the bits of (sometimes somewhat tricky) functionality required of Makefiles by the GNU Coding Standards. Run the `at-modules` commands above for documentation on each of them. Formal interface ---------------- System requirements: - A version of GNU Make that supports `undefine` (ie, version 3.82 and above). If the user attempts to use your Autothing-using Makefile with an older version of GNU Make, `Makefile.head.mk` will print an error message and refuse to proceed: $ make-3.81 build-aux/Makefile.head.mk:58: *** Autothing: We need a version of Make that supports 'undefine'. Stop. Inputs: - In each `Makefile`: - Before `Makefile.head.mk`: - Variable (mandatory) : `topoutdir` - Variable (mandatory) : `topsrcdir` (must not be a subdirectory of `$(topoutdir)`) - Variable (optional) : `at.Makefile` (Default: `Makefile`) - Between `Makefile.head.mk` and `Makefile.tail.mk`: - Variable: `at.targets` (Default: empty) - Variable: `at.subdirs` (Default: empty) - Files: - `${topsrcdir}/build-aux/Makefile.{each,once}.{head,tail}/*.mk` Unfortunately, a limitation of Autothing is that it does require a designated "top" directory; it can't be used to have a sub-project that can also be totally separate and built alone. In your Makefiles, before you include `Makefile.head.mk`, you must tell Autothing what the top directory is by setting `topoutdir` and `topsrcdir`. If you wish for your per-directory Makefiles to have a name other than `Makefile` (such as `GNUmakefile` or `makefile`, which GNU Make also looks for by default; or another name for project-specific reasons), Autothing supports this by setting the `at.Makefile` variable. Unfortunately, Autothing does not support having a list of filenames to try; so one must be consistent about the filename throughout the project. In the body of each Makefile, you may set the `at.targets` variable to list which targets should have access to the variables defined in the body of that Makefile. In the body of each Makefile, you may set the `at.subdirs` variable to list of directories which have their own Makefile which produces targets that targets in this directory depend on. Directories listed in `at.subdirs` may be relative or absolute; if relative, they are interpreted as relative to `$(outdir)`. Outputs: - Global: - Variable (function): `$(call at.is_subdir, parent, child)` - Variable (function): `$(call at.is_strict_subdir, parent, child)` - Variable (function): `$(call at.relbase, parent, children...)` - Variable (function): `$(call at.relto, parent, children...)` - Variable (function): `$(call at.path, paths...)` - Variable (function): `$(call at.out2src, paths...)` - Variable (function): `$(call at.addprefix, prefix, paths...)` - Variable : `$(at.nl)` # a single newline - Per-directory: - Variable: `$(outdir)` - Variable: `$(srcdir)` For dealing with path normalization problems that arise because of the way Autothing inclusions work, several global functions are provided for dealing with paths; see the above "Tools for dealing with paths" section for documentation on each of these functions. For convenience, it also provides `$(at.nl)` which is a single newline, as newlines are very difficult to type in Make variable values. Tips, notes ----------- If you use Autoconf (or similar), I recommend having a file at `$(topsrcdir)/config.mk.in` of the form ifeq ($(origin topsrcdir),undefined) topoutdir := $(patsubst %/,%,$(dir $(lastword $(MAKEFILE_LIST)))) topsrcdir := $(topoutdir)/@top_srcdir@ # Any other global variables you might want to set endif Then have `./configure` generate `$(topoutdir)/config.mk` from it by placing `AC_CONFIG_FILES([config.mk])` in your `configure.ac`. I recommend that you have `config.mk` be the _only_ Makefile edited by `./configure`; which will require manual support to have `./configure` link/copy the Makefiles unedited into `$(topoutdir)`; you can do this by placing something like this in your `configure.ac`: AC_OUTPUT([], [], [ if test "$srcdir" != .; then find "$srcdir" -name Makefile -printf '%P\n' \ | while read -r filename; do mkdir -p "\$(dirname "\$filename")" ln -srfT "$srcdir/\$filename" "\$filename" done fi ]) This will allow you to write your Makefiles in the form: include $(dir $(lastword $(MAKEFILE_LIST)))/../../config.mk include $(topsrcdir)/build-aux/Makefile.head.mk # your Makefile here include $(topsrcdir)/build-aux/Makefile.tail.mk Where you only need to adjust the number of `../` segments in the first line based on how deep that directory is. Further development ------------------- Most of the modules distributed along with Autothing have the goal of combining to provide the things that the GNU Coding Standards require. Between `gnuconf`, `dist`, `files`, and `texinfo`; the GNU Coding Standards for Makefiles are nearly entirely satisfied. However, there are a few targets that are required, but aren't implemented by a module (yet!): - `install-strip` - `TAGS` - `check` - `installcheck` (optional, but recommended) TODO ---- - Write documentation on `srcdir`, `outdir`, and out-of-tree builds; I don't think discussions involving the separate `srcdir` and `outdir` make much sense without that context. Bugs/Limitations ---------------- - This documentation file is almost three times as long as the code that it documents. - The "parse time" for projects with hundreds of sub-directories (each having a Makefile) can be slow (ex: a project with 166 directories has a parse time of around 12 seconds on my box). I blame GNU Make's garbage collector; I don't think it was ever designed to deal with as much "garbage" as Autothing's variable namespacing throws at it. - Requires a designated "top" directory; see discussion above. - Does not support varying per-directory Makefile names; see discussion above. ---- Copyright (C) 2016-2017 Luke Shumaker This documentation file is placed into the public domain. If that is not possible in your legal system, I grant you permission to use it in absolutely every way that I can legally grant to you.