[ACCEPTED]-What is your experience with non-recursive make?-makefile

We use a non-recursive GNU Make system in 44 the company I work for. It's based on Miller's 43 paper and especially the "Implementing non-recursive 42 make" link you gave. We've managed to refine 41 Bergen's code into a system where there's 40 no boiler plate at all in subdirectory makefiles. By 39 and large, it works fine, and is much better 38 than our previous system (a recursive thing 37 done with GNU Automake).

We support all the 36 "major" operating systems out there (commercially): AIX, HP-UX, Linux, OS 35 X, Solaris, Windows, even the AS/400 mainframe. We 34 compile the same code for all of these systems, with 33 the platform dependent parts isolated into 32 libraries.

There's more than two million 31 lines of C code in our tree in about 2000 30 subdirectories and 20000 files. We seriously 29 considered using SCons, but just couldn't 28 make it work fast enough. On the slower 27 systems, Python would use a couple of dozen 26 seconds just parsing in the SCons files 25 where GNU Make did the same thing in about 24 one second. This was about three years 23 ago, so things may have changed since then. Note 22 that we usually keep the source code on 21 an NFS/CIFS share and build the same code on 20 multiple platforms. This means it's even 19 slower for the build tool to scan the source 18 tree for changes.

Our non-recursive GNU Make 17 system is not without problems. Here are 16 some of biggest hurdles you can expect to 15 run into:

• Making it portable, especially to Windows, is a lot of work.
• While GNU Make is almost a usable functional programming language, it's not suitable for programming in the large. In particular, there are no namespaces, modules, or anything like that to help you isolate pieces from each other. This can cause problems, although not as much as you might think.

The major wins over our old recursive 14 makefile system are:

• It's fast. It takes about two seconds to check the entire tree (2k directories, 20k files) and either decide it's up to date or start compiling. The old recursive thing would take more than a minute to do nothing.
• It handles dependencies correctly. Our old system relied on the order subdirectories were built etc. Just like you'd expect from reading Miller's paper, treating the whole tree as a single entity is really the right way to tackle this problem.
• It's portable to all of our supported systems, after all the hard work we've poured into it. It's pretty cool.
• The abstraction system allows us to write very concise makefiles. A typical subdirectory which defines just a library is just two lines. One line gives the name of the library and the other lists the libraries this one depends on.

Regarding the last item 13 in the above list. We ended up implementing 12 a sort of macro expansion facility within 11 the build system. Subdirectory makefiles 10 list programs, subdirectories, libraries, and 9 other common things in variables like PROGRAMS, SUBDIRS, LIBS. Then 8 each of these are expanded into "real" GNU 7 Make rules. This allows us to avoid much 6 of the namespace problems. For example, in 5 our system it's fine to have multiple source 4 files with the same name, no problem there.

In 3 any case, this ended up being a lot of work. If 2 you can get SCons or similar working for 1 your code, I'd advice you look at that first.

After reading the RMCH paper, I set out 53 with the goal of writing a proper non-recursive 52 Makefile for a small project I was working 51 on at the time. After I finished, I realized 50 that it should be possible to create a generic 49 Makefile "framework" which can be used to 48 very simply and concisely tell make what 47 final targets you would like to build, what 46 kind of targets they are (e.g. libraries 45 or executables) and what source files should 44 be compiled to make them.

After a few iterations 43 I eventually created just that: a single 42 boilerplate Makefile of about 150 lines 41 of GNU Make syntax that never needs any 40 modification -- it just works for any kind 39 of project I care to use it on, and is flexible 38 enough to build multiple targets of varying 37 types with enough granularity to specify 36 exact compile flags for each source file 35 (if I want) and precise linker flags for 34 each executable. For each project, all I 33 need to do is supply it with small, separate 32 Makefiles that contain bits similar to this:

TARGET := foo

TGT_LDLIBS := -lbar

SOURCES := foo.c baz.cpp

SRC_CFLAGS   := -std=c99
SRC_CXXFLAGS := -fstrict-aliasing
SRC_INCDIRS  := inc /usr/local/include/bar

A 31 project Makefile such as the above would 30 do exactly what you'd expect: build an executable 29 named "foo", compiling foo.c (with CFLAGS=-std=c99) and 28 baz.cpp (with CXXFLAGS=-fstrict-aliasing) and 27 adding "./inc" and "/usr/local/include/bar" to 26 the #include search path, with final linking including 25 the "libbar" library. It would also notice 24 that there is a C++ source file and know 23 to use the C++ linker instead of the C linker. The 22 framework allows me to specify a lot more 21 than what is shown here in this simple example.

The 20 boilerplate Makefile does all the rule building 19 and automatic dependency generation required 18 to build the specified targets. All build-generated 17 files are placed in a separate output directory 16 hierarchy, so they're not intermingled with 15 source files (and this is done without use 14 of VPATH so there's no problem with having 13 multiple source files that have the same 12 name).

I've now (re)used this same Makefile 11 on at least two dozen different projects 10 that I've worked on. Some of the things 9 I like best about this system (aside from 8 how easy it is to create a proper Makefile for 7 any new project) are:

• It's fast. It can virtually instantly tell if anything is out-of-date.
• 100% reliable dependencies. There is zero chance that parallel builds will mysteriously break, and it always builds exactly the minimum required to bring everything back up-to-date.
• I will never need to rewrite a complete makefile again :D

Finally I'd just mention 6 that, with the problems inherent in recursive 5 make, I don't think it would have been possible 4 for me to pull this off. I'd probably have 3 been doomed to rewriting flawed makefiles 2 over and over again, trying in vain to create 1 one that actually worked properly.

Let me stress one argument of Miller's paper: When 46 you start to manually resolve dependency 45 relationships between different modules 44 and have a hard time to ensure the build 43 order, you are effectively reimplementing 42 the logic the build system was made to solve 41 in the first place. Constructing reliable recursive make build systems is very hard. Real life projects 40 have many interdependent parts whose build 39 order is non-trivial to figure out and thus, this 38 task should be left to the build system. However, it 37 can only resolve that problem if it has 36 global knowledge of the system.

Furthermore, recursive 35 make build-systems are prone to fall apart 34 when building concurrently on multiple processors/cores. While 33 these build systems may seem to work reliably 32 on a single processor, many missing dependencies 31 go undetected until you start to build your 30 project in parallel. I've worked with a 29 recursive make build system which worked 28 on up to four processors, but suddenly crashed 27 on a machine with two quad-cores. Then I 26 was facing another problem: These concurrency issues are next 25 to impossible to debug and I ended up drawing 24 a flow-chart of the whole system to figure 23 out what went wrong.

To come back to your 22 question, I find it hard to think of good 21 reasons why one wants to use recursive make. The 20 runtime performance of non-recursive GNU 19 Make build systems is hard to beat and, quite 18 the contrary, many recursive make systems 17 have serious performance problems (weak 16 parallel build support is again a part of 15 the problem). There is a paper in which I evaluated 14 a specific (recursive) Make build system 13 and compared it to a SCons port. The performance 12 results are not representative because the 11 build system was very non-standard, but 10 in this particular case the SCons port was 9 actually faster.

Bottom line: If you really 8 want to use Make to control your software 7 builds, go for non-recursive Make because 6 it makes your life far easier in the long 5 run. Personally, I would rather use SCons 4 for usability reasons (or Rake - basically 3 any build system using a modern scripting 2 language and which has implicit dependency 1 support).

I made a half-hearted attempt at my previous 27 job at making the build system (based on 26 GNU make) completely non-recursive, but 25 I ran into a number of problems:

• The artifacts (i.e. libraries and executables built) had their sources spread out over a number of directories, relying on vpath to find them
• Several source files with the same name existed in different directories
• Several source files were shared between artifacts, often compiled with different compiler flags
• Different artifacts often had different compiler flags, optimization settings, etc.

One feature 24 of GNU make which simplifies non-recursive 23 use is target-specific variable values:

foo: FOO=banana
bar: FOO=orange

This means that when building target 22 "foo", $(FOO) will expand to "banana", but 21 when building target "bar", $(FOO) will 20 expand to "orange".

One limitation 19 of this is that it is not possible to have 18 target-specific VPATH definitions, i.e. there 17 is no way to uniquely define VPATH individually 16 for each target. This was necessary in our 15 case in order to find the correct source 14 files.

The main missing feature of GNU make 13 needed in order to support non-recursiveness 12 is that it lacks namespaces. Target-specific variables 11 can in a limited manner be used to "simulate" namespaces, but 10 what you really would need is to be able 9 to include a Makefile in a sub-directory 8 using a local scope.

EDIT: Another very useful 7 (and often under-used) feature of GNU make 6 in this context is the macro-expansion facilities 5 (see the eval function, for example). This is 4 very useful when you have several targets 3 which have similar rules/goals, but differ 2 in ways which cannot be expressed using 1 regular GNU make syntax.

I agree with the statements in the refered 14 article, but it took me a long time to find 13 a good template which does all this and 12 is still easy to use.

Currenty I'm working 11 on a small research project, where I'm experimenting 10 with continuous integration; automatically 9 unit-test on pc, and then run a system test 8 on a (embedded) target. This is non-trivial 7 in make, and I've searched for a good solution. Finding 6 that make is still a good choice for portable 5 multiplatform builds I finally found a good 4 starting point in http://code.google.com/p/nonrec-make

This was a true relief. Now 3 my makefiles are

• very simple to modify (even with limited make knowledge)
• fast to compile
• completely checking (.h) dependencies with no effort

I will certainly also use 2 it for the next (big) project (assuming 1 C/C++)

I have developed a non-recursive make system 133 for a one medium sized C++ project, which 132 is intended for use on unix-like systems 131 (including macs). The code in this project 130 is all in a directory tree rooted at a src/ directory. I 129 wanted to write a non-recursive system in 128 which it is possible to type "make 127 all" from any subdirectory of the top 126 level src/ directory in order to compile 125 all of the source files in the directory 124 tree rooted at the working directory, as 123 in a recursive make system. Because my solution 122 seems to be slightly different from others 121 I have seen, I'd like to describe it here 120 and see if I get any reactions.

The main 119 elements of my solution were as follows:

1) Each 118 directory in the src/ tree has a file named 117 sources.mk. Each such file defines a makefile 116 variable that lists all of the source files 115 in the tree rooted at the directory. The 114 name of this variable is of the form [directory]_SRCS, in 113 which [directory] represents a canonicalized 112 form of the path from the top level src/ directory 111 to that directory, with backslashes replaced 110 by underscores. For example, the file src/util/param/sources.mk 109 defines a variable named util_param_SRCS 108 that contains a list of all source files 107 in src/util/param and its subdirectories, if 106 any. Each sources.mk file also defines a 105 variable named [directory]_OBJS that contains 104 a list of the the corresponding object file 103 *.o targets. In each directory that contains 102 subdirectories, the sources.mk includes 101 the sources.mk file from each of the subdirectories, and 100 concatenates the [subdirectory]_SRCS variables 99 to create its own [directory]_SRCS variable.

2) All 98 paths are expressed in sources.mk files 97 as absolute paths in which the src/ directory 96 is represented by a variable $(SRC_DIR). For 95 example, in the file src/util/param/sources.mk, the 94 file src/util/param/Componenent.cpp would 93 be listed as $(SRC_DIR)/util/param/Component.cpp. The 92 value of $(SRC_DIR) is not set in any sources.mk 91 file.

3) Each directory also contains a Makefile. Every 90 Makefile includes a global configuration 89 file that sets the value of the variable 88 $(SRC_DIR) to the absolute path to the root 87 src/ directory. I chose to use a symbolic 86 form of absolute paths because this appeared 85 to be the easiest way to create multiple 84 makefiles in multiple directories that would 83 interpret paths for dependencies and targets 82 in the same way, while still allowing one 81 to move the entire source tree if desired, by 80 changing the value of $(SRC_DIR) in one 79 file. This value is set automatically by 78 a simple script that the user is instructed 77 to run when the package is dowloaded or 76 cloned from the git repository, or when 75 the entire source tree is moved.

4) The makefile 74 in each directory includes the sources.mk 73 file for that directory. The "all" target 72 for each such Makefile lists the [directory]_OBJS 71 file for that directory as a dependency, thus 70 requiring compilation of all of the source 69 files in that directory and its subdirectories.

5) The 68 rule for compiling *.cpp files create a 67 dependency file for each source file, with 66 a suffix *.d, as a side-effect of compilation, as 65 described here: http://mad-scientist.net/make/autodep.html. I chose to use the gcc 64 compiler for dependency generation, using 63 the -M option. I use gcc for dependency 62 generation even when using another compiler 61 to compile the source files, because gcc 60 is almost always available on unix-like 59 systems, and because this helps standardize 58 this part of the build system. A different 57 compiler can be used to actually compile 56 the source files.

6) The use of absolute 55 paths for all files in the _OBJS and _SRCS 54 variables required that I write a script 53 to edit the dependency files generated by 52 gcc, which creates files with relative paths. I 51 wrote a python script for this purpose, but 50 another person might have used sed. The 49 paths for dependencies in the resulting 48 dependency files are literal absolute paths. This 47 is fine in this context because the dependency 46 files (unlike the sources.mk files) are 45 generated locally and rather than being 44 distributed as part of the package.

7) The 43 Makefile in each director includes the sources.mk 42 file from the same directory, and contains 41 a line "-include $([directory]_OBJS:.o=.d)" that 40 attempts to include a dependency files for 39 every source file in the directory and its 38 subdirectories, as described in the URL 37 given above.

The main difference between 36 this an other schemes that I have seen that 35 allow "make all" to be invoked 34 from any directory is the use of absolute 33 paths to allow the same paths to be interpreted 32 consistently when Make is invoked from different 31 directories. As long as these paths are 30 expressed using a variable to represent 29 the top level source directory, this does 28 not prevent one from moving the source tree, and 27 is simpler than some alternative methods 26 of achieving the same goal.

Currently, my 25 system for this project always does an "in-place" build: The 24 object file produced by compiling each source 23 file is placed in the same directory as 22 the source file. It would be straightforward 21 to enable out-of place builds by changing 20 the script that edits the gcc dependency 19 files so as to replace the absolute path 18 to the src/ dirctory by a variable $(BUILD_DIR) that 17 represents the build directory in the expression 16 for the object file target in the rule for 15 each object file.

Thus far, I've found this 14 system easy to use and maintain. The required 13 makefile fragments are short and comparatively 12 easy for collaborators to understand.

The 11 project for which I developed this system 10 is written in completely self-contained 9 ANSI C++ with no external dependencies. I 8 think that this sort of homemade non-recursive 7 makefile system is a reasonable option for 6 self-contained, highly portable code. I 5 would consider a more powerful build system 4 such as CMake or gnu autotools, however, for 3 any project that has nontrivial dependencies 2 on external programs or libraries or on 1 non-standard operating system features.

I know of at least one large scale project 6 (ROOT), which advertises using [powerpoint link] the mechanism described in Recursive 5 Make Considered Harmful. The framework exceeds 4 a million lines of code and compiles quite 3 smartly.

And, of course, all the largish 2 projects I work with that do use recursive 1 make are painfully slow to compile. ::sigh::

I written a not very good non-recursive 31 make build system, and since then a very 30 clean modular recursive make build system 29 for a project called Pd-extended. Its basically kind 28 of like a scripting language with a bunch 27 of libraries included. Now I'm also working 26 Android's non-recursive system, so that's 25 the context of my thoughts on this topic.

I 24 can't really say much about the differences 23 in performance between the two, I haven't 22 really paid attention since full builds 21 are really only ever done on the build server. I 20 am usually working either on the core language, or 19 a particular library, so I am only interested 18 in building that subset of the whole package. The 17 recursive make technique has the huge advantage 16 of making the build system be both standalone 15 and integrated into a larger whole. This 14 is important to us since we want to use 13 one build system for all libraries, whether 12 they are integrated in or written by an 11 external author.

I'm now working on building 10 custom version of Android internals, for 9 example an version of Android's SQLite classes 8 that are based on the SQLCipher encrypted 7 sqlite. So I have to write non-recursive 6 Android.mk files that are wrapping all sorts 5 of weird build systems, like sqlite's. I 4 can't figure out how to make the Android.mk 3 execute an arbitrary script, while this 2 would be easy in a traditional recursive 1 make system, from my experience.