compiletools

ct-cake --auto

Authors

Zomojo Pty, Ltd. - Geoff Ericksson (geoff@zomojo.com)

Download

You can download this project from releases

You can also clone the project with Git by running:

$ git clone git://github.com/Zomojo/compiletools

Or pip install using

pip install compiletools

Dependencies

sudo dnf install python2-setuptools python2-configargparse python2-appdirs python2-psutil

sudo dnf install python3-setuptools python3-configargparse python3-appdirs python3-psutil

Getting Started

The ct-* tools can run from the cloned git repository. Simply edit your PATH variable to include the repo. For example, if you cloned into your home directory then

PATH=~/compiletools:$PATH

#include <stdio.h>
#include <pthread.h>
#include <iostream>
//#LDFLAGS=-pthread -lrt

static void pclock(const char *msg, clockid_t cid)
{
   struct timespec ts;
   printf("%s", msg);
   if (clock_gettime(cid, &ts) == -1)
       perror("clock_gettime");
   printf("%4ld.%03ld\n", ts.tv_sec, ts.tv_nsec / 1000000);
}
    
    
int main()
{
    clockid_t cid;
    pthread_t pt = pthread_self();
        
    if (pthread_getcpuclockid(pt, &cid))
        perror("getcpuclockid failed");
        
    pclock("the clock is", cid);
    return 0;
}

$ ct-cake --auto && ./bin/example

make -s -j 3 -f /home/geoff/tmp/bin/gcc.debug/Makefile build
... /home/geoff/tmp/example.cpp
... /home/geoff/tmp/bin/gcc.debug/example
bin/example
the clock is   0.001

What occured was ct-cake searched all the files to find a source file containing the word "main". It then searched the source file to extract the link flags that are annotated in the source code, so it wasn't necessary to supply them at the command-line. compiletools is based on the principle that the source code and the build settings are kept together. This will increase the likelihood of keeping the compile flags in sync with the header includes. A Makefile was created that expressed the required build and then make was called. Finally the execuatable was executed.

Overview

The compiletools project provides a variety of tools for building C/C++ projects and for exploring code dependencies. ct-cake is the Swiss army knife of build tools that combines many of the compiletools into one uber-tool. For many C/C++ projects you can compile simply using

ct-cake --auto

Cake works off the same principles as Ruby on Rails. It will make your life easy if you don't arbitrarily name things. The main rules are:

1. ct-cake only builds C and C++. Everything can be done just fine with other tools, so there's no point reinventing them. Anyway, it's easy to embed cake into other toolchains, see the last section.
2. All binaries end up in the bin directory, with the same base name as their source filename. You can override this at the command-line, but it's against the spirit of the tool.
3. The implementation file for point.hpp should be called point.cpp. This is so ct-cake can compile it and recursively hunt down its dependencies.
4. If a header or implementation file will not work without being linked with a certain flag, add a //#LDFLAGS=myflag directly to the source code.
5. Likewise, if a special compiler option is needed, use //#CXXFLAGS=myflag.
6. Minimise the use of "-I" include flags. They make it hard not only for cake to generate dependencies, but also autocomplete tools like Eclipse and ctags. You can avoid -I flags by structuring your code in the same way you refer to paths in your source code.
7. Currently only varieties of Linux are actively supported because that's what the compiletools developers have easy access to. That said, it stands a good chance of working on *BSD and macOS. We are interested in receiving patches for other platforms including Windows.

ct-cake works off a "pull" philosophy of building, unlike the "push" model of most build processes. Often, there is the monolithic build script that rebuilds everything. Users iterate over changing a file, relinking everything and then rerunning their binary. A hierarchy of libraries is built up and then linked in to the final executables. All of this takes a lot of time, particularly for C++.

With ct-cake, you only pull in what is strictly necessary to what you need to run right now. Say, you are testing a particular tool in a large project, with a large base of 2000 library files for string handling, sockets, etc. There is simply no make file. You might want to create a build.sh for regression testing, but it's not essential.

The basic workflow is to simply type:

ct-cake --auto

   ct-cake path/to/src/app.cpp

Only the library cpp files that are needed, directly, or indictly to create ./bin/app are actually compiled. If you don't #include anything that refers to a library file, you don't pay for it. Also, only the link options that are strictly needed to generate the app are included. Its possible to do in make files, but such fine-level granularity is rarely set up in practice, because its too error-prone to do manually, or with recursive make goodness.

How it Works

ct-cake generates the header dependencies for the "main.cpp" file you specify at the command line by either examining the "#include" lines in the source code or by executing "gcc -MM -MF". For each header file found in the source file, it looks for an underlying implementation (cpp) file with the same name, and adds that implementation file to the build. ct-cake also reads the first 8KB of each file for special comments that indicate needed link and compile flags. Then it recurses through the dependencies of the cpp file, and uses this spidering to generate complete dependency information for the application. A Makefile is generated and finally it calls make.

Magic Comments

ct-ake works very differently to other build systems, which specify a hierarchy of link flags and compile options, because ct-cake ties the compiler flags directly to the source code. If you have compress.hpp that requires "-lzip" on the link line, add the following comment in the first 8KB of the header file:

    //#LDFLAGS=-lzip

Whenever the header is included (either directly or indirectly), the -lzip will be automatically added to the link step. If you stop using the header, for a particular executable, cake will figure that out and not link against it.

If you want to compile a cpp file with a particular optimization enabled, add, say:

    //#CXXFLAGS=-fstrict-aliasing

Performance

Because ct-cake internally generates a makefile to build the C++ file, cake is about as fast as a handrolled Makefile that uses the same lazily generated dependencies. A typical project takes 0.04 seconds to build if nothing is out of date, versus 2 seconds for, say, Boost.Build.

ct-cake also eliminates the redundant generation of static archive files that a more hierarchical build process would generate as intermediaries, saving the cost of running 'ar'.

Note that ct-cake doesn't build all cpp files that you have checked out, only those strictly needed to build your particular binary, so you only pay for what you use. This difference alone should see a large improvement on most projects, especially for incremental rebuilds.

Cake configuration

The compiletools programs require almost no configuration. However, it is still useful to have some shortcut build templates such as 'release', 'profile' etc.

Config files for the ct-* applications are programmatically located using python-appdirs, which on linux is a wrapper around the XDG specification. Thus default locations are /etc/xdb/ct/ and $HOME/.config/ct/. Configuration parsing is done using python-configargparse which automatically handles environment variables, command line arguments, system configs and user configs. Specifically, the config files are searched for in the following locations (from lowest to highest priority):

• same path as exe,
• system config (XDG compliant, so usually /etc/xdg/ct)
• python virtual environment system configs (${python-site-packages}/etc/xdg/ct)
• user config (XDG compliant, so usually ~/.config/ct)

    command line > environment variables > config file values > defaults

ID=GNU
CC=gcc
CXX=g++
LD=g++
CFLAGS=-fPIC -g -Wall -O3 -DNDEBUG -finline-functions -Wno-inline
CXXFLAGS=-std=c++11 -fPIC -g -Wall -fdiagnostics-color=auto -O3 -DNDEBUG -finline-functions -Wno-inline
LDFLAGS=-fPIC -Wall -Werror -Xlinker --build-id
TESTPREFIX=timeout 300 valgrind --quiet --error-exitcode=1

CXXFLAGS lists the flags appended to each compilation job. The value in /etc/xdg/ct/gcc.debug.conf is completely overridden by the environment variable, which is completely overridden by the command-line argument --CXXFLAGS=. Likewise, LDFLAGS sets the default options used for linking.

TESTPREFIX specifies a command prefix to place in front of unit test runs. This should ideally be a tool like valgrind, gdb or purify that can be configured to execute the app and return a non-zero exit code on any failure.

Build variants

    $ ct-cake --variant=release a.cpp && ./bin/a

Unit Tests

ct-cake integrates with unit tests in a fairly simple (and perhaps simplistic) way.

     $ ct-cake utilities/*.cpp    # builds all apps and places them under bin/

     $ ct-cake utilities/*.cpp --tests tests/*.cpp

export TESTPREFIX="valgrind --quiet --error-exitcode=1"

Putting it all together - a typical build setup

For most simple projects, a build.sh script that looks like the following is quite useful. You can simply add more cpp to the apps directory to generate more tools from the project, or add test scripts to the regression directory to improve test coverage.

Code generation steps can be added at the beginning of the build.sh, before cake runs.

#!/bin/bash

set -e

ct-cake apps/*.cpp --tests tests/*.cpp "$@"

The special "$@" marker is the recommended way of forwarding arguments to an application. You can then run the build script like this:

    $ ./build.sh --variant=release

    $ ./build.sh --variant=release --append-CXXFLAGS=-DSPECIALMODE