Refactor transitive dependencies to work on all targets

[builder.git] / Readme.txt

Builder
Copyright © 2006-2012 Mikkosoft Productions
Version 2.0 readme

-------------------------------------------------------------------------------

Contents

1. Description
2. Building Builder
3. Using Builder
4. Locating packages
5. Command line options
6. Virtual targets
7. Tarballs
8. Architectures and cross-compilation
9. Dependency analysis
10. Logging
11. Writing Build files
12. Contact information
13. License

-------------------------------------------------------------------------------

1. Description

Builder is a program for building other programs, similar to make, scons and
others.  Its main design goals are minimizing duplication of configuration
between packages and being as fast as possible.  Features include integrated
configuration, dependency resolution for C and C++ sources, multi-package
builds, cross-compilation and pkg-config integration.

-------------------------------------------------------------------------------

2. Building Builder

Builder is normally built using itself.  This creates a conundrum if you just
downloaded the source and don't yet have a Builder binary.  To resolve the
problem, there's a script called bootstrap.sh in the Builder main directory.
Before running it, make sure you have the following libraries available:

MSP libraries: core datafile

Others: sigc++-2.0

You should be able to get the MSP libraries from the same place you got
Builder itself from.  Since they are also normally built with Builder, the
script will need to have their sources available.  By default, it will look at
the parent directory of builder.  You can change this with the --libpath
parameter for the script.  If everything goes well, the script will build a
builder-stage1 binary and then proceed to build a normal version with it.

-------------------------------------------------------------------------------

3. Using Builder

In the simplest and most common case, Builder is invoked in the package's root
directory without any arguments.  It will look for a file called Build and
load it.  An alternative file can be specified with the -f option.  The first
package loaded from this file is called the main package and will determine
what to build by default.

While loading the package, other packages may be pulled in through require
statements.  Refer to section 4 for how packages are located.  A package that
can't be found will cause the build to fail.  If the -x option was specified,
builder will only consider binary packages when looking for dependencies.

When building, the primary targets of each package are placed in the package's
root directory.  To build a subset of targets from the package, the desired
targets can be named on the command line.  Paths are interpreted relative to
Builder's working directory.  The analyzer can be useful in finding out what
targets are available; see section 9 for details.

To configure optional package features, arguments of the form key=value may be
given.  To obtain a list of available features, use the --help option in the
package directory.

Temporary files produced during the build are stored in a dedicated temporary
directory.  By default a directory named temp is created for each package, but
this can be changed with the --tempdir option.  Subdirectories are used to
separate files for different build types and architectures.

When multiple packages are involved in a build, Builder will need to install
files for other packages to use them.  By default these go in $HOME/local, but
this can be changed with the --prefix option.

-------------------------------------------------------------------------------

4. Locating packages

Most packages are not self-contained, but depend on other packages.  When such
dependencies are encountered, the packages need to be located.  This section
describes the various places Builder looks for them from.

First, Builder asks pkg-config if it knows the location of the source
directory of the package.  This is stored in the source variable in the .pc
file, and is automatically put there by Builder.  If the variable is found,
Builder further verifies that it refers to an existing directory which
contains a Build file, so that removed sources do not cause undue failures.

If pkg-config does not know about the source, Builder consults its own package
path.  Currently this is hardcoded to contain the main package's source
directory and its parent directory.  To be found this way, the directory
should be named the same as the package, optionally followed by a dash and a
version number (or really anything; the part after the last dash is ignored).

If the source can't be found at all, Builder will then turn to using a binary
package.  Builder's native packages are considered first.  They are normally
stored in <prefix>/share/builder with the extension bpk.  The base part of the
filename should be the as the package's name.

Finally, pkg-config is consulted for the build flags of the package in the
normal way.  The flags need to be translated to Builder's internal
representation, so if the package requires very unusual flags, it may not
work.

A package that can't be found after all these steps might as well not exist.

-------------------------------------------------------------------------------

5. Command line options

When both Builder and GNU make offer the same functionality, the command line
option to invoke it is the same as well.

-a, --analyze MODE
  Perform dependency analysis.  See section 9 for details.

-b, --build
  Perform a build.  This is the default action.  This option can be used to
  build even of some other action is specified.

-c, --clean
  Clean buildable targets.  Any dependencies are cleaned up as well.  Giving
  this option twice cleans all packages.

-f, --file=FILE
  Read build instructions from FILE.  The default is Build.

-h, --help
  Display a summary of command line options as well as configuration for the
  main package.

-j, --jobs NUM
  Run up to NUM tasks in parallel.  It's not very useful to specify a number
  far in excess of the amount of logical CPU cores in the system.

-l, --log LIST
  Enable output from a comma-separated list of log channels.  This is mainly
  useful for debugging Builder itself; for normal use the -v option is almost
  always enough.  Refer to section 10 for the available log channels.

-n, --dry-run
  Show what would be done without actually doing it.  This is useful in
  combination with -W or -l to find out what Builder would do in different
  situations.

-s, --silent
  Don't print any messages other than errors.

-t, --build-type TYPE
  Sets the build type.  The supported types are defined in the builderrc file.
  The shipped configuration contains build types debug, optimized_debug,
  release and static_release.

-v, --verbose
  Print more information about what's going on.  This option may be specified
  multiple times to enable progressively more output.  Internally it translates
  to enabling predefined sets of log channels.

-x, --no-externals
  Do not load external source packages.

-A, --conf-all
  Apply configuration to all packages.  Normally, only the main package is
  configured, but with this option all packages can be configured at once.

-B, --build-all
  Build all targets, even if their dependencies haven't been modified.

-C, --chdir DIR
  Change to DIR before doing anything else.  This is equivalent to doing cd DIR
  before invoking builder.  It's of little practical value in a shell, but may
  be useful in scripts to avoid cd commands.  All filenames on the command line
  will be interpreted relative to this directory.

-P, --progress
  Display progress while building.

-W, --what-if FILE
  Pretend that FILE has changed.

--arch ARCH
  Build for architecture ARCH.  If the architecture is not native, this will
  result in cross-compilation.  See section 8 for details.

--conf-only
  Stop after configuring packages.  This can be useful if you want to
  configure a package being used in a multi-package build.

--full-paths
  Output full paths in analysis.  Normally only target names are displayed.

--max-depth NUM
  Show up to NUM levels in analysis.  Depths less than 3 are generally not very
  useful.  The default is 4.

--prefix DIR
  Install files to DIR.  The default is $HOME/local.

--tempdir DIR
  Store temporary files in DIR.  If a relative path is specified, each package
  will have its own temp directory.  The default is ./temp.

-------------------------------------------------------------------------------

6. Virtual targets

Builder provides virtual targets for certain tasks, as well as for internal
organization of the build graph.  They can appear on the command line along
with file-based targets.

world
  Depends on everything that can be built.

default
  Depends on the targets that are built by default.  If no targets are
  specified on the command line, then this target will be built.

install
  Depends on all installed files.  This also includes .pc files, which are
  otherwise not normally built.

tarballs
  Depends on source tarballs of all packages.

goals
  Depends on all targets on the command line.  Trying to add this target to the
  command line will cause Builder to abort due to a circular dependency.

-------------------------------------------------------------------------------

7. Tarballs

Since the source tree usually contains files that do not belong to a release
archive, be they version control metadata or temporary build files, Builder
provides a way to create a clean source tarball for a package.  It will
contain all non-buildable files belonging to the package, including its Build
file.

Source tarballs are named as <package>-<version>.tar.  Currently compression
is not supported.

-------------------------------------------------------------------------------

8. Architectures and cross-compilation

If the necessary compilers and other tools are available, Builder can cross-
compile binaries for other systems.  To activate cross-compilation, use the
--arch option.

An architecture specification consists of cpu type and model, bitness and
operating system.  The fields are separated by dashes and can appear in any
order.  A partial architecture specification can also be given; any missing
fields are filled in from the native architecture.  

The --arch option can also be used to build native binaries optimized for a
specific cpu model.  Simply specify the desired model as an architecture.  By
default, no cpu model is used, which will generally result in code that can be
executed by all cpus of the same type.

In cross-compile mode, the build results are stored in a directory named after
the architecture.  Similarly, the default prefix is $HOME/local/$ARCH.

-------------------------------------------------------------------------------

9. Dependency analysis

Builder can perform a dependency analysis in various ways.  This can provide
useful insight to the relations between targets.  It may also be helpful if
you suspect Builder isn't picking up all of your files.

The following analysis modes are available:

deps
  Display most targets, but skip over those that have a predictable 1-to-1
  relationship with their sources.  Currently this includes ObjectFiles and
  InstalledFiles.

alldeps
  Display all targets.

rebuild
  Display all targets, but stop descending as soon as a target not in need of
  rebuilding is encountered.

rdeps
  Instead of displaying the dependencies of the targets on the command line,
  display which targets depend on them.  This provides similar information as
  builder -a rebuild -W target world, but displayed in a different way.

-------------------------------------------------------------------------------

10. Logging

Sometimes you need to figure out what Builder is doing, perhaps because it's
doing it wrong or you're just curious.  To this end, a number of logging
channels are provided.  To specify which channels to enable, use the -l
option.  Some channels are also turned on by the -v option; the default
verbosity level is 1.  The -s option disables all channels.

auxcommands
  Prints commands that are not directly related to building targets, such as
  pkg-config invocations.  Enabled at verbosity level 3.

cache
  Displays cached values as they are loaded.  Generates a lot of spam.

commands
  Prints commands used for building targets.  Internal commands will print
  "<internal>".  Enabled at verbosity level 2.

configure
  Shows conditionals encountered in package Build files and whether they
  matched.

environment
  Displays overall parameters used in the build.  Enabled at verbosity level 2.

files
  Indicates files that are being read or written, as well as directory scans.
  Enabled at verbosity level 3.

packagemgr
  Shows what's being done to locate packages.

packages
  Displays a list of active packages at startup, together with their target
  counts if they are source packages.  Enabled at verbosity level 2.

rebuild
  Shows why targets are being rebuilt.  Similar to -a rebuild.

summary
  Displays a brief summary of what will be done.  Enabled by default.

tasks
  Displays abstract descriptions of what is being done.  Enabled by default.

tools
  Displays information related to tools, such as versions.

vfs
  Shows what files are being looked for and where.  Generates a lot of spam.

-------------------------------------------------------------------------------

11. Writing Build files

Before going into details about Builder instruction files, it's beneficial to
understand the basic concepts used in Builder.  There are three kinds of major
entities: packages, components and targets.

Packages are distributable units of software.

Components make up the contents of a package.  They define the high-level
goals such as libraries and executables.

Targets are the individual files taking part in the build process.  They are
created automatically from components.

The canonical name for the instruction file is Build.  It is written in a
structured declarative language, with syntax somewhat resembling the C
language.  The basic unit is a statement, which consists of a keyword,
optional arguments and an optional block of sub-statements.  Each statement is
terminated with a semicolon.

Typically, the Build file for a project defines a single package:

  package "mylib"
  {
    ...
  };

Inside the package can be some informational statements:

  version "0.1";
  description "My awesome library";

Packages may also depend on other packages:

  require "otherlib";

See section 4 for how packages are located.

To be useful, a package must define one or more components.  At present, there
are four types:
- program: Build an executable program
- library: Build a library (both shared and static)
- module: Build a module for a program
- tarball: Pack files up in a .tar file

Each of these statements takes the component name as an argument.  The final
target name is composed of the component name and a possible platform-specific
prefix and suffix.  For example, the statement:

  library "mylib"
  {
    ...
  };

Will create a file called libmylib.so on Linux and libmylib.dll on Windows.

Components must contain one or more source statements telling where sources
are found:

  source "mylib.cpp";

Directories may be specified; each file inside the directory will be added to
the source list.  Subdirectories are not included.  The list is filtered
according to the component type and only applicable files are used.

If you want a component to be installed, you can state it:

  install true;

The default is not to install any components.  The installation directory is
automatically determined from the component type.  Library components will also
install any headers which were used to build the library.

Components can have requirements, just like packages.  They can also use
library components from the same package:

  use "mylib_common";

This can be useful in organizing code when multiple components in a package
share a common part.  If the used library is not specified to be installed,
it will be linked statically.  

Packages may want to offer optional features, for example to allow the user to
choose whether to use a particular external library:

  feature "foolib"
  {
    description "Use foolib to perform magic";
  };

Enabled features will automatically cause a macro to be defined when compiling
C or C++ code.  The macro consists of a WITH_ prefix followed by the feature
name in uppercase.  The above feature would define the macro WITH_FOOLIB.

To specify additional effects of a feature, such as required packages, a
conditional statement may be used:

  if_feature "foolib"
  {
    ...
  };

Any statements inside the conditional block are evaluated if the feature is
enabled.  This can be negated with a !.  Conditionals can appear at package and
component scopes and may contain anything that the enclosing statement can.

When making cross-platform software, it's often necessary to use different
libraries on different platforms.  Another kind of conditional can be used for
this:

  if_arch "windows"
  {
    ...
  };

See section 8 for more information on how to specify an architecture.  The
conditional block is evaluated if all parts in the condition match the target
architecture.

Sometimes a library doesn't support pkg-config and can't be pulled in through a
require statement.  This is commonly the case with system libraries.  The
build_info statement can be used for this:

  build_info
  {
    library "gdi32";
  };

In many cases these libraries are specific to that system, so it's best to wrap
them in an if_arch statement.

Another case for using build_info is if you want to keep your headers separate
from source files:

  incpath "include";

Paths are interpreted relative to the package's root directory.  It is not
recommended to use absolute paths, as this can make your package difficult to
build for others.

To include additional files in the package's source tarball, such as
documentation, use the source_tarball statement.  It takes no argument, as the
name is generated internally, but otherwise it behaves as a tarball component.

For real-life examples, see the Build files in Builder itself or any of the
MSP libraries.

-------------------------------------------------------------------------------

12. Contact information

The latest releases can be found at http://www.tdb.fi/builder.shtml

Bug reports, feature requests etc. may be sent to tdb@tdb.fi

Read-only git access is available at git://git.tdb.fi/builder

-------------------------------------------------------------------------------

13. License

Builder is copyright © 2006-2012  Mikko Rasa, Mikkosoft Productions

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

Full license text can be found in License.txt.