Table of Contents
For a number of reasons, you may want to build binary packages for a large selected set of packages in pkgsrc, or even for all pkgsrc packages. For instance, when you have multiple machines that should run the same software, it is wasted time if they all build their packages themselves from source. Or you may want to build a list of packages you want and check them before deploying onto production systems. There is a way of getting a set of binary packages: the bulk build system, or pbulk ("p" stands for "parallel"). This chapter describes how to set it up.
First of all, you have to decide whether you build all packages or a limited set of them. Full bulk builds usually consume a lot more resources, both space and time, than builds for some practical sets of packages. A number of particularly heavy packages exist that are not actually interesting to a wide audience. (The approximate resource consumption for a full bulk build is given in section Section 8.3, “Requirements of a full bulk build”.) For limited bulk builds you need to make a list of packages you want to build. Note that all their dependencies will be built, so you don't need to track them manually.
During bulk builds various packages are installed and deinstalled
in /usr/pkg (or whatever LOCALBASE is),
so make sure that you don't need any package during the builds.
Essentially, you should provide a fresh system, either a chroot environment
or something even more restrictive, depending on what the operating system provides,
or dedicate the whole physical machine.
As a useful side effect this makes sure that bulk builds cannot
break anything in your system. There have been numerous cases where
certain packages tried to install files outside the
LOCALBASE or wanted to edit some files in
/etc.
Running a bulk build works roughly as follows:
First, build the pbulk infrastructure in a fresh pkgsrc location.
Then, build each of the packages from a clean installation directory using the infrastructure.
To simplify configuration, we provide the helper script mk/pbulk/pbulk.sh.
In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine). Configure network access to fetch distribution files. Create a user with name "pbulk".
Fetch and extract pkgsrc. Use a command like one of these:
#(cd /usr && ftp -o - https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)#(cd /usr && fetch -o - https://cdn.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)#(cd /usr && cvs -Q -z3 -d anoncvs@anoncvs.NetBSD.org:/cvsroot get -P pkgsrc)
Or any other way that fits (e.g., curl, wget).
Deploy and configure pbulk tools, e.g.:
#sh pbulk.sh -n # use native make, no bootstrap kit needed (for use on NetBSD)#sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment#sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build
mk.conf.frag is a fragment of
mk.conf that contains settings you want to
apply to packages you build. For instance,
PKG_DEVELOPER= yes # perform more checks
X11_TYPE= modular # use pkgsrc X11
SKIP_LICENSE_CHECK= yes # accept all licences (useful
# when building all packages)
If configured for limited list, replace the list in /usr/pbulk/etc/pbulk.list
with your list of packages, one per line without empty lines or comments. E.g.:
www/firefox mail/thunderbird misc/libreoffice4
At this point you can also review configuration in /usr/pbulk/etc
and make final amendments, if wanted.
Start it:
#/usr/pbulk/bin/bulkbuild
After it finishes, you'll have /mnt filled with distribution files, binary packages, and reports,
plain text summary in /mnt/bulklog/meta/report.txt
The pbulk.sh script does not cover all possible use cases.
While being ready to run, it serves as a good starting point to understand and build more complex setups.
The script is kept small enough for better understanding.
The pbulk.sh script supports running
unprivileged bulk build and helps configuring distributed bulk builds.
Distributed bulk builds support either building in worker chroots
(each node is a path to a different chroot)
that replicate the target system, including the pbulk prefix,
or remote machines (each node is
an IP address that must be accessible over SSH without a password).
A complete bulk build requires lots of disk space. Some of the disk space can be read-only, some other must be writable. Some can be on remote filesystems (such as NFS) and some should be local. Some can be temporary filesystems, others must survive a sudden reboot.
70 GB for the distfiles (read-write, remote, temporary)
60 GB for the binary packages (read-write, remote, permanent)
1 GB for the pkgsrc tree (read-only, remote, permanent)
5 GB for LOCALBASE (read-write, local, temporary)
10 GB for the log files (read-write, remote, permanent)
5 GB for temporary files (read-write, local, temporary)
To ensure that pkgsrc packages work in different configurations, it makes sense to run non-default bulk builds from time to time. This section lists some ideas for bulk builds that intentionally let packages fail if they don't follow the pkgsrc style.
Add the following line to mk.conf.
GNU_CONFIGURE_STRICT= yes
When a package fails this additional check, the most common cause is that the configure option was valid for an older version of the package but does not apply anymore. In that case, just remove it.
The job of a compiler is not restricted to producing executable code, most compilers also detect typical programming mistakes. The pkgsrc compiler wrappers make it easy to force compiler options when the package is built. This can be used to find typical bugs across all packages that are in pkgsrc. By reporting these bugs upstream, the packages will be more reliable with the next updates.
Add some of the following lines to mk.conf:
CFLAGS+= -Werror=char-subscripts CFLAGS+= -Werror=implicit-function-declaration
When a package fails to build using these stricter compiler options, document the circumstances in which the compiler produced the error message. This includes:
The platform
(MACHINE_PLATFORM)
The source file
An excerpt of the code. GCC and Clang already do this as part of the diagnostic.
The exact error message from the compiler.
If a package produces these error messages, but the package is
fine, record this in your local mk.conf, like this, to skip this check
in the next builds:
.if ${PKGPATH} == category/package
# Version ${VERSION} failed on ${MACHINE_PLATFORM}:
# error message
# code
# Reason why the code does not need to be fixed.
BUILDLINK_TRANSFORM+= rm:-Werror=char-subscripts
.endif
If the error messages from the compiler are valid and the code
needs to be fixed, prepare a local patch (see
LOCALPATCHES) and report the bug to the upstream
authors of the package, providing them with the information you collected
above.
Patches that are not essential for the package to work should only be reported upstream but not committed to pkgsrc, to make future updates easier.
When adding custom compiler flags via CFLAGS,
these apply to all phases of the package build process. Especially in the
configure phase, adding -Werror leads to wrong
decisions. The GNU configure scripts feed many small test programs to the
C compiler to see whether certain headers are available, functions are
defined in a library and programs can be run. In many cases these
programs would not survive a strict compiler run with -Wall
-Wextra -Werror.
The pkgsrc infrastructure is flexible enough to support compiler
options being added between the configure and
build phases. It's a little more complicated than the
other examples in this section but still easy enough.
The basic idea is to use the pkgsrc compiler wrapper to inject the
desired compiler options. The compiler wrapper's original task is to hide
unwanted directories of include files and to normalize compiler options.
It does this by wrapping the compiler command and rewriting the command
line. To see this in action, run bmake patch in a
package directory and examine the
work/.cwrappers/config directory. It contains
individual configurations for the C compiler and the related tools. The
plan is to find a hook between the configure and build phases, and to
modify these configuration files at that point.
To find this hook, have a look at
mk/build/build.mk, which contains among others the
pre-build-checks-hook. The word
checks doesn't quite fit, but the
pre-build-hook sounds good enough.
The code to be included in mk.conf is:
# Just a few example options.
BUILD_ONLY_CFLAGS= -Wall -Werror -O2 -DTEMPDIR='"/tmp"'
.if ${BUILD_ONLY_CFLAGS:U:M*}
pre-build-checks-hook: add-build-only-cflags
add-build-only-cflags: .PHONY
${RUN} cd ${CWRAPPERS_CONFIG_DIR}; \
${TEST} ! -f ${.TARGET} || exit 0; \
for flag in ${BUILD_ONLY_CFLAGS}; do \
${ECHO} "append=$$flag" >> cc; \
done; \
> ${.TARGET}
.endif
(When editing the mk.conf, make sure that the commands of the
add-build-only-cflags target are indented with a tab,
not with spaces.)
The condition in the .if statement contains the
:U modifier to prevent parse errors if the variable
should be undefined, possibly because it is only defined for a subset of
the packages. The :M* modifier ensures that there is
at least one compiler option, to prevent a syntax error in the shell
parser.
The code around the ${.TARGET} variable ensures
that the additional compiler options are only appended once, even if
bmake build is run multiple times. To do this, it
creates a marker file.
To verify that this setup works, run bmake
configure in a package directory. Up to now, everything works
as usual. Examine the directory
work/.cwrappers/config to see that the compiler
options from BUILD_ONLY_CFLAGS are not yet added to
the file cc. Examine the tail of the
work/.work.log file to see that the normal compiler
options are used.
Now run bmake build. This will append the
options to the file cc and will create the marker
file in the same directory. After that, the build starts as usual, but
with the added compiler options. Examine the tail of the file
work/.work.log to see that the lines starting with
[*] don't contain the compiler options, but the
corresponding lines starting with <.> do end
with these options.
Building packages using this setup variant and fixing the resulting bugs is the same as in Section 8.4.2, “Detect classes of bugs by forcing compiler warnings”.
Some directories like PREFIX,
VARBASE, PKG_SYSCONFDIR,
PKGMANDIR, PKG_INFODIR can be
configured in pkgsrc. Set these to arbitrary paths during bootstrap or
afterwards in mk.conf.
PREFIX= /a-random-uuid PKG_SYSCONFDIR= /a-random-uuid VARBASE= /a-random-uuid PKGMANDIR= a-random-uuid PKG_INFODIR= a-random-uuid
When building a package, warnings are typically ignored since they
just flow by and do not cause the build to fail immediately. To find
these warnings, redefine them to errors in mk.conf.
DELAYED_WARNING_MSG= ${DELAYED_ERROR_MSG} "(was warning)"
WARNING_MSG= ${FAIL_MSG} "(was warning)"
(There are many more classes of warnings in pkgsrc, and most of them can be redefined with a simple definition like above.
If a package suggests to add USE_TOOLS+=perl to
the package Makefile, research whether the package actually needs Perl.
If it does, add USE_TOOLS+=perl to the package
Makefile, and if it doesn't, add
TOOLS_BROKEN+=perl.
Using pkglint as part of the regular build process is mostly a waste of time. If you want to fix some of the warnings, just run pkglint recursively on the whole pkgsrc tree. This will take a few minutes (up to 10), which is much faster than a complete bulk build.
To ensure that the binary packages don't contain references to the
build directory, there is already CHECK_WRKREF. If
that variable includes the item extra, it is
possible to define additional patterns that must not appear in any
installed file. This is specified in mk.conf.
CHECK_WRKREF= extra CHECK_WRKREF_EXTRA_DIRS+= /usr/local CHECK_WRKREF_EXTRA_DIRS+= /usr/pkg CHECK_WRKREF_EXTRA_DIRS+= @[A-Z][A-Z]*@
The above patterns will probably generate many false positives, therefore the results need to be taken with a grain of salt.
To run the test suites that come with each package, add this line
to mk.conf.
PKGSRC_RUN_TEST= yes
Be prepared that even the most basic packages fail this test. When doing a bulk build with this, it will often abort in the early phase where the packages are scanned for their dependencies since there are cyclic dependencies. There is still a lot to do in this area.
To catch typos in the shell snippets from the Makefile fragments,
add the -u flag to most of the commands by adding this
line to mk.conf.
RUN= @set -eu;
After that, ensure that none of the bulk build log files (even
those for successfully built packages) contains the string
parameter not set or whatever error message the
command sh -ceu '$undefined' outputs.
See mk/misc/common.mk for the existing
definition.
The build logs of a package are often quite long. This allows error
messages or other interesting details to hide between the noise. To make
the actual error message stand out more, add these lines to
mk.conf.
GNU_CONFIGURE_QUIET= yes MAKE_FLAGS+= -s
The -s option works for both GNU Make and BSD
Make. On exotic platforms with their own make, it may be a little
different.
After your pkgsrc bulk-build has completed, you may wish to
create a CD-ROM set of the resulting binary packages to assist
in installing packages on other machines. The
pkgtools/cdpack package provides
a simple tool for creating the ISO 9660 images.
cdpack arranges the packages on the CD-ROMs in a
way that keeps all the dependencies for a given package on the same
CD as that package.
Complete documentation for cdpack is found in the cdpack(1)
man page. The following short example assumes that the binary
packages are left in
/usr/pkgsrc/packages/All and that
sufficient disk space exists in /u2 to
hold the ISO 9660 images.
#mkdir /u2/images#pkg_add /usr/pkgsrc/packages/All/cdpack#cdpack /usr/pkgsrc/packages/All /u2/images
If you wish to include a common set of files
(COPYRIGHT, README,
etc.) on each CD in the collection, then you need to create a
directory which contains these files, e.g.:
#mkdir /tmp/common#echo "This is a README" > /tmp/common/README#echo "Another file" > /tmp/common/COPYING#mkdir /tmp/common/bin#echo "#!/bin/sh" > /tmp/common/bin/myscript#echo "echo Hello world" >> /tmp/common/bin/myscript#chmod 755 /tmp/common/bin/myscript
Now create the images:
#cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images
Each image will contain README,
COPYING, and bin/myscript
in their root directories.