This chapter defines some of the basic concepts that the Buildbot uses. You’ll need to understand how the Buildbot sees the world to configure it properly.
В этой главе определяются некоторые из основных концепций, которые использует Buildbot. Вам нужно понять, как Buildbot видит мир, чтобы правильно его настроить.
Build is a single compile or test run of a particular version of the source code, and is comprised of a series of steps. The steps may be arbitrary. For example, for compiled software a build generally consists of the checkout, configure, make, and make check sequence. For interpreted projects like Python modules, a build is generally a checkout followed by an invocation of the bundled test suite.
BuildFactory describes the steps a build will perform. The builder which starts a build uses its configured build factory to determine the build’s steps.
A Change is an abstract way Buildbot uses to represent a single change to the source files performed by a developer. In version control systems that support the notion of atomic check-ins a change represents a changeset or commit.
Change comprises the following information:
- the developer that is responsible for the change
- the list of files that the change added, removed or modified
- the message of the commit
- the repository, the codebase and the project that the change corresponds to
- the revision and the branch of the commit
Buildbot uses the concept of source stamp set to identify exact source code that needs to be built for a certain project. A source stamp set is a collection of one or more source stamps.
A source stamp is a collection of information needed to identify a particular version of code on a certain codebase. This information most often is a revision and possibly a branch.
A codebase is a collection of related files and their history tracked as a unit by version control systems. A single codebase may appear in multiple repositories which themselves are identified by URLs. For example,
http://hg.mozilla.org/mozilla-release both contain the Firefox codebase, although not exactly the same code.
A project corresponds to a set of one or more codebases that together may be built and produce some end artifact. For example, a company may build several applications based on the same core library. The “app” codebase and the “core” codebase are in separate repositories, but are compiled together and constitute a single project. Changes to either codebase should cause a rebuild of the application.
A revision is an identifier used by most version control systems to uniquely specify a particular version of the source code. Sometimes in order to do that a revision may make sense only if used in combination with a branch.
To sum up the above, to build a project, Buildbot needs to know exactly which version of each codebase it should build. It uses a source stamp to do so for each codebase, each of which informs Buildbot that it should use a specific revision from that codebase. Collectively these source stamps are called source stamp set for each project.
Buildbot supports a significant number of version control systems, so it treats them abstractly.
For purposes of deciding when to perform builds, Buildbot’s change sources monitor repositories, and represent any updates to those repositories as changes. These change sources fall broadly into two categories: pollers which periodically check the repository for updates; and hooks, where the repository is configured to notify Buildbot whenever an update occurs. For more information see Change Sources and Changes and How Different VC Systems Specify Sources.
When it comes time to actually perform a build, a scheduler prepares a source stamp set, as described above, based on its configuration. When the build begins, one or more source steps use the information in the source stamp set to actually check out the source code, using the normal VCS commands.
Each Buildmaster has a set of scheduler objects, each of which gets a copy of every incoming
Change. The Schedulers are responsible for deciding when
Builds should be run. Some Buildbot installations might have a single scheduler, while others may have several, each for a different purpose.
For example, a quick scheduler might exist to give immediate feedback to developers, hoping to catch obvious problems in the code that can be detected quickly. These typically do not run the full test suite, nor do they run on a wide variety of platforms. They also usually do a VC update rather than performing a brand-new checkout each time.
A separate full scheduler might run more comprehensive tests, to catch more subtle problems. It might be configured to run after the quick scheduler, to give developers time to commit fixes to bugs caught by the quick scheduler before running the comprehensive tests. This scheduler would also feed multiple
Many schedulers can be configured to wait a while after seeing a source-code change - this is the tree stable timer. The timer allows multiple commits to be “batched” together. This is particularly useful in distributed version control systems, where a developer may push a long sequence of changes all at once. To save resources, it’s often desirable only to test the most recent change.
Schedulers can also filter out the changes they are interested in, based on a number of criteria. For example, a scheduler that only builds documentation might skip any changes that do not affect the documentation. Schedulers can also filter on the branch to which a commit was made.
There is some support for configuring dependencies between builds - for example, you may want to build packages only for revisions which pass all of the unit tests. This support is under active development in Buildbot, and is referred to as “build coordination”.
Periodic builds (those which are run every N seconds rather than after new Changes arrive) are triggered by a special
Each scheduler creates and submits
BuildSet objects to the
BuildMaster, which is then responsible for making sure the individual
BuildRequests are delivered to the target
Scheduler instances are activated by placing them in the
schedulers list in the buildmaster config file. Each scheduler must have a unique name.
BuildSet is the name given to a set of
Builds that all compile/test the same version of the tree on multiple
Builders. In general, all these component
Builds will perform the same sequence of
Steps, using the same source code, but on different platforms or against a different set of libraries.
BuildSet is tracked as a single unit, which fails if any of the component
Builds have failed, and therefore can succeed only if all of the component
Builds have succeeded. There are two kinds of status notification messages that can be emitted for a
firstFailure type (which fires as soon as we know the
BuildSet will fail), and the
Finished type (which fires once the
BuildSet has completely finished, regardless of whether the overall set passed or failed).
BuildSet is created with set of one or more source stamp tuples of
(branch, revision, changes, patch), some of which may be
None, and a list of
Builders on which it is to be run. They are then given to the BuildMaster, which is responsible for creating a separate
BuildRequest for each
There are a couple of different likely values for the
(revision=None, changes=CHANGES, patch=None)
SourceStampused when a series of
Changes have triggered a build. The VC step will attempt to check out a tree that contains CHANGES (and any changes that occurred before CHANGES, but not any that occurred after them.)
(revision=None, changes=None, patch=None)
SourceStampthat would be used on a
Buildthat was triggered by a user request, or a
Periodicscheduler. It is also possible to configure the VC Source Step to always check out the latest sources rather than paying attention to the
Changes in the
SourceStamp, which will result in same behavior as this.
(branch=BRANCH, revision=None, changes=None, patch=None)
(revision=REV, changes=None, patch=(LEVEL, DIFF, SUBDIR_ROOT))
patch -pLEVEL <DIFF) from inside the relative directory SUBDIR_ROOT. Item SUBDIR_ROOT is optional and defaults to the builder working directory. The
trycommand creates this kind of
None, the patching step is bypassed.
The buildmaster is responsible for turning the
BuildSet into a set of
BuildRequest objects and queueing them on the appropriate
BuildRequest is a request to build a specific set of source code (specified by one or more source stamps) on a single
Builder runs the
BuildRequest as soon as it can (i.e. when an associated worker becomes free).
BuildRequests are prioritized from oldest to newest, so when a worker becomes free, the
Builder with the oldest
BuildRequest is run.
BuildRequest contains one
SourceStamp specification per codebase. The actual process of running the build (the series of
Steps that will be executed) is implemented by the
Build object. In the future this might be changed, to have the
Build define what gets built, and a separate
BuildProcess (provided by the Builder) to define how it gets built.
BuildRequest may be mergeable with other compatible
BuildRequests. Builds that are triggered by incoming
Changes will generally be mergeable. Builds that are triggered by user requests are generally not, unless they are multiple requests to build the latest sources of the same branch. A merge of buildrequests is performed per codebase, thus on changes having the same codebase.
Builder handles the process of scheduling work to workers. Each
Builder is responsible for a certain type of build, which usually consist of identical or very similar sequence of steps.
The class serves as a kind of queue for that particular type of build. In general, each
Builder runs independently, but it’s possible to constrain the behavior of
Builders using various kinds of interlocks.
Each builder is a long-lived object which controls a sequence of
Builder is created when the config file is first parsed, and lives forever (or rather until it is removed from the config file). It mediates the connections to the workers that do all the work, and is responsible for creating the
Build objects - Builds.
Each builder gets a unique name, and the path name of a directory where it gets to do all its work. This path is used in two ways. On the buildmaster-side a directory is created for keeping status information. On the worker-side a directory is created where the actual checkout, compile and test commands are executed.
A builder also has a
BuildFactory, which is responsible for creating new
Build instances: because the
Build instance is what actually performs each build, choosing the
BuildFactory is the way to specify what happens each time a build is done (Builds).
Workers corresponds to an environment where builds are executed. A single physical machine must run at least one
Workers in order for Buildbot to be able to utilize it for running builds. Multiple
Workers may run on a single machine to provide different environments that can reuse the same hardware by means of containers or virtual machines.
Each builder is associated with one or more
Workers. For example, a builder which is used to perform macOS builds (as opposed to Linux or Windows builds) should naturally be associated with a Mac worker.
If multiple workers are available for any given builder, you will have some measure of redundancy: in case one worker goes offline, the others can still keep the
Builder working. In addition, multiple workers will allow multiple simultaneous builds for the same
Builder, which might be useful if you have a lot of forced or
try builds taking place.
Worker that is configured for a builder should be identical. Otherwise build or test failures will be dependent on which worker the build is ran and this will complicate investigation of failures.
Buildbot has a somewhat limited awareness of users. It assumes the world consists of a set of developers, each of whom can be described by a couple of simple attributes. These developers make changes to the source code, causing builds which may succeed or fail.
Users also may have different levels of authorization when issuing Buildbot commands, such as forcing a build from the web interface or from an IRC channel.
Each developer is primarily known through the source control system. Each
Change object that arrives is tagged with a
who field that typically gives the account name (on the repository machine) of the user responsible for that change. This string is displayed on the HTML status pages and in each
To do more with the User than just refer to them, this username needs to be mapped into an address of some sort. The responsibility for this mapping is left up to the status module which needs the address. In the future, the responsibility for managing users will be transferred to User Objects.
who fields in
git Changes are used to create User Objects, which allows for more control and flexibility in how Buildbot manages users.
Each build has a set of Build Properties, which can be used by its build steps to modify their actions. These properties, in the form of key-value pairs, provide a general framework for dynamically altering the behavior of a build based on its circumstances.
Properties form a simple kind of variable in a build. Some properties are set when the build starts, and properties can be changed as a build progresses – properties set or changed in one step may be accessed in subsequent steps. Property values can be numbers, strings, lists, or dictionaries - basically, anything that can be represented in JSON.
Properties are very flexible, and can be used to implement all manner of functionality. Here are some examples:
Most Source steps record the revision that they checked out in the
got_revision property. A later step could use this property to specify the name of a fully-built tarball, dropped in an easily-accessible directory for later testing.
In builds with more than one codebase, the
got_revision property is a dictionary, keyed by codebase.
Some projects want to perform nightly builds as well as building in response to committed changes. Such a project would run two schedulers, both pointing to the same set of builders, but could provide an
is_nightly property so that steps can distinguish the nightly builds, perhaps to run more resource-intensive tests.
Some projects have different build processes on different systems. Rather than create a build factory for each worker, the steps can use worker properties to identify the unique aspects of each worker and adapt the build process dynamically.
What if an end-product is composed of code from several codebases? Changes may arrive from different repositories within the tree-stable-timer period. Buildbot will not only use the source-trees that contain changes but also needs the remaining source-trees to build the complete product.
For this reason a Scheduler can be configured to base a build on a set of several source-trees that can (partly) be overridden by the information from incoming
As described above, the source for each codebase is identified by a source stamp, containing its repository, branch and revision. A full build set will specify a source stamp set describing the source to use for each codebase.
Configuring all of this takes a coordinated approach. A complete multiple repository configuration consists of:
a codebase generator
Every relevant change arriving from a VC must contain a codebase. This is done by a
codebaseGeneratorthat is defined in the configuration. Most generators examine the repository of a change to determine its codebase, using project-specific rules.
schedulerhas to be configured with a set of all required
codebasesto build a product. These codebases indicate the set of required source-trees. In order for the scheduler to be able to produce a complete set for each build, the configuration can give a default repository, branch, and revision for each codebase. When a scheduler must generate a source stamp for a codebase that has received no changes, it applies these default values.
multiple source steps - one for each codebase
A Builders’s build factory must include a source step for each codebase. Each of the source steps has a
codebaseattribute which is used to select an appropriate source stamp from the source stamp set for a build. This information comes from the arrived changes or from the scheduler’s configured default values.
Each source step has to have its own
workdirset in order for the checkout to be done for each codebase in its own directory.
Ensure you specify the codebase within your source step’s Interpolate() calls (ex.
http://.../svn/%(src:codebase:branch)s). See Interpolate for details.
codebaseGenerator that returns non-empty (not
'') codebases will change the behavior of all the schedulers.
Buildbot Multimaster is considered experimental. There are still some companies using it in production. Don’t hesitate to use the mailing lists to share your experience.
Buildbot supports interconnection of several masters. This has to be done through a multi-master enabled message queue backend. As of now the only one supported is wamp and crossbar.io. see wamp
There are then several strategy for introducing multimaster in your buildbot infra. A simple way to say it is by adding the concept of symmetrics and asymmetrics multimaster (like there is SMP and AMP for multi core CPUs)
Symmetric multimaster is when each master share the exact same configuration. They run the same builders, same schedulers, same everything, the only difference is that workers are connected evenly between the masters (by any means (e.g. DNS load balancing, etc)) Symmetric multimaster is good to use to scale buildbot horizontally.
Asymmetric multimaster is when each master have different configuration. Each master may have a specific responsibility (e.g schedulers, set of builder, UI). This was more how you did in 0.8, also because of its own technical limitations. A nice feature of asymmetric multimaster is that you can have the UI only handled by some masters.
Separating the UI from the controlling will greatly help in the performance of the UI, because badly written BuildSteps?? can stall the reactor for several seconds.
The fanciest configuration would probably be a symmetric configuration for everything but the UI. You would scale the number of UI master according to your number of UI users, and scale the number of engine masters to the number of workers.
Depending on your workload and size of master host, it is probably a good idea to start thinking of multimaster starting from a hundred workers connected.
Multimaster can also be used for high availability, and seamless upgrade of configuration code. Complex configuration indeed requires sometimes to restart the master to reload custom steps or code, or just to upgrade the upstream buildbot version.
In this case, you will implement following procedure:
As buildbot nine has been designed to allow such procedure, it has not been implemented in production yet as we know. There is probably a new REST api needed in order to graceful shutdown a master, and the details of gracefully dropping the connection to the workers to be sorted out.