2.5.6.2. Worker Options

Properties

Worker objects can also be created with an optional properties argument, a dictionary specifying properties that will be available to any builds performed on this worker. For example:

c['workers'] = [
    worker.Worker('bot-solaris', 'solarispasswd',
                  properties={'os': 'solaris'}),
]

Worker properties have priority over other sources (Builder, Scheduler, etc.). You may use the defaultProperties parameter that will only be added to Build Properties if they are not already set by another source:

c['workers'] = [
    worker.Worker('fast-bot', 'fast-passwd',
                  defaultProperties={'parallel_make': 10}),
]

Worker collects and exposes /etc/os-release fields for interpolation. These can be used to determine details about the running operating system, such as distribution and version. See https://www.linux.org/docs/man5/os-release.html for details on possible fields. Each field is imported with os_ prefix and in lower case. os_id, os_id_like, os_version_id and os_version_codename are always set, but can be null.

Limiting Concurrency

The Worker constructor can also take an optional max_builds parameter to limit the number of builds that it will execute simultaneously:

c['workers'] = [
    worker.Worker('bot-linux', 'linuxpassword',
                  max_builds=2),
]

Master-Worker TCP Keepalive

By default, the buildmaster sends a simple, non-blocking message to each worker every hour. These keepalives ensure that traffic is flowing over the underlying TCP connection, allowing the system’s network stack to detect any problems before a build is started.

The interval can be modified by specifying the interval in seconds using the keepalive_interval parameter of Worker (defaults to 3600):

c['workers'] = [
    worker.Worker('bot-linux', 'linuxpasswd',
                  keepalive_interval=3600)
]

The interval can be set to None to disable this functionality altogether.

When Workers Go Missing

Sometimes, the workers go away. One very common reason for this is when the worker process is started once (manually) and left running, but then later the machine reboots and the process is not automatically restarted.

If you’d like to have the administrator of the worker (or other people) be notified by email when the worker has been missing for too long, just add the notify_on_missing= argument to the Worker definition. This value can be a single email address, or a list of addresses:

c['workers'] = [
    worker.Worker('bot-solaris', 'solarispasswd',
                  notify_on_missing='bob@example.com')
]

By default, this will send an email when the worker has been disconnected for more than one hour. Only one email per connection-loss event will be sent. To change the timeout, use missing_timeout= and give it a number of seconds (the default is 3600).

You can have the buildmaster send an email to multiple recipients by providing a list of addresses instead of a single one:

c['workers'] = [
    worker.Worker('bot-solaris', 'solarispasswd',
                  notify_on_missing=['bob@example.com', 'alice@example.org'],
                  missing_timeout=300)  # notify after 5 minutes
]

The email sent this way will use a MailNotifier (see MailNotifier) status target, if one is configured. This provides a way for you to control the from address of the email, as well as the relayhost (aka smarthost) to use as an SMTP server. If no MailNotifier is configured on this buildmaster, the worker-missing emails will be sent using a default configuration.

Note that if you want to have a MailNotifier for worker-missing emails but not for regular build emails, just create one with builders=[], as follows:

from buildbot.plugins import status, worker
m = status.MailNotifier(fromaddr='buildbot@localhost', builders=[],
                        relayhost='smtp.example.org')
c['reporters'].append(m)

c['workers'] = [
    worker.Worker('bot-solaris', 'solarispasswd',
                  notify_on_missing='bob@example.com')
]

Workers States

There are some times when a worker misbehaves because of issues with its configuration. In those cases, you may want to pause the worker, or maybe completely shut it down.

There are three actions that you may take (in the worker’s web page Actions dialog):

• Pause: If a worker is paused, it won’t accept new builds. The action of pausing a worker will not affect any ongoing build.

• Graceful Shutdown: If a worker is in graceful shutdown mode, it won’t accept new builds, but will finish the current builds. When all of its build are finished, the buildbot-worker process will terminate.

• Force Shutdown: If a worker is in force shutdown mode, it will terminate immediately, and the build it was currently doing will be put to retry state.

Those actions will put the worker in either of two states:

• paused: the worker is paused if it is connected but doesn’t accept new builds.

• graceful: the worker is graceful if it doesn’t accept new builds, and will shutdown when builds are finished.

A worker might not be able to accept a job for a period of time if buildbot detects a misbehavior. This is called the quarantine timer.

Quarantine timer is an exponential back-off mechanism for workers. This prevents a misbehaving worker from eating the build queue by quickly finishing builds in EXCEPTION state. When misbehavior is detected, the timer will pause the worker for 10 seconds, and then the time will double with each misbehavior detection until the worker finishes a build.

The first case of misbehavior is for a latent worker to not start properly. The second case of misbehavior is for a build to end with an EXCEPTION status.

Pausing and unpausing a worker will force it to leave quarantine immediately. The quarantine timeout will not be reset until the worker finishes a build.

Worker states are stored in the database, can be queried via REST API, and are visible in the UI’s workers page.