Configuring warm Bazel

Characteristics of a CI job that influence the Bazel server and its caches must be predefined, so Engflow can prepare the right runner for each job. For example, a job for x64 needs a different starting point from a arm64 job.

We support the following settings.

Bento name

The name by which you'll reference this Bento in Buildkite YAMLs or Github workflows, e.g. dev_x64. You'll then reference it in the workflow .yaml via the engflow-bento-name=dev_x64 job label.

Repo name

The URL of the repository where CI jobs will come from, e.g. github.com/MyCompany/myproject. (There should be no protocol, like https://, nor a trailing / or .git.)

CI provider

Currently: Buildkite or GitHub

Image URL

The Docker image the CI host will use. The Buildkite/GitHub agent (and Git, and Bazel) will run inside of this.

For testing, a recent Ubuntu image will work best, e.g. docker.io/library/ubuntu@sha256:4fbb8e6a8395de5a7550b33509421a2bafbc0aab6c06ba2cef9ebffbc7092d90)

For production use, a bit of time can be saved by pre-installing dependencies in the image. This entails the following steps:

• Install all tools required for the CI host to execute the job (e.g. bazel, etc) must be installed into this image. This includes the following OS packages: ca-certificates git openssh-client curl bash python3.

• Dependencies for the GitHub agent (if using): libicu, libssl, zlib, liblttng-ust1, libkrb5-3, zlib1g. See GitHub source code for details.

• Bazelisk as /usr/bin/bazel, to execute Bazel.

• Upload the resulting image to a hosting location. Workers in the selected pool (specified using the Pool name attribute) must be granted permission to pull this image.

Hosting location

You most likely already have some execution images that EngFlow workers use for normal RE builds. Hosting the CI images in the same registry will make sure the workers can pull them.

• Refer to the image by using its pinned URL, for example: my.registry/x64/ci_dev@sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef.

Pool name

The cluster pool where CI hosts will run. EngFlow will set up this pool for you as part of enabling CI runners on your cluster and provide you with the exact value for this field.

Allow snapshots

If selected (recommended), then we take a snapshot of the container (under some circumstances) and reuse it for later CI jobs.

If not selected, all jobs run cold, and will not benefit from the time savings from using warm Bazel.

Branch name

The name of the branch where snapshots can be taken from. CI jobs from other branches in the repo can also be executed, but the system won't take snapshots from such branches. We recommend setting this field to master or main branch so that snapshots are only captured for post-submit jobs, i.e. commits merged to master or main.

Advanced options

This field is for enabling experimental/advanced options. Currently, the following settings are supported:

• cpu_overcommit (float): scale the CPU allowance for the CI container. When set and nonzero, the cluster-provided CPU limit is multiplied by this factor; a value greater than 1 lets the container burst above its nominal CPU share. Must not be negative.
• disk_size_gb (integer): size of the CI disk image (default: 50Gb). This setting must be synchronized with the cluster pool configuration's max input/output size.
• insert_invocation.post_agent_sleep (string): sleep after job completes, for debugging. Protobuf duration format (eg. "3600s", only unit accepted is "s")
• insert_invocation.bazel_rc_lines (object): custom Bazel flags. Contents:
• section (string): the Bazel section name (eg. common:config)
• flags (list of string): the flags
• insert_invocation.suppress_warnings (list of string): IDs of Bazel flag warnings to omit; see Bazel flag warnings.
• isolation_strategy (string): the process-isolation backend. "RUNC" (default; OCI container via runc) or "FIRECRACKER" (microVM).
• max_snapshot_age (string): if a warm run's snapshot is older than this value, force a cold run (which will then save a fresh snapshot). Protobuf duration format specified in seconds (eg. "3600s"). When set, this overrides the cluster-wide --experimental_bento_max_snapshot_age flag for this Bento only. Applies to all CI providers.
• max_test_fail_ratio (float): take snapshots even if this quantity of tests failed, between 0.0 and 1.0.
• nested_container_strategy (string): enable Docker on the CI machine. "DOCKER_SIBLING" is the only accepted value. Not supported for FIRECRACKER.
• network_isolation (boolean): enable network isolation
• vm_kernel (string): only for isolation_strategy: "FIRECRACKER". The guest kernel image, as a host path or a file://, https://, s3://, or gs:// URL. Defaults to Amazon's minimal kernel image.

Here is an example that exercises all the fields:

{
   "cpu_overcommit": 1.5,
   "disk_size_gb": 150,
   "insert_invocation": {
       "post_agent_sleep": "1200s",
       "bazel_rc_lines": [
           {
               "section": "build",
               "flags": [ "--jobs=150", "--nostamp" ]
           }
       ],
       "suppress_warnings": [ "disk_cache" ]
    },
    "isolation_strategy": "FIRECRACKER",
    "max_snapshot_age": "3600s",
    "max_test_fail_ratio": 0.02,
    "network_isolation": true,
    "vm_kernel": "s3://my-bucket/firecracker-kernel/vmlinux-6.1.155"
}

Bazel flag warnings

A warm Bazel snapshot holds the entire disk and memory of the CI runner, so everything Bazel stores locally makes saving and restoring snapshots slower. After each build, test, run or coverage command, EngFlow checks the effective Bazel options and reports options that grow the snapshot as warnings on the job. Each warning starts with its ID in brackets, for example [disk_cache].

By default, the runner disables the repository cache (--repository_cache=) and runfiles symlink trees (--nobuild_runfile_links). Your own .bazelrc files take precedence.

ID	Reported when
remote_executor	The runner has credentials for your EngFlow cluster, but --remote_executor is not set, so actions run on the CI runner.
remote_download_minimal	A remote cache or executor is used, but --remote_download_outputs is not minimal (set it with --remote_download_minimal).
remote_repo_contents_cache	Bazel 9.1.0 or later uses a remote cache or executor, but --experimental_remote_repo_contents_cache is not set.
repository_cache	--repository_cache is set to a directory, duplicating the external repositories in the output base.
disk_cache	--disk_cache is set to a directory, duplicating the action outputs in the output base.
build_runfile_links	--build_runfile_links is enabled.
materialize_param_files	--materialize_param_files is enabled.
sandbox_debug	--sandbox_debug is enabled.

To omit warnings, list their IDs in insert_invocation.suppress_warnings under Advanced options.