Skip to content

CI runners: Frequently asked questions

What's the cost of using EngFlow CI runners?

Since jobs executed with warm Bazel are run on EngFlow worker machines, the execution occupies licensed cores and incurs additional costs. Contact us to receive pricing information customized for your EngFlow cluster.

Where can I see in-progress CI jobs?

As soon as your EngFlow cluster detects and picks up a CI job, you can see it in the Build and Test UI on the CI Runners Status page accessed from the sidebar.

When does a CI job use a snapshot?

CI jobs always use snapshots if there is one available for the job's Bento.

What Bazel version is required to use CI runners?

Bazel 9 is preferred for optimal performance. At least Bazel 5.4.0 is required to support credential helper.

When are snapshots created?

A snapshot from a CI job is captured if it meets all of the following conditions:

  • In your Bento definition, you selected Allow snapshots. See Define a Bento (Buildkite) or Define a Bento (GitHub Actions) for more information.
  • The job comes from the branch specified in the Branch name field in the Bento.
  • The job is a post-submit job, i.e., it was triggered by commits merged into the branch specified in the Bento.
  • The job completes successfully.
  • Either:
    • The job ran with the environment variable ENGFLOW_BENTO_FORCE_SAVE=1 which forces snapshot creation or,
    • The latest snapshot is older than the predefined threshold.

How can I force snapshot creation?

Specifying the environment variable ENGFLOW_BENTO_FORCE_SAVE=1 in the job forces snapshot creation.

When should I force snapshot creation?

You should force creating a snapshot if you've updated .bazelversion since the latest snapshot, or have made any other changes that would force the server to restart; see How do I know if the Bazel server restarted? For example, changing the --output_root causes the old server to be invalidated.

Where can I see the snapshot creation time?

If a snapshot exists for a Bento, you can see it in on the Bentos page in your cluster's Build and Test UI. Each Bento that has a snapshot lists a Last snapshot at date.

Latest snapshot

How do I know if the Bazel server restarted?

You can determine this by inspecting the action results for a specific CI job.

To inspect the action result in your EngFlow Build and Test UI:

  1. Access the CI Runners Status page.
  2. Find the listing for the CI job, and select the corresponding View execution result link. This opens the Action Details page.
  3. On the Action Details page, expand the Action Result card.
  4. Inspect the Stderr output. If the Bazel server restarted, you will see a “Died” entry near the end of the output.

Alternatively you can also see this in the CI logs, typically as a Extracting Bazel installation and/or a Starting local Bazel server [...] and connecting to it log line.

If I update a Bento definition, will the existing snapshot be lost?

Yes. Updating a Bento definition creates a new internal Bento ID. The snapshot associated with the older ID won't be used with the new ID.

This holds even if you update a Bento then restore its old value, i.e. you do an A-B-A edit. Every edit yields a new internal Bento ID. Snapshots correspond to one specific Bento ID.

Where are snapshots stored?

Snapshots are stored in the EngFlow CAS, like other remote execution artifacts, and can be downloaded by anyone with EngFlow access. They are garbage collected like any other blob.

What are the security implications of using EngFlow CI runners?

The CI jobs run inside a privileged container, secured by an Apparmor/seccomp profile.

Snapshots are stored as CAS blobs so they can be downloaded. They contain the following sensitive data:

  • Source code snapshot
  • Potentially: EngFlow cluster credential; this credential has a limited lifetime.
  • Potentially: Any additional secrets passed in JSON files uploaded to the secrets manager in your cluster.

Users with access to EngFlow typically already have access to source code, so this is not an additional exfiltration risk. However, CI steps that need privileged credentials, e.g. deploy steps that need GCP or AWS credentials should use secret support provided by your CI provider. If these processes need snapshotting, the workflows should ensure that any such credentials are either revoked or deleted, both from the file system and from any persistent processes.

CI jobs for pre-submit code in GitHub Actions

If you're using the GitHub Actions CI provider, the client cannot control which job gets assigned to a worker. This means that a malicious user can cause a snapshot to be based on a pre-submit (unreviewed) source code state.

How does build startup time compare between Github Actions and Buildkite?

GitHub Actions jobs take slightly longer to get started: both agents have startup time, and GitHub servers have internal buffering. This causes GitHub Actions jobs to take an additional 10 seconds to start.