Overview

Speed up your build by hosting CI Runners inside your existing EngFlow cluster and benefit from efficiency gains from co-location, warm Bazel instances, and warm Bazel caches.

CI Runners availability

CI Runners are not yet available to the public. Contact us to know when this feature becomes available.

CI Runners work equally well with

• GitHub Actions (with a GitHub App)
• Buildkite
• And more coming soon!

CI Runners can be registered both at an organization and repository level, but they require a different pool separate from the other remote execution pools.

Configure GitHub Action CI Runners with a GitHub App

To configure GitHub Action CI Runners you'll need to be a GitHub organization admin.

1. Create a GitHub App in your organization with permission to register GitHub Self-hosted runners in the organization.
2. Create a Private Key for the GitHub App.

3. Create a .json file containing both the App ID and the Private Key, using the format.

   {
     "github_app_id": "<app id>",
     "private_key": "<private key with newlines escaped>"
   }
   

4. Upload that .json file to the Secrets Manager running in your EngFlow cluster. Note the address of the secret.

5. Contact EngFlow support to configure the cluster for CI Runners. We will:

   1. Adding one or more pools for the CI Runners (one for each combination of architecture and OS).
   2. Providing the address for the GitHub App credential stored in Secrets Manager.
   3. (optional) Enabling sysbox docker runtime for docker-in-docker.

Configure GitHub Webhooks

1. Contact EngFlow support to enable webhooks on the cluster for GitHub actions. EngFlow support will provide you with a confidential UUID for the GitHub webhook.

2. Configure the GitHub webhook, in https://github.com/<organization>/ Settings Webhooks Add webhook, with the following settings:

   • Payload URL:

     https://<name>.cluster.engflow.com/webhooks/github/runners/<uuid> (provided by EngFlow)

   • Content type:

     application/json

   • Secret:

     none (we use a secret embedded in the webhook URL)

   • Select "Let me select individual events" and make sure that the only selected event type is "Workflow Jobs".

Use GitHub Action CI Runners

Once you've configured the GitHub Action CI Runners, add them to any workflow by adding the following tags to the workflow YAML, including at least one that starts with engflow-*:

• self-hosted
• arch=x64 (x64 or arm64)
• engflow-container-image=docker://<docker URL>
• engflow-pool=<pool name>
• engflow-runtime=sysbox-runc (only if Docker-in-Docker is needed)
• engflow-runner-id=${{ github.repository_id }}_<job-name>_${{ github.run_id }}_${{ github.run_number }}_${{ github.run_attempt }}
• os=linux (the arch and os must match the pool configuration)

The engflow-runner-id tag is mandatory -- it is an attempt to uniquely identify the job for EngFlow's CI Runners orchestration. Unfortunately, the <job-name> identifier needs to be entered manually (the ${{ github.job }} context field is not populated until the job is run, so we couldn't use that). It can be any string as long as it is unique within the current workflow -- but we recommend setting it to the actual job name for readability, e.g. build-release-images.

Important

If the job has a matrix strategy, meaning it is actually a set of jobs, the runs-on labels need to include all the dimensions in the matrix as well, e.g. foo=${{ matrix.foo }} needs to be copied into the runs-on labels for every matrix dimension foo. This is so that the overall label-set values identify the job uniquely, which is required for EngFlow to correctly follow up on the job status.

If you were previously running the action outside a Docker container, and if you are relying on secrets present on the action runner machines, you may have to make changes to the GitHub action. Below an example of a GitHub Action workflow running on CI Runners:

  action:
    runs-on:
      - self-hosted
      - "arch=x64"
      - "engflow-container-image=docker://ghcr.io/engflow/debian12-amd64@sha256:bf092335d2e79871b37a3c77f52733c2e0e522001cebdeb6d34e615d2ba57330"
      - "engflow-pool=ci_sysbox_x64"
      - "engflow-runtime=sysbox-runc"
      - "os=linux"
    steps:
    - uses: actions/checkout@v3
    - name: Run all tests
      run: bazel test //...

Requirements on images for CI Runners

The GitHub agent must be able to run on the container image that you provide. For your convenience, we have example Debian 12 base images at ghcr.io/engflow/debian12-amd64 and ghcr.io/engflow/debian12-arm64.

Configure Buildkite CI Runners

Buildkite CI Runners are referred to as Buildkite agents in the Buildkite documentation. EngFlow Buildkite agents are registered at the organization level, and are available to every pipeline within that organization.

You'll need a Buildkite token to register the agents.

1. Create the Buildkite agent token.

2. Upload a JSON secrets file containing the Buildkite agent token and an SSH private key for the source repo to the Secrets Manager in your EngFlow cluster:

   {
     "BUILDKITE_AGENT_TOKEN": "<Buildkite Agent Token>",
     "SSH_PRIVATE_KEY": "<Multi-line SSH private key with newline as \n>",
     "ADDITIONAL_SECRET": "<Whatever you want>"
   }
   

   The SSH key is written to ~/.ssh/id_rsa. Other variables are written to hooks/environment.

3. IMPORTANT, unset the secrets before running Bazel so they are not uploaded to the BES.

   unset BUILDKITE_AGENT_TOKEN
   unset ADDITIONAL_SECRET
   

4. Contact EngFlow support to configure the cluster for CI Runners. We will:

   1. Adding one or more pools for the CI Runners (one for each combination of architecture and OS).
   2. Providing the address for the GitHub App credential stored in Secrets Manager.
   3. Providing a confidential UUID for the Buildkite webhook.

5. Configure Buildkite Webhooks in Organization Settings Notification Settings Add Webhook, with the following settings:

   • Webhook URL:

     https://<name>.cluster.engflow.com/webhooks/buildkite/agents/<uuid> (provided by EngFlow)

   • Make sure that the only selected event type is "ping" and "job.scheduled".

6. To avoid Bazel creating a new directory for every build, set BUILDKITE_BUILD_CHECKOUT_PATH to a fixed path:

   BUILDKITE_BUILD_CHECKOUT_PATH=/home/engflow/.buildkite-agent/build
   

7. To speed up Git checkout, set BUILDKITE_GIT_MIRRORS_PATH to a fixed path:

   BUILDKITE_GIT_MIRRORS_PATH=/home/engflow/.buildkite-agent/git-mirrors/.
   

Use Buildkite CI Runners

Once you've configured the Buildkite CI Runners, add them to any workflow.

In your Buildkite Pipeline configuration, add the following tags:

• arch: "amd64" (amd64 or arm64)
• engflow-container-image: "docker://<docker URL>"
• engflow-pool: "<pool name>"
• engflow-runtime: "sysbox-runc" (only if needed)
• os: "linux" (arch and os are currently not inferred from the pool name)

steps:
  - label: ":buildkite: Upload steps"
    command: "buildkite-agent pipeline upload .buildkite/pipeline.yml"
    agents:
      arch: "arm64"
      engflow-container-image: "docker://ghcr.io/engflow/debian12-arm64@sha256:be9e2cfde8cce9388cafd7600d77ea12d2f5826857987ea65858a19cd35155cc"
      engflow-runtime: "sysbox-runc"
      engflow-pool: "ci_sysbox_arm64"
      os: "linux"