Skip to content

Bazel First-Time Setup

This document describes how to configure Bazel to connect to an EngFlow Remote Execution cluster.

1. Prepare a Docker container image for remote actions

Remote actions executed on Linux are executed in a Docker container to ensure a consistent execution environment. Your Docker container image must include all system dependencies of actions. For example, a C++ compiler may be required.

If there are no special requirements, Bazel toolchains project provides Ubuntu 16.04 base images.

You may want to add an engflow user within your container. Linux builds are executed with the UID 108. To register this user, add the snippet below to your Dockerfile:

Docker
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Create a user with specific ID used by EngFlow RBE
#   -r Create a system account
#   -m Create the user's home directory
#   -d Home directory of the new account
#   -s Login shell of the new account
#   -g Name or ID of the primary group
#   -G List of supplementary groups
#   -u Specify user ID
ENV ENGFLOW_RBE_GROUP_ID=114
ENV ENGFLOW_RBE_GROUP_NAME=engflow
ENV ENGFLOW_RBE_USER_ID=108
ENV ENGFLOW_RBE_USER_NAME=engflow
RUN groupadd --gid "${ENGFLOW_RBE_GROUP_ID}" "${ENGFLOW_RBE_GROUP_NAME}" \
    && useradd \
    -rm \
    -d "/home/${ENGFLOW_RBE_USER_NAME}" \
    -s /bin/bash \
    -g root \
    -u "${ENGFLOW_RBE_USER_ID}" \
    --gid "${ENGFLOW_RBE_GROUP_ID}" \
    -G "${ENGFLOW_RBE_GROUP_NAME}" \
    "${ENGFLOW_RBE_USER_NAME}" \
    && usermod -aG sudo "${ENGFLOW_RBE_USER_NAME}"

USER "${ENGFLOW_RBE_USER_ID}":"${ENGFLOW_RBE_GROUP_ID}"
WORKDIR "/home/${ENGFLOW_RBE_USER_NAME}"

2. Create a remote configuration

In Bazel, a platform expresses an environment where an action can run. When performing remote execution, Bazel is instructed to build for a special EngFlow platform. This platform carries the Docker image and other environmental configuration to the remote execution service.

Once a Docker container is ready, use the rbe_configs_gen tool to generate Bazel configuration for the image. For example, the following will generate configuration for a Ubuntu 20.04 image into a engflow_config directory:

Bash
1
2
3
4
5
6
7
rbe_configs_gen \
    --toolchain_container=gcr.io/engflow-public/bazel/ubuntu2004-bazel-java11@sha256:c7b874d00b5a5c0cdc9e2d24eea11f3ca689723a42d6c6d088e9e659d389a0f1 \
    --exec_os=linux \
    --target_os=linux \
    --generate_java_configs=false \
    --output_config_path engflow_config \
    --output_src_root .

rbe_configs_gen generates a remote platform target for EngFlow in engflow_config/config/BUILD. It also outputs configuration for the C++ toolchain in the container into engflow_config/cc/.

If you already have a custom C++ compiler crosstool, it's likely easiest to manually write the EngFlow platform rule rather than using rbe_configs_gen. Here is an example engflow_config/config/BUILD for an x86_64 Linux image:

BUILD
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
platform(
    name = "platform",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    exec_properties = {
        "container-image": "docker://gcr.io/bazel-public/ubuntu2004-java11@sha256:69a78f121230c6d5cbfe2f4af8ce65481aa3f2acaaaf8e899df335f1ac1b35b5",
    },
)
If your remote actions require internet access, add "dockerNetwork": "standard" to exec_properties.

3. Set up .bazelrc.

Configure Bazel to use the EngFlow platform in .bazelrc.

  1. Add common flags:

    .bazelrc
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
    build:engflow --define=EXECUTOR=remote
build:engflow --disk_cache=
build:engflow --experimental_inmemory_dotd_files
build:engflow --experimental_inmemory_jdeps_files
build:engflow --incompatible_strict_action_env=true
build:engflow --remote_timeout=600
build:engflow --nolegacy_important_outputs
build:engflow --grpc_keepalive_time=30s
# Compress uploads and downloads between Bazel and the cluster for faster data transfer.
#
# This is recommended for Bazel 5 and later.
build:engflow --experimental_remote_cache_compression=true

  2. Add platform flags:

    .bazelrc
    1
2
3
    build:engflow --host_platform=//engflow_config/config:platform
build:engflow --platforms=//engflow_config/config:platform
build:engflow --extra_execution_platforms=//engflow_config/config:platform

  3. If not using a custom C++ toolchain already, add flags for the container's C++ toolchain:

    .bazelrc
    1
2
    build:engflow --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
build:engflow --crosstool_top=//engflow_config/cc:toolchain

  4. If Java is required, optionally use the hermetic JDK shipped with Bazel:

    .bazelrc
    1
    build:engflow --java_runtime_version=remotejdk_11

  5. Add the remote executor flags. For the <CLUSTER_ENDPOINT>, use the host name or IP address of the Load Balancer. You may also have to specify the port, if using a non-standard one. The default ports are 443 for grpc/s, and 80 for http/s.

    If you use TLS with our certificate (engflow-ca.crt):

    .bazelrc
    1
2
3
4
5
    build:engflow --remote_cache=grpcs://<CLUSTER_ENDPOINT>
build:engflow --remote_executor=grpcs://<CLUSTER_ENDPOINT>
build:engflow --bes_backend=grpcs://<CLUSTER_ENDPOINT>
build:engflow --bes_results_url=https://<CLUSTER_ENDPOINT>/invocation/
build:engflow --tls_certificate=<PATH>/engflow-remote-execution/engflow-ca.crt

    If you don't use TLS at all:

    .bazelrc
    1
2
3
4
    build:engflow --remote_cache=grpc://<CLUSTER_ENDPOINT>
build:engflow --remote_executor=grpc://<CLUSTER_ENDPOINT>
build:engflow --bes_backend=grpc://<CLUSTER_ENDPOINT>
build:engflow --bes_results_url=http://<CLUSTER_ENDPOINT>/invocation/

Usage

We recommend using the latest available Bazel version.

Bash
1
bazel build --config=engflow //foo:bar