Skip to content

Bazel First-Time Setup

This document describes how to configure Bazel to connect to an EngFlow Remote Execution cluster.

1. Prepare a Docker container image for remote actions

Remote actions executed on Linux are executed in a Docker container to ensure a consistent execution environment. Your Docker container image must include all system dependencies of actions. For example, a C++ compiler may be required.

If there are no special requirements, Bazel toolchains project provides Ubuntu 16.04 base images.

You may want to add an engflow user within your container. Linux builds are executed with the UID 108. To register this user, add the snippet below to your Dockerfile:

Docker
# Create a user with specific ID used by EngFlow RBE
#   -r Create a system account
#   -m Create the user's home directory
#   -d Home directory of the new account
#   -s Login shell of the new account
#   -g Name or ID of the primary group
#   -G List of supplementary groups
#   -u Specify user ID
ENV ENGFLOW_RBE_GROUP_ID=114
ENV ENGFLOW_RBE_GROUP_NAME=engflow
ENV ENGFLOW_RBE_USER_ID=108
ENV ENGFLOW_RBE_USER_NAME=engflow
RUN groupadd --gid "${ENGFLOW_RBE_GROUP_ID}" "${ENGFLOW_RBE_GROUP_NAME}" \
    && useradd \
    -rm \
    -d "/home/${ENGFLOW_RBE_USER_NAME}" \
    -s /bin/bash \
    -g root \
    -u "${ENGFLOW_RBE_USER_ID}" \
    --gid "${ENGFLOW_RBE_GROUP_ID}" \
    -G "${ENGFLOW_RBE_GROUP_NAME}" \
    "${ENGFLOW_RBE_USER_NAME}" \
    && usermod -aG sudo "${ENGFLOW_RBE_USER_NAME}"

USER "${ENGFLOW_RBE_USER_ID}":"${ENGFLOW_RBE_GROUP_ID}"
WORKDIR "/home/${ENGFLOW_RBE_USER_NAME}"

2. Create a remote configuration

In Bazel, a platform expresses an environment where an action can run. When performing remote execution, Bazel is instructed to build for a special EngFlow platform. This platform carries the Docker image and other environmental configuration to the remote execution service.

Once a Docker container is ready, use the rbe_configs_gen tool to generate Bazel configuration for the image. For example, the following will generate configuration for a Ubuntu 20.04 image into a engflow_config directory:

Bash
1
2
3
4
5
6
7
rbe_configs_gen \
    --toolchain_container=gcr.io/engflow-public/bazel/ubuntu2004-bazel-java11@sha256:c7b874d00b5a5c0cdc9e2d24eea11f3ca689723a42d6c6d088e9e659d389a0f1 \
    --exec_os=linux \
    --target_os=linux \
    --generate_java_configs=false \
    --output_config_path engflow_config \
    --output_src_root .

rbe_configs_gen generates a remote platform target for EngFlow in engflow_config/config/BUILD. It also outputs configuration for the C++ toolchain in the container into engflow_config/cc/.

If you already have a custom C++ compiler crosstool, it's likely easiest to manually write the EngFlow platform rule rather than using rbe_configs_gen. Here is an example engflow_config/config/BUILD for an x86_64 Linux image:

BUILD
platform(
    name = "platform",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    exec_properties = {
        "container-image": "docker://gcr.io/bazel-public/ubuntu2004-java11@sha256:69a78f121230c6d5cbfe2f4af8ce65481aa3f2acaaaf8e899df335f1ac1b35b5",
    },
)
If your remote actions require internet access, add "dockerNetwork": "standard" to exec_properties.

3. Set up .bazelrc.

Configure Bazel to use the EngFlow platform in .bazelrc.

  1. Add common flags:

    .bazelrc
    build:engflow --define=EXECUTOR=remote
    build:engflow --disk_cache=
    build:engflow --experimental_inmemory_dotd_files
    build:engflow --experimental_inmemory_jdeps_files
    build:engflow --incompatible_strict_action_env=true
    build:engflow --remote_timeout=600
    build:engflow --nolegacy_important_outputs
    build:engflow --grpc_keepalive_time=30s
    # Compress uploads and downloads between Bazel and the cluster for faster data transfer.
    #
    # This is recommended for Bazel 5 and later.
    build:engflow --experimental_remote_cache_compression=true
    

  2. Add platform flags:

    .bazelrc
    1
    2
    3
    build:engflow --host_platform=//engflow_config/config:platform
    build:engflow --platforms=//engflow_config/config:platform
    build:engflow --extra_execution_platforms=//engflow_config/config:platform
    

  3. If not using a custom C++ toolchain already, add flags for the container's C++ toolchain:

    .bazelrc
    build:engflow --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
    build:engflow --crosstool_top=//engflow_config/cc:toolchain
    

  4. If Java is required, optionally use the hermetic JDK shipped with Bazel:

    .bazelrc
    build:engflow --java_runtime_version=remotejdk_11
    
  5. Add the remote executor flags. For the <CLUSTER_ENDPOINT>, use the host name or IP address of the Load Balancer. You may also have to specify the port, if using a non-standard one. The default ports are 443 for grpc/s, and 80 for http/s.

    If you use TLS with our certificate (engflow-ca.crt):

    .bazelrc
    1
    2
    3
    4
    5
    build:engflow --remote_cache=grpcs://<CLUSTER_ENDPOINT>
    build:engflow --remote_executor=grpcs://<CLUSTER_ENDPOINT>
    build:engflow --bes_backend=grpcs://<CLUSTER_ENDPOINT>
    build:engflow --bes_results_url=https://<CLUSTER_ENDPOINT>/invocation/
    build:engflow --tls_certificate=<PATH>/engflow-remote-execution/engflow-ca.crt
    

    If you don't use TLS at all:

    .bazelrc
    1
    2
    3
    4
    build:engflow --remote_cache=grpc://<CLUSTER_ENDPOINT>
    build:engflow --remote_executor=grpc://<CLUSTER_ENDPOINT>
    build:engflow --bes_backend=grpc://<CLUSTER_ENDPOINT>
    build:engflow --bes_results_url=http://<CLUSTER_ENDPOINT>/invocation/
    

Usage

We recommend using the latest available Bazel version.

Bash
bazel build --config=engflow //foo:bar