Skip to content

Create a Bazel C++ toolchain configuration for remote actions

To construct C/C++ commands that can run remotely, Bazel needs a C++ toolchain configuration.

For local builds, Bazel configures this automatically based on toolchains installed on your host system. This may not be desirable: different developers may have different toolchains installed.

For remote builds, you need to configure a toolchain explicitly because Bazel doesn't know what's installed in the remote environment.

This page explains how to configure a Bazel C++ toolchain configuration for remote execution. You must have already created a container image with your C++ toolchain installed. You'll also need a host machine with Docker installed with the same OS and architecture as the remote execution environment.

Generating configuration with rbe_configs_gen

rbe_configs_gen is a tool that can automatically configure a toolchain for you, given a Docker image tag. Visit that page and follow the instructions there to build and run it.

rbe_configs_gen works for Linux only. You may need to install additional tools such as Java in your container image, or pass flags like --generate_java_configs=false for rbe_configs_gen to work.

Generating configuration with Bazel

You can generate your C++ toolchain configuration with Bazel by building a C++ target locally in a container, then extracting the automatic configuration. This works more reliably and has fewer restrictions than rbe_configs_gen, but it's a more manual process.

You can find a complete example workspace at https://github.com/EngFlow/example. The remote toolchains are in the platform/linux_x64 and platform/windows_x64 directories.

Before you begin, you'll need a Docker tag for your container image. We'll use engflow-container-image in the instructions here. If it's in a remote registry, you must be able to docker pull from that registry (docker login first).

To build your configuration:

  1. Run a new container based on your image. The next few steps should be performed inside the container.

    Bash
    1
    docker run --interactive --tty --user=0 engflow-container-image bash
    PowerShell
    1
    docker run --interactive --tty --user=ContainerAdministrator engflow-container-image powershell

  2. Install Bazelisk.

    Bash
    1
2
    curl --location --output /usr/local/bin/bazel https://github.com/bazelbuild/bazelisk/releases/download/v1.19.0/bazelisk-linux-amd64
chmod +x /usr/local/bin/bazel
    PowerShell
    1
2
3
4
5
6
7
8
    $global:ProgressPreference = "SilentlyContinue"
New-Item -Type directory -Path 'C:\Program Files\bazel'
Invoke-WebRequest `
  -Out 'C:\Program Files\bazel\bazel.exe' `
  -Uri 'https://github.com/bazelbuild/bazelisk/releases/download/v1.19.0/bazelisk-windows-amd64.exe'
$env:PATH = [Environment]::GetEnvironmentVariable('PATH', 'Machine')
$env:PATH = "$env:PATH;C:\Program Files\bazel"
[Environment]::SetEnvironmentVariable('PATH', $env:PATH, 'Machine')

  3. Create a Bazel C++ workspace, then build the target. You may need to set the CC environment variable if Bazel doesn't find your compiler automatically.

    Bash
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
    mkdir /work
cd /work
touch WORKSPACE
cat >BUILD <<'EOF'
cc_binary(
    name = "hello",
    srcs = ["hello.cc"],
)
EOF
cat >hello.cc <<'EOF'
#include <iostream>
int main() {
    std::cout << "hello" << std::endl;
    return 0;
}
EOF
export CC=clang
bazel build //:hello
    PowerShell
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
    New-Item -Type directory -Path 'C:/work'
Set-Location 'C:/work'
Set-Content -Path WORKSPACE -Value ''
Set-Content -Path BUILD -Value @"
cc_binary(
    name = "hello",
    srcs = ["hello.cc"],
)
"@
Set-Content -Path hello.cc -Value @"
#include <iostream>
int main() {
    std::cout << "hello" << std::endl;
    return 0;
}
"@
bazel build //:hello

  4. Find the local_config_cc repository, and copy the files. They are short enough that you can read them, then copy from your terminal.

    Bash
    1
2
3
4
    cd $(bazel info output_base)/external/local_config_cc
cat BUILD
cat cc_toolchain_config.bzl
...
    PowerShell
    1
2
3
4
5
    $outputBase = bazel info output_base
Set-Location "${outputBase}/external/local_config_cc"
Get-Content BUILD
Get-Content cc_toolchain_config.bzl
...

  5. On the host machine, create a directory in the repository where you want to use remote execution. The directory name is up to you, but its name should reflect the remote environment. If you use remote execution on multiple platforms, you'll probably want multiple directories. In our example repository, we have platform/linux_x64 and platform/windows_x64. Paste the local_config_cc files into that new directory.

  6. Modify the BUILD file, keeping only the toolchains you need, discarding others. You may combine it with an existing BUILD file, perhaps containing other toolchains and a platform definition. In particular, keep the following targets:

    • cc_toolchain_suite toolchain: this is the top-level definition for the C++ toolchains. It has a toolchains attribute mapping cpu|compiler and cpu entries to cc_toolchain labels. Keep only the entries for cc_toolchain targets you need. The --crosstool_top flag will refer to this target.
    • cc_toolchain cc-compiler-k8: this contains high-level information for an individual toolchain. There may be multiple toolchains, and the name may be different, depending on your CPU architecture (k8 is x86_64). You can drop the armeabi-v7a target; this is for Android only.
    • cc_toolchain_config local: this describes your toolchain in detail. It lists paths of commands, paths of include directories, and flags to pass. You can modify this if you want to change which flags are passed automatically.
    • filegroup empty: used in places where the toolchain doesn't depend on anything inside the Bazel workspace. You can replace ":compiler_deps" with ":empty".

  7. Add a toolchain target to the BUILD file. This is part of Bazel's toolchain system, and adding this will help Bazel select your toolchain automatically. Make sure the toolchain attribute refers to a cc_toolchain target.

    Text Only
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
    toolchain(
    name = "cc-toolchain",
    exec_compatible_with = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    target_compatible_with = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    toolchain = ":cc-compiler-k8",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)
    Text Only
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
    toolchain(
    name = "cc-toolchain",
    exec_compatible_with = [
        "@platforms//os:windows",
        "@platforms//cpu:x86_64",
        "@bazel_tools//tools/cpp:msvc",
    ],
    target_compatible_with = [
        "@platforms//os:windows",
        "@platforms//cpu:x86_64",
        "@bazel_tools//tools/cpp:msvc",
    ],
    toolchain = ":cc-compiler-msvc",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)

  8. In your project's .bazelrc file, ensure that your toolchain is registered when you are using your remote configuration. Ensure that --crosstool_top and --host_crosstool_top refer to your cc_toolchain_suite target, and that --cpu, --host_cpu, --compiler, and --host_compiler refer to the CPU and compiler for toolchains entry.

    Text Only
    1
2
3
4
5
6
7
8
9
    build:remote_linux_x64 --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
build:remote_linux_x64 --extra_execution_platforms=//platform/linux_x64
build:remote_linux_x64 --extra_toolchains=//platform/linux_x64:cc-toolchain
build:remote_linux_x64 --crosstool_top=//platform/linux_x64:toolchain
build:remote_linux_x64 --host_crosstool_top=//platform/linux_x64:toolchain
build:remote_linux_x64 --cpu=k8
build:remote_linux_x64 --host_cpu=k8
build:remote_linux_x64 --compiler=clang
build:remote_linux_x64 --host_compiler=clang
    Text Only
    1
2
3
4
5
6
7
8
9
    build:remote_windows_x64 --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
build:remote_windows_x64 --extra_execution_platforms=//platform/windows_x64
build:remote_windows_x64 --extra_toolchains=//platform/windows_x64:cc-toolchain
build:remote_windows_x64 --crosstool_top=//platform/windows_x64:toolchain
build:remote_windows_x64 --host_crosstool_top=//platform/windows_x64:toolchain
build:remote_windows_x64 --cpu=k8
build:remote_windows_x64 --host_cpu=k8
build:remote_windows_x64 --compiler=msvc-cl
build:remote_windows_x64 --host_compiler=msvc-cl

  9. Verify your new toolchain configuration works by building the same C++ target using your new remote configuration from a compatible host environment. You may need to configure a platform and add additional flags to .bazelrc to enable remote execution, if you haven't done so already. Refer to Bazel First-Time Setup.