Windows remote execution¶
You can use Bazel on Windows to build software quickly with remote execution. However, there are a number of differences you should be aware of, compared with building software locally or building remotely on Linux.
Containers and toolchains¶
EngFlow executes most remote actions in containers using Docker or Sysbox. This provides three major benefits:
- Clients have greater control over their runtime environment.
- It isolates actions, so they can't clobber each other's files.
- It restricts actions' resource usage, so that one memory-hungry test can't cause a linker running on the same worker to run out of memory.
Windows Containers provides built-in support for containers. We use Docker to run containers in process isolation mode (see Isolation modes for the difference with Hyper-V mode).
Before using EngFlow, you'll need to prepare a container image. Follow Create a container image for remote actions for detailed instructions and requirements. Windows container images are much larger than Linux images. A minimal Windows Server Core image is at least 6 GiB. Installing Visual Studio tools can double the size.
EngFlow workers run on Windows Server 2022. This means you'll need a Windows Server 2022 or Windows 11 host to build and run your container. See Windows container version compatibility.
After you've built a container image, you'll need to configure a C++ toolchain for use with Bazel. See Create a Bazel C++ toolchain configuration for remote actions.
Worker startup time and autoscaling¶
Windows worker machines take longer to start, compared with Linux workers. Depending on the size and location of your container image, it may take 10–20 minutes for a new worker to boot, download the image, extract it, and start serving requests.
To avoid queueing actions during peak times, we recommend a Windows pool with a fixed number of instances or a larger minimum number of instances than a Linux pool. For added stability, you may wish to consider using on-demand instances instead of spot instances.
Contact our support team for all cluster performance and tuning issues.
No cross compilation¶
Bazel has limited support for cross-compilation from a Windows host with a non-Windows executor or vice versa. If you're using Windows remote execution, we recommend doing so only from a Windows host.
A common issue is caused by an assumption in Bazel's file system library that host paths have the same structure as remote paths. For example, Bazel does not correctly identify C:/a/b.exe as an absolute path on a Linux host.
Runfiles¶
Bazel runfiles are files made available to actions at run-time, other than explicit input files.
When Bazel runs test actions locally, it creates a runfiles manifest file that maps runfile names to absolute paths in Bazel's local cache. This cannot be used with remote actions: Bazel sets the RUNFILES_MANIFEST_ONLY and RUNFILES_MANIFEST_FILE environment variables but cannot provide the manifest, since it doesn't know where the files will be located on the remote machine.
As a workaround, EngFlow removes the RUNFILES_MANIFEST_ONLY and RUNFILES_MANIFEST_FILE environment variables from all actions. Most runfiles libraries will then expect to find runfiles in a location relative to the current working directory.
Long paths¶
Windows paths are limited by default to 260 characters. See Maximum Path Length Limitation for details.
EngFlow workers have long paths enabled. We recommend you enable long paths in your container image.
Not all Windows functions support long paths. In particular, CreateProcess does not work reliably with long paths. We recommend you keep paths within your workspace short to avoid this limitation, especially for commands.
MSVC include paths workaround¶
Bazel performs include validation after each C/C++ compile command. Bazel invokes the compiler with a flag causing it to report all files that were included. Bazel reads the output and verifies that all files were either part of the toolchain or were explicit input files. Bazel reports an error if other header files are included.
The Microsoft Visual Studio C/C++ compiler (MSVC) reports included files using absolute paths. Bazel expects these paths to match the host system and reports errors when they don't (#19733).
To work around this issue, we provide an open source wrapper, msvc_filter_includes.go which converts absolute paths to relative paths within the exec root directory. To use this wrapper, see our example toolchain and Dockerfile.
Out of memory behavior¶
When a Windows process attempts to allocate more memory than is available, the function that allocates memory returns an error code. For example, in C, malloc may return NULL. Programs may handle this differently: most treat it as a fatal error and attempt to exit gracefully.
In contrast, Linux overcommits memory: a process may allocate a large amount of virtual memory even if there is not enough physical memory to back it. When a process attempts to use more memory than is available (by reading or writing, causing a page fault), the kernel selects a process at random (weighted by memory usage) and terminates it with SIGKILL.
EngFlow assumes that a Linux action that terminates with SIGKILL ran out of memory. Schedulers may retry such an action on another pool with more memory.
This is not possible on Windows, since processes that run out of memory don't terminate in a way the worker can reliably detect.