This document describes how to configure and use executor pools in an EngFlow Remote Execution cluster. An executor pool is a set of identical units each of which can execute one action.
Named Executor Pools
Whenever it receives an action, the scheduler selects a worker instance and an executor on that worker instance to run the action. With no further configuration, the scheduler selects any available instance.
This selection process can be statically controlled through the use of named executor pools. To do so, you have to configure the names in the worker pool and also configure the client to specify the name of the executor pool for some or all actions.
Configuring the Executor Pool Name
The executor pool name is configured through the
flag, which accepts a
pool parameter, e.g.,
All executors with the same pool name are considered interchangable. The scheduler only executes actions with the exact same action pool name in a given executor pool, and also disregards any other platform settings for executor selection.
Autoconfiguring the Pool
Worker pools can be configured to use all available CPUs using the value
Configuring the Action Pool Name
For Bazel, you can configure the pool name either globally for all actions, or on a rule-by-rule basis. We recommend using the bazel-toolchains project to configure executor selection as well as action isolation (see Action Environment).
In order to set a pool name for all actions, you can use the
stanza in your
WORKSPACE file. If your
WORKSPACE file looks like this:
rbe_autoconfig( name = "engflow_remote_config", detect_java_home = True, registry = "<DOCKER_REGISTRY>", # TODO: use actual registry address repository = "<REPOSITORY>", # TODO: use actual repository name digest = "sha256:<DIGEST>", # TODO: use the actual digest; or use tag="latest"
then add the pool configuration like this:
exec_properties = create_rbe_exec_properties_dict( pool = "linux", ), )
Alternatively, you can specify a pool for a single test rule, like so:
sh_test( name = "pool", srcs = ["pool.sh"], exec_properties = create_rbe_exec_properties_dict( pool = "linux", ), )