Executor Pools

This document describes how to configure and use executor pools in an EngFlow Remote Execution cluster. An executor pool is a set of identical units each of which can execute one action.

Named Executor Pools

Whenever it receives an action, the scheduler selects a worker instance and an executor on that worker instance to run the action. With no further configuration, the scheduler selects any available instance.

This selection process can be statically controlled through the use of named executor pools. To do so, you have to configure the names in the worker pool and also configure the client to specify the name of the executor pool for some or all actions.

Configuring the Executor Pool Name

The executor pool name is configured through the --worker_config flag, which accepts a pool parameter, e.g., --worker_config=1*cpu=2,pool=linux.

All executors with the same pool name are considered interchangeable. The scheduler only executes actions with the exact same action pool name in a given executor pool, and also disregards any other platform settings for executor selection.

Autoconfiguring the Pool

Worker pools can be configured to use all available CPUs using the value auto: --worker_config=auto.

Configuring the Action Pool Name

For Bazel, you can configure the pool name either globally for all actions, or on a rule-by-rule basis. We recommend using the bazel-toolchains project to configure executor selection as well as action isolation (see Action Environment).

In order to set a pool name for all actions, you can set the "Pool" execution property in your remote platform rule:

platform(
    name = "engflow_remote",
    ...

then add the pool configuration like this:

    exec_properties = {
        "Pool": "linux",
    },
)

Alternatively, you can specify a pool for a single test rule, like so:

sh_test(
    name = "pool",
    srcs = ["pool.sh"],
    exec_properties = {
        "Pool": "linux",
    },
)