Auto-Scaling

This document describes how to configure auto-scaling worker pools in an EngFlow Remote Execution cluster. Auto-scaling describes the process of automatically adjusting the number of workers or schedulers to the incoming load. Some of the notes here also apply to the use of VM instances with reduced availability such as GCP preemptible or AWS spot instances.

While the EngFlow Remote Execution software is designed to gracefully handle instance failures and use additional resources that are added to the cluster, auto-scaling may remove more machines than the cluster can handle at once.

We first give an overview of the built-in failure recovery and scaling mechanisms, and then discuss options for handling cases that go beyond the built-in facilities. Finally, we provide an example for setting up auto-scaling on GCP, and describe the cluster tier that auto-scaling needs on Azure Kubernetes Service (AKS).

Built-in Failure Recovery & Scaling Mechanisms

The EngFlow Remote Execution software is designed to transparently handle individual instance failures, both of workers and schedulers, and to automatically start using new instances as they are added to the cluster.

If the built-in distributed CAS is used, files are replicated up to three times (see --replica_count), so the system can handle up to two simultaneous worker failures without client-visible failures or data loss.

Similarly, action cache entries are replicated up to three times (see --metadata_replica_count), so the system can handle up to two simultaneous scheduler failures. Note that the action cache is exclusively stored on the schedulers.

However, auto-scaling will typically remove more than two instances at the same time when a significant drop in incoming load is detected.

Auto-Scaling More than 2 Instances

There are three options for handling the case where more than two machines are shut down at the same time:

1. Rely on the client to re-upload the files / re-run the actions
2. Separate CAS and workers, and disable auto-scaling for CAS instances
3. Use an external persistent storage service

1. Relying on the Client

If the client has a copy of all the relevant files, then a loss of files in the CAS can be automatically recovered from, by having the client re-upload the corresponding files. Unless the client is explicitly configured to not keep copies, and it correctly handles server errors, this approach requires no additional configuration. Similarly for loss of action cache entries.

Note that Bazel's Build-without-the-Bytes feature explicitly disables downloading output files to the client, which can result in build errors if more than two instances fail or are removed simultaneously.

If you are planning to enable this feature in Bazel, we strongly recommend also enabling one of the other options below to avoid intermittent build failures, and also increasing the replica retention time (see --default_replica_timeout).

2. Separating CAS and Workers

A second option to increase reliability with auto-scaling is to use separate CAS and worker pools, and only enable auto-scaling for the worker pool. This requires configuring the worker pool without replica space - this is a signal to the worker to avoid storing file metadata.

Otherwise loss of workers can lead to loss of file metadata, which can make files unrecoverable even if they are still present on the CAS instances. Note that even when workers store no replicas, they still need CAS space for action input and output files as well as local cache.

You cannot use the generic --disk_size option, but you have to explicitly configure the CAS layout.

This is an example configuration:

• CAS instances:

  --worker_config=
  --max_cas_size=200gb
  --max_replica_size=200gb
  

• Workers:

  --worker_config=1*cpu=2
  --disk_size=0
  --max_cas_size=8gb
  --max_replica_size=0
  

3. Configuring Additional File Backups

If the built-in mechanisms are not sufficient, it is possible to configure a cluster to rely on an external persistent storage service to keep an additional copy of files and action cache entries. If files or file metadata is lost, the cluster then automatically falls back to the persistent storage service.

See Content-Addressable Storage for details.

GCP Auto-Scaling

You can use GCP's existing auto-scaler in combination with Google Cloud Operations (formerly StackDriver) monitoring and Google Cloud Storage backups.

You should first enable metrics export to GCO and verify that you can see metrics in the GCP Cloud Console. Enable metrics export to GCO by adding this to your worker configuration:

--enable_stackdriver=true
--monitoring_stackdriver_project=${var.project_name}

Automatic Setup with Terraform

If you are using the included Terraform configuration file as part of your GCP setup, then you may have already created an auto-scaler for your worker pool that uses the used_executors metric.

In this case, you only need to update the Terraform configuration by setting the enable_autoscaler Terraform variable to true:

variable "enable_autoscaler" {
        type    = bool
        default = true
}

Manual Setup

Alternatively, you can manually configure the auto-scaler through the GCP console. The auto-scaler is tied to your worker pool's instance group manager. Select custom.googleapis.com/opencensus/com.engflow.re.exec/used_executors as the metric with a target utilization of 0.25.

AKS Auto-Scaling

If you run EngFlow on Azure Kubernetes Service (AKS), create the cluster with the Standard pricing tier or higher. In EngFlow's testing, AKS clusters on the Free tier scaled workers slowly, while clusters on the Standard tier were much faster. Microsoft recommends the Free tier only for development and test clusters with fewer than 10 nodes, and it has no uptime SLA for the Kubernetes API server.

The Terraform resource azurerm_kubernetes_cluster defaults to Free, so set the tier explicitly:

resource "azurerm_kubernetes_cluster" "engflow" {
  # ...

  # Free (the default) is meant for dev/test clusters and scales workers slowly.
  sku_tier = "Standard"
}

To check or change the tier of an existing cluster, use the Azure CLI:

az aks show --resource-group <resource group> --name <cluster> --query sku.tier -o tsv
az aks update --resource-group <resource group> --name <cluster> --tier standard

The Standard tier adds a per-cluster hourly charge. See AKS pricing.