Authentication

This document describes how to setup server and client authentication for a EngFlow Remote Execution cluster.

Server Authentication

The only supported mechanism for server authentication is TLS. In order to use it, you have to set the --tls_certificate and --tls_key flags. You can either use an existing certificate authority, or you can use self-signed certificates.

For Bazel users: by default, Bazel trusts the set of root certificates that is shipped with the JDK. If you are using a non-standard certificate authority, you have to configure Bazel to accept its certificate using Bazel's --tls_certificate flag.

Beware that Bazel only allows specifying one custom CA certificate. If your --remote_executor, --remote_cache, and --bes_backend endpoints are different, their TLS certificates must all be signed by this authority, or by a publicly trusted authority.

Client Authentication (gRPC)

The EngFlow Remote Execution Service supports several mechanisms to authenticate gRPC clients and authorize requests.

To configure authentication, use the --client_auth flag: none, mtls, gcp_rbe, and external.

To configure authorization, check out the details of the specific authentication mechanisms below.

Which --client_auth to use?

If you already setup GCP RBE access roles, and every user has a Google account, then we recommend using the gcp_rbe mechanism, which allows continued use of your existing IAM setup. This method relies on Google authenticating the clients, so schedulers need to be able to talk to Google's servers.

If you want to use a custom authentication service, use external. Please contact us for details; you'll need to implement a binary with a specific gRPC interface.

If you already have a TLS certificate authority and distribute client certificates to your clients, and you use a client that supports mTLS (e.g. Bazel 3.1 or newer), then we recommend using the mtls mechanism. This mechanism works even without internet access, because the client CA's certificate is deployed to schedulers.

Otherwise you will have to decide whether to setup a certificate authority (which may require setting up additional infrastructure, such as HashiCorp Vault or smallstep), or use a VPN (none). Like mtls, these mechanism don't rely on external identity providers, so they work on clusters that have no internet access.

None

Usage: --client_auth=none

If you disable client authentication, anyone who can initiate a network connection to the cluster can use it. This must only be used in combination with network-based usage restrictions, e.g., over a VPN.

Authorization: The --principal_based_permissions flag controls permissions. With --client_auth=none, the principal name is always empty, so the only meaningful setting is --principal_based_permissions=*->ROLE (where ROLE is for example user or admin.)

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

Deny

Usage: --client_auth=deny

Denies all gRPC requests.

Apart from testing, this could be used to cut off gRPC access to a cluster without shutting it down. The cluster could still serve HTTP requests with the right --http_auth flag.

Authorization: With --client_auth=deny, no principals can be authorized.

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every authenticated client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

Example:

Generating a htpasswd file:

htpasswd -cm /etc/engflow/htpasswd alice

The generated entry will look like:

alice:$apr1$ILfUslW9$cXOjd6w4WoZrFKfD3Xhe91

By default, all authenticated users have the "user" role. You can specify more detailed permissions with --principal_based_permissions.

Example (/etc/engflow/config snippet):

--basic_auth_htpasswd=/etc/engflow/htpasswd
--principal_based_permissions=alice->admin
--principal_based_permissions+=bob->cache-reader

(Note the = and += operators: --principal_based_permissions is a list-type flag; = overrides the default flag value.)

If you are using Bazel, create a ~/.netrc file with an entry for the remote execution service address.

Example (~/.netrc):

machine demo.engflow.com
login alice
password foo

To use an alternative netrc file, set its path in the NETRC environment variable before running Bazel.

mTLS

Usage: --client_auth=mtls

mTLS or mutual TLS authentication requires each client to present a signed client TLS certificate whenever it establishes a connection to the cluster. Use the --tls_trusted_certificate flag to configure the certificate authority that is trusted to authenticate clients.

Remember to configure fine-grained permissions with --principal_based_permissions flag. The principal is the Common Name field in the client TLS certificate.

If you are using Bazel, you can configure it using the --tls_client_certificate and --tls_client_key flags.

Authorization: The --principal_based_permissions flag controls permissions. With --client_auth=mtls, the principal name is the Common Name (CN) of the client certificate. The CN can be any string supported by X.509 certificates, such as plain user names, email addresses, or FQDNs. If it's an email address then you can use the *@example.com syntax to mean everyone under that email domain. Otherwise you must use the exact principal name, or * to mean "everyone".

--principal_based_permissions=alice->admin
--principal_based_permissions+=bob@example.com->user
--principal_based_permissions+=*@example.com->cache-reader
--principal_based_permissions+=chuck@example.com->none

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every authenticated client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every authenticated client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

GCP RBE

Usage: --client_auth=gcp_rbe

GCP RBE-based authentication also uses GCP OAuth 2.0 bearer tokens. However, instead of relying on verified email addresses, it queries GCP's IAM for the Google-defined Remote Build Executor permissions, and unlike other --client_auth methods it therefore ignores --principal_based_permissions.

In order to use this authentication mechanism, you must specify a GCP project using the --gcp_rbe_auth_project flag.

If you are using Bazel, you can configure it using either --google_default_credentials or --google_credentials.

Authorization: Unlike other --client_auth mechanisms, the --principal_based_permissions flag is ignored with --client_auth=gcp_rbe. You have to specify permissions in GCP IAM, as described below.

The Google GCP permissions and roles are documented here: https://cloud.google.com/iam/docs/understanding-roles#other-roles

The mapping between GCP permissions and EngFlow permissions is incomplete: the EngFlow Remote Execution service supports only a subset of the RBE permissions and roles, and GCP defines only a subset of the permissions that EngFlow supports.

GCP RBE Permissions:

• remotebuildexecution.actions.create to run an action remotely
• remotebuildexecution.actions.delete to delete an action cache entry (this is not used by Bazel)
• remotebuildexecution.actions.get to read an action cache entry
• remotebuildexecution.actions.write to write an action cache entry from a client (remotely executed actions always have write access to the action cache)
• remotebuildexecution.blobs.create to write an entry to the CAS
• remotebuildexecution.blobs.get to read an entry from the CAS or query whether an entry is in the CAS based on its digest

These are the relevant roles, though note that you can create custom roles for different subsets of permissions:

• Remote Build Execution Artifact Creator aka roles/remotebuildexecution.artifactCreator

  Can run actions remotely. This is the most commonly used role, and maps to the user role defined in EngFlow RE.

• Remote Build Execution Artifact Admin aka roles/remotebuildexecution.artifactAdmin

  Can run actions remotely, and also delete actions. Cannot write actions to the cache.

• Remote Build Execution Action Cache Writer aka roles/remotebuildexecution.actionCacheWriter

  Can write CAS and action cache entries. This is primarily useful when using the system as a pure cache without remote execution. In this use case, the CI system should be allowed to read and write the cache (requires both this role and the Remote Build Execution Artifact Viewer), while individual engineers are only allowed to read the cache.

• Remote Build Execution Artifact Viewer aka roles/remotebuildexecution.artifactViewer

  Can read CAS and action cache entries. This is primarily useful when using the system as a pure cache without remote execution.

Build Event Streams with --client_auth=gcp_rbe

As of 2021-09-29, GCP IAM has no permissions to control Build Event Stream uploads that we could map to EngFlow Remote Execution permissions.

When using --client_auth=gcp_rbe, EngFlow RE versions support Build Event Stream uploads the following way:

v1.52 and older: don't let clients upload Build Event Streams.

v1.53 and newer: allow clients with remotebuildexecution.blobs.create permission to also upload Build Event Streams.

External

Usage: --client_auth=external

Since v1.53 you can use an external service to authenticate and authorize gRPC client requests.

This must be a service running on localhost next to the scheduler, communicating over gRPC. Please contact us for details.

Authorization: Unlike other --client_auth mechanisms, the --principal_based_permissions flag is ignored with --client_auth=external. The external service is responsible for authorization too.

GitHub Tokens

Usage: --client_auth=github_token --experimental_github_auth_container=foo/hello-world:1.0

You can use GitHub-signed tokens as credentials. The token can be a GITHUB_TOKEN (available to GitHub Actions runners) or a Personal Access Token. More info: https://docs.github.com/en/actions/security-guides/automatic-token-authentication

Assuming the token is stored in an envvar ($GITHUB_TOKEN), run Bazel with --remote_header=Authorization=Bearer $GITHUB_TOKEN.

Example (.github/workflows/engflow.yml GitHub Actions snippet):

...
- env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  name: 'Run tests'
  run: bazel test --config=remote-engflow --remote_header="Authorization=Bearer $GITHUB_TOKEN" //...
...

In order to use GitHub tokens you need to add to your remote configuration a GitHub container using --experimental_github_auth_container. It specifies an existing container on ghcr.io. The container should be private. Only GitHub Action runners of the organisation with a valid GITHUB_TOKEN should have access. Other than that the container can be anything; it won't be pulled nor ran, only have its existence checked.

Example: for the foo GitHub organization with a private container hello-world:1.0:

--client_auth=github_token
--experimental_github_auth_container=foo/hello-world:1.0

Authorization: The --principal_based_permissions flag controls permissions. Set this to --principal_based_permissions=*->admin.

This kind of client_auth is still a work in progress. You should try to use other methods if possible. If you have any questions, please contact the EngFlow team.

Client Authentication (HTTP)

The EngFlow Remote Execution service can be configured to store build events and serve a build results UI; see --enable_bes.

When enabled, you can visit the Remote Execution address in a browser and view build results. Client request authentication can be configured with the --http_auth flag. As of v2.29 the supported values include: none, deny, basic, google_login, oidc_login and okta_login.

None

Usage: --http_auth=none

If you disable client authentication, anyone who can initiate a network connection to the cluster can use it. This must only be used in combination with network-based usage restrictions, e.g., over a VPN.

In order to change the default permissions, use the --principal_based_permissions flag, for example --principal_based_permissions=*->admin. For guidance, see UI Access Permissions.

Authorization: same as --client_auth=none.

Deny

Usage: --http_auth=deny

Denies all HTTP requests.

This is useful when the Build Event Service is disabled on the cluster, or when you want to cut off HTTP access to a cluster without shutting it down. The cluster could still serve gRPC requests with the right --client_auth flag.

Authorization: same as --client_auth=deny.

Basic

Usage: --http_auth=basic

Basic authentication uses user name / password pairs.

Starting with v2.15.0, this does not use HTTP basic auth. Instead, it requires login through a web page.

To enable basic authentication: create a htpasswd file with the user-password pairs, copy it to every scheduler instance, then add its path to the config file with --basic_auth_htpasswd.

The password entries must be in apr1 format (Apache MD5). See https://httpd.apache.org/docs/2.4/misc/password_encryptions.html for details.

Authorization: The --principal_based_permissions flag controls permissions. With --http_auth=basic, the principal name is the user name in the Basic Auth credentials. An example setting:

--principal_based_permissions+=alice->admin

Additionally, review the version support for restrictions and customization options.

Google Login

Usage: --http_auth=google_login

Authenticates clients through a Google sign-in page. You need to create an OAuth 2.0 Client ID in GCP, then configure --principal_based_permissions for allowed clients.

Authorization: The --principal_based_permissions flag controls permissions. With --http_auth=google_login, the principal name is the email address of the client. You can use exact principal names, or *@example.com to mean everyone under that email domain, or * to mean "everyone".

Additionally, review the version support for restrictions and customization options.

Setup:

1. Configure the OAuth Consent Screen in GCP.

   You can use the same GCP project as the one hosting the Remote Execution cluster, or use a different one.

   If all clients are in the same organization (e.g. everyone has an @your-company.com Google account), then you can set User Type to Internal.

   To support clients from multiple email domains (e.g. *@your-company.com, *@partner-company.com), you need to set User Type to External.

   Do not upload a Logo, and don't configure any App Domains nor any Scopes. If you do, the app must be verified by Google. Once you upload a Logo it's not possible to take it down, nor to cancel the verification process). So just leave every non-mandatory field blank.

   Once you've completed the steps, click on Publish. This will immediately change the status to In Production, without needing verification.

2. Create an OAuth 2.0 Client ID in GCP.

   For Application Type, select Web application.

   For Authorized JavaScript origins, add the base URL of the cluster's HTTP endpoint, e.g. https://YOUR-CLUSTER.engflow.com:443/.

   For Authorized redirect URIs, add https://YOUR-CLUSTER.engflow.com:443/google-redirect, adjusting the base URL to the cluster's HTTP endpoint.

   Once you are done, note down the Client ID. You do not need the Client secret.

3. Set the Client ID, and allowed principals in the EngFlow config.

   Example:

   --http_auth=google_login
   --google_client_id=123456789-ab12cd34.apps.googleusercontent.com
   --principal_based_permissions=*@example.com->admin
   --principal_based_permissions+=*@example-partner.com->user
   

Open ID Connect Login

For v2.18.1 and newer only.

Usage: --http_auth=oidc_login

Authenticates clients through Open ID Connect. You need to create an OAuth 2.0 Client ID, then configure --principal_based_permissions for allowed clients.

Authorization: The --principal_based_permissions flag controls permissions. With --http_auth=oidc_login, the principal name is the email address of the client. You can use exact principal names, or *@example.com to mean everyone under that email domain, or * to mean "everyone".

Additionally, review the version support for restrictions and customization options.

Setup:

1. Create an OAuth 2.0 Client ID configured for the authorization code flow. Ensure the supported scopes include email.

   With this flow, the initial request for authentication will first return a code. A second, internal request which includes the client_secret is then sent to receive the id_token used for authorizing the user.

   • The sign-in redirect URIs need to include https://YOUR-CLUSTER.engflow.com:8080/oidc-redirect, adjusting the base URL YOUR-CLUSTER.engflow.com:8080 to your cluster's HTTP endpoint.
   • The sign-out redirect URIs needs to include your cluster's HTTP endpoint, https://YOUR-CLUSTER.engflow.com:8080/.

2. Note down the client_id and client_secret.

3. Locate the URI that lists the OAuth config, including the endpoints.

   • For okta, the URI is listed as the Metadata URI in the details of the Authorization Server, found under Security > API. The URI usually has the form https://{yourOktaDomain}/oauth2/{authorizationServerName}/.well-known/oauth-authorization-server.
   • For Keycloak, the URI is listed under Endpoints in the Realm Settings > General . Depending on the version of Keycloak you are using, the URI may take on one of the following forms:
     • http://{keycloakhost:keycloakport}/auth/realms/{realm}/.well-known/openid-configuration
     • http://{keycloakhost:keycloakport}/realms/{realm}/.well-known/openid-configuration

4. From the OAuth config, note down the entries authorization_endpoint, token_endpoint and jwks_uri. Also confirm that scopes_supported includes email.

5. Create a JSON file with the following contents:

   {
     "client_id": "[`client_id` noted down previously]",
     "client_secret": "[`client_secret` noted down previously]",
     "auth_token_endpoint": "[`authorization_endpoint` from the OAuth config]",
     "token_endpoint": "[`token_endpoint` from the OAuth config]",
     "keys_endpoint": "[`jwks_uri` from the OAuth config]"
   }
   

6. Update the EngFlow config.

   Example:

   --http_auth=oidc_login
   --oidc_config=/path/to/json_file
   --principal_based_permissions=*@example.com->admin
   --principal_based_permissions+=*@example-partner.com->user
   

   As the config includes the client_secret, in production the JSON file should be stored in a secrets manager. The value for --oidc_config then does not specify an absolute path to the file, but is prepended by secretstore://, followed by the secret's location, e.g.

   --oidc_config=secretstore://arn:aws:secretsmanager:{region}:{accountId}:secret:{secretName}
   

Okta Login (Implicit flow only)

For v2.15 and newer only.

Usage: --http_auth=okta_login

Info

This authentication method requires Okta to be set up to use the implicit flow, whereby an ID token is granted directly. If you want to use the authorization code flow instead, use --http_auth=oidc_login instead.

Authenticates clients through an okta sign-in page. You need to create an OAuth 2.0 Client ID in okta, then configure --principal_based_permissions for allowed clients.

Authorization: The --principal_based_permissions flag controls permissions. With --http_auth=okta_login, the principal name is the email address of the client. You can use exact principal names, or *@example.com to mean everyone under that email domain, or * to mean "everyone".

Additionally, review the version support for restrictions and customization options.

Setup:

1. Create an OAuth 2.0 Client ID in okta using the App Integration Wizard (AIW).

   In the Applications section, create a app integration.

   For Sign-in method, select OIDC - OpenID Connect.

   For Application Type, select Web Application.

   For Grant type, select Implicit (hybrid).

   For Sign-in redirect URIs, add https://YOUR-CLUSTER.engflow.com:8080/okta-redirect, adjusting the base URL to the cluster's HTTP endpoint.

   For Sign-out redirect URIs, add the base URL of the cluster's HTTP endpoint, e.g. https://YOUR-CLUSTER.engflow.com:8080/.

   For Assignments choose the appropriate setting for your use case.

   Upon saving, ensure the Grant type setting has Allow ID Token with implicit grant type enabled. Allow Access Token with implicit grant type is not required.

   In the Okta API Scopes of your application grant the scope okta.users.read.self.

   Once you are done, note down the Client ID. You do not need the Client secret.

   Visit okta's Help Center for more information on how to create a OIDC app integrations.

2. Configure the Issuer URI in okta.

   In the Security section, subsection API either set up an authorization server or use the default setup. Ensure the server includes the scopes openid and email.

   Once done, note down the Issuer URI.

3. Update the EngFlow config.

   Example:

   --http_auth=okta_login
   --okta_client_id=1234567890abcdefghij
   --okta_issuer_uri=https://dev-12345678.okta.com/oauth2/default
   --principal_based_permissions=*@example.com->admin
   --principal_based_permissions+=*@example-partner.com->user
   

Version Support

UI Access Permissions

To grant access to the UI --principal_based_permissions needs to be specified.

• v1.54 and older:
  • The default value of --principal_based_permissions is ["*->user"], therefore every client has the user permissions.
  • However, clients need admin access to view the UI.
• v1.55 to v2.14:
  • The default value of --principal_based_permissions is [].
  • Clients need admin access to view the UI.
• v2.15 and newer:
  • The default value of --principal_based_permissions is [].
  • Clients need admin or user access to view the UI.

Note that a user may authenticate even if no permission is set up for their principal. However, they will not be able to access the UI (or any other resources), unless a permission is set up that authorizes them to do so. This means it is safe to use a less restrictive authentication mechanism (e.g. a common Google or okta auth setup), and control access here more strictly.

JWT Expiration

Some methods for HTTP client authentication create a session cookie using a JSON Web Token (JWT) for authentication.

• v2.14 and older: google_login
• v2.15 and newer: basic, google_login and okta_login

You can set the expiry of such JWTs using --experimental_web_login_expiration.

Multi-value support

For v2.14 and older --http_auth can only have one value.

For v2.15 and newer, you can enable multiple authentication methods for HTTP client authentication, provided they all create a cookie with a JSON Web Token (JWT) for authentication. These include basic, google_login and okta_login. Other methods cannot be combined.

Example:

--http_auth=google_login
--google_client_id=123456789-ab12cd34.apps.googleusercontent.com

--http_auth+=okta_login
--okta_client_id=1234567890abcdefghij
--okta_issuer_uri=https://dev-12345678.okta.com/oauth2/default

--http_auth+=basic
--basic_auth_htpasswd=/etc/engflow/htpasswd

--principal_based_permissions=*@example.com->admin
--principal_based_permissions+=*@example-partner.com->user

Note the = and += operators:

Both --http_auth and --principal_based_permissions are list-type flags. The = operator drops all previous values, whereas += adds another value.