Authentication

This document describes how to setup server and client authentication for a EngFlow Remote Execution cluster.

Server Authentication

The only supported mechanism for server authentication is TLS. In order to use it, you have to set the --tls_certificate and --tls_key flags. You can either use an existing certificate authority, or you can use self-signed certificates.

For Bazel users: by default, Bazel trusts the set of root certificates that is shipped with the JDK. If you are using a non-standard certificate authority, you have to configure Bazel to accept its certificate using Bazel's --tls_certificate flag.

Beware that Bazel only allows specifying one custom CA certificate. If your --remote_executor, --remote_cache, and --bes_backend endpoints are different, their TLS certificates must all be signed by this authority, or by a publicly trusted authority.

Client Authentication (gRPC)

The EngFlow Remote Execution Service supports several mechanisms to authenticate gRPC clients and authorize requests.

To configure authentication, use the --client_auth flag: none, mtls, gcp_rbe, and external.

To configure authorization, check out the details of the specific authentication mechanisms below.

Which --client_auth to use?

If you already setup GCP RBE access roles, and every user has a Google account, then we recommend using the gcp_rbe mechanism, which allows continued use of your existing IAM setup. This method relies on Google authenticating the clients, so schedulers need to be able to talk to Google's servers.

If you want to use a custom authentication service, use external. Please contact us for details; you'll need to implement a binary with a specific gRPC interface.

If you already have a TLS certificate authority and distribute client certificates to your clients, and you use a client that supports mTLS (e.g. Bazel 3.1 or newer), then we recommend using the mtls mechanism. This mechanism works even without internet access, because the client CA's certificate is deployed to schedulers.

Otherwise you will have to decide whether to setup a certificate authority (which may require setting up additional infrastructure, such as HashiCorp Vault or smallstep), or use a VPN (none). Like mtls, these mechanism don't rely on external identity providers, so they work on clusters that have no internet access.

None

Usage: --client_auth=none

If you disable client authentication, anyone who can initiate a network connection to the cluster can use it. This must only be used in combination with network-based usage restrictions, e.g., over a VPN.

Authorization: The --principal_based_permissions flag controls permissions. With --client_auth=none, the principal name is always empty, so the only meaningful setting is --principal_based_permissions=*->ROLE (where ROLE is for example user or admin.)

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

Deny

Usage: --client_auth=deny

Denies all gRPC requests.

Apart from testing, this could be used to cut off gRPC access to a cluster without shutting it down. The cluster could still serve HTTP requests with the right --http_auth flag.

Authorization: With --client_auth=deny, no principals can be authorized.

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every authenticated client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

Example:

Generating a htpasswd file:

htpasswd -cm /etc/engflow/htpasswd alice

The generated entry will look like:

alice:$apr1$ILfUslW9$cXOjd6w4WoZrFKfD3Xhe91

By default, all authenticated users have the "user" role. You can specify more detailed permissions with --principal_based_permissions.

Example (/etc/engflow/config snippet):

--basic_auth_htpasswd=/etc/engflow/htpasswd
--principal_based_permissions=alice->admin
--principal_based_permissions+=bob->cache-reader

(Note the = and += operators: --principal_based_permissions is a list-type flag; = overrides the default flag value.)

If you are using Bazel, create a ~/.netrc file with an entry for the remote execution service address.

Example (~/.netrc):

machine demo.engflow.com
login alice
password foo

To use an alternative netrc file, set its path in the NETRC environment variable before running Bazel.

mTLS

Usage: --client_auth=mtls

mTLS or mutual TLS authentication requires each client to present a signed client TLS certificate whenever it establishes a connection to the cluster. Use the --tls_trusted_certificate flag to configure the certificate authority that is trusted to authenticate clients.

Remember to configure fine-grained permissions with --principal_based_permissions flag. The principal is the Common Name field in the client TLS certificate.

If you are using Bazel, you can configure it using the --tls_client_certificate and --tls_client_key flags.

Authorization: The --principal_based_permissions flag controls permissions. With --client_auth=mtls, the principal name is the Common Name (CN) of the client certificate. The CN can be any string supported by X.509 certificates, such as plain user names, email addresses, or FQDNs. If it's an email address then you can use the *@example.com syntax to mean everyone under that email domain. Otherwise you must use the exact principal name, or * to mean "everyone".

--principal_based_permissions=alice->admin
--principal_based_permissions+=bob@example.com->user
--principal_based_permissions+=*@example.com->cache-reader
--principal_based_permissions+=chuck@example.com->none

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every authenticated client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every authenticated client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

GCP RBE

Usage: --client_auth=gcp_rbe

GCP RBE-based authentication also uses GCP OAuth 2.0 bearer tokens. However, instead of relying on verified email addresses, it queries GCP's IAM for the Google-defined Remote Build Executor permissions, and unlike other --client_auth methods it therefore ignores --principal_based_permissions.

In order to use this authentication mechanism, you must specify a GCP project using the --gcp_rbe_auth_project flag.

If you are using Bazel, you can configure it using either --google_default_credentials or --google_credentials.

Authorization: Unlike other --client_auth mechanisms, the --principal_based_permissions flag is ignored with --client_auth=gcp_rbe. You have to specify permissions in GCP IAM, as described below.

The Google GCP permissions and roles are documented here: https://cloud.google.com/iam/docs/understanding-roles#other-roles

The mapping between GCP permissions and EngFlow permissions is incomplete: the EngFlow Remote Execution service supports only a subset of the RBE permissions and roles, and GCP defines only a subset of the permissions that EngFlow supports.

GCP RBE Permissions:

• remotebuildexecution.actions.create to run an action remotely
• remotebuildexecution.actions.delete to delete an action cache entry (this is not used by Bazel)
• remotebuildexecution.actions.get to read an action cache entry
• remotebuildexecution.actions.write to write an action cache entry from a client (remotely executed actions always have write access to the action cache)
• remotebuildexecution.blobs.create to write an entry to the CAS
• remotebuildexecution.blobs.get to read an entry from the CAS or query whether an entry is in the CAS based on its digest

These are the relevant roles, though note that you can create custom roles for different subsets of permissions:

• Remote Build Execution Artifact Creator aka roles/remotebuildexecution.artifactCreator

  Can run actions remotely. This is the most commonly used role, and maps to the user role defined in EngFlow RE.

• Remote Build Execution Artifact Admin aka roles/remotebuildexecution.artifactAdmin

  Can run actions remotely, and also delete actions. Cannot write actions to the cache.

• Remote Build Execution Action Cache Writer aka roles/remotebuildexecution.actionCacheWriter

  Can write CAS and action cache entries. This is primarily useful when using the system as a pure cache without remote execution. In this use case, the CI system should be allowed to read and write the cache (requires both this role and the Remote Build Execution Artifact Viewer), while individual engineers are only allowed to read the cache.

• Remote Build Execution Artifact Viewer aka roles/remotebuildexecution.artifactViewer

  Can read CAS and action cache entries. This is primarily useful when using the system as a pure cache without remote execution.

Build Event Streams with --client_auth=gcp_rbe

As of 2021-09-29, GCP IAM has no permissions to control Build Event Stream uploads that we could map to EngFlow Remote Execution permissions.

When using --client_auth=gcp_rbe, EngFlow RE versions support Build Event Stream uploads the following way:

v1.52 and older: don't let clients upload Build Event Streams.

v1.53 and newer: allow clients with remotebuildexecution.blobs.create permission to also upload Build Event Streams.

External

Usage: --client_auth=external

Since v1.53 you can use an external service to authenticate and authorize gRPC client requests.

This must be a service running on localhost next to the scheduler, communicating over gRPC. Please contact us for details.

Authorization: Unlike other --client_auth mechanisms, the --principal_based_permissions flag is ignored with --client_auth=external. The external service is responsible for authorization too.

GitHub Tokens

Usage: --client_auth=github_token --experimental_github_auth_container=foo/hello-world:1.0

You can use GitHub-signed tokens as credentials. The token can be a GITHUB_TOKEN (available to GitHub Actions runners) or a Personal Access Token. More info: https://docs.github.com/en/actions/security-guides/automatic-token-authentication

Assuming the token is stored in an envvar ($GITHUB_TOKEN), run Bazel with --remote_header=Authorization=Bearer $GITHUB_TOKEN.

Example (.github/workflows/engflow.yml GitHub Actions snippet):

...
- env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  name: 'Run tests'
  run: bazel test --config=remote-engflow --remote_header="Authorization=Bearer $GITHUB_TOKEN" //...
...

In order to use GitHub tokens you need to add to your remote configuration a GitHub container using --experimental_github_auth_container. It specifies an existing container on ghcr.io. The container should be private. Only GitHub Action runners of the organisation with a valid GITHUB_TOKEN should have access. Other than that the container can be anything; it won't be pulled nor ran, only have its existence checked.

Example: for the foo GitHub organization with a private container hello-world:1.0:

--client_auth=github_token
--experimental_github_auth_container=foo/hello-world:1.0

Authorization: The --principal_based_permissions flag controls permissions. Set this to --principal_based_permissions=*->admin.

This kind of client_auth is still a work in progress. You should try to use other methods if possible. If you have any questions, please contact the EngFlow team.

Client Authentication (HTTP)

The EngFlow Remote Execution service can be configured to store build events and serve a build results UI; see --enable_bes.

When enabled, you can visit the Remote Execution address in a browser and view build results. Client request authentication can be configured with the --http_auth flag. As of v2.29 the supported values include: none, deny, basic, google_login, and oidc_login.

None

Usage: --http_auth=none

If you disable client authentication, anyone who can initiate a network connection to the cluster can use it. This must only be used in combination with network-based usage restrictions, e.g., over a VPN.

In order to change the default permissions, use the --principal_based_permissions flag, for example --principal_based_permissions=*->admin. For guidance, see UI Access Permissions.

Authorization: same as --client_auth=none.

Deny

Usage: --http_auth=deny

Denies all HTTP requests.

This is useful when the Build Event Service is disabled on the cluster, or when you want to cut off HTTP access to a cluster without shutting it down. The cluster could still serve gRPC requests with the right --client_auth flag.

Authorization: same as --client_auth=deny.

Basic

Usage: --http_auth=basic

Basic authentication uses user name / password pairs.

Starting with v2.15.0, this does not use HTTP basic auth. Instead, it requires login through a web page.

To enable basic authentication: create a htpasswd file with the user-password pairs, copy it to every scheduler instance, then add its path to the config file with --basic_auth_htpasswd.

The password entries must be in apr1 format (Apache MD5). See https://httpd.apache.org/docs/2.4/misc/password_encryptions.html for details.

Authorization: The --principal_based_permissions flag controls permissions. With --http_auth=basic, the principal name is the user name in the Basic Auth credentials. An example setting:

--principal_based_permissions+=alice->admin

Additionally, review the version support for restrictions and customization options.

Google Login

Usage: --http_auth=google_login

Warning

This value is deprecated. Use --http_auth=oidc_login instead.

The Google-specific setup instructions still apply, except Authorized redirect URIs need to include https://YOUR-CLUSTER.cluster.engflow.com/oidc-redirect, rather than https://YOUR-CLUSTER.cluster.engflow.com/google-redirect.

Authenticates clients through a Google sign-in page. You need to create an OAuth 2.0 Client ID in GCP, then configure --principal_based_permissions for allowed clients.

Authorization: The --principal_based_permissions flag controls permissions. With --http_auth=google_login, the principal name is the email address of the client. You can use exact principal names, or *@example.com to mean everyone under that email domain, or * to mean "everyone".

Additionally, review the version support for restrictions and customization options.

Setup

1. Configure the OAuth Consent Screen in GCP.

   You can use the same GCP project as the one hosting the Remote Execution cluster, or use a different one.

   If all clients are in the same organization (e.g. everyone has an @your-company.com Google account), then you can set User Type to Internal.

   To support clients from multiple email domains (e.g. *@your-company.com, *@partner-company.com), you need to set User Type to External.

   Do not upload a Logo, and don't configure any App Domains nor any Scopes. If you do, the app must be verified by Google. Once you upload a Logo it's not possible to take it down, nor to cancel the verification process). So just leave every non-mandatory field blank.

   Once you've completed the steps, click on Publish. This will immediately change the status to In Production, without needing verification.

2. Under Credentials, create an OAuth 2.0 Client ID in GCP.

   For Application Type, select Web application.

   For Authorized JavaScript origins, add the base URL of the cluster's HTTP endpoint, e.g. https://YOUR-CLUSTER.cluster.engflow.com/.

   For Authorized redirect URIs, add https://YOUR-CLUSTER.cluster.engflow.com/google-redirect, adjusting the base URL to the cluster's HTTP endpoint.

   Once you are done, note down the Client ID. You do not need the Client secret.

3. Set the Client ID, and allowed principals in the EngFlow config.

   Example:

   --http_auth=google_login
   --google_client_id=123456789-ab12cd34.apps.googleusercontent.com
   --principal_based_permissions=*@example.com->admin
   --principal_based_permissions+=*@example-partner.com->user
   

OpenID Connect Login

Usage: --http_auth=oidc_login

Authenticates clients through OpenID Connect (OIDC). You need to create an OAuth 2.0 Client ID with your identity provider (IdP), then configure --principal_based_permissions for allowed clients.

Authorization: The --principal_based_permissions flag controls permissions. With --http_auth=oidc_login, the principal name is the email address of the client. You can use exact principal names, or *@example.com to mean everyone under that email domain, or * to mean "everyone".

Additionally, review the version support for restrictions and customization options.

Setup: EngFlow supports using OIDC's Implicit Flow, Authorization Code flow and Hybrid Flow.

• Implicit flow: Upon authentication, the IdP directly returns an ID token.
• Authorization flow: The IdP returns an authorization code. An ID token can be requested from the IdP using the authorization code and a client secret.
• Hybrid flow: Combines both models. Tokens may be returned directly or an authorization code is sent.

Your identity provider may support multiple flows. In this case, choose the flow according to your organization's policy. EngFlow only requests very basic information about the account:

• scope openid to enable OIDC login,
• scope email to identify the user,
• optionally scope profile. If profile[picture] is supported by the IdP, signed-in users will see their public profile picture, instead of a default profile icon.

Setup: Authorization Code flow

1. Create an OAuth 2.0 Client ID configured for the authorization code flow.

   • supported scopes: include openid, email, and optionally profile
   • sign-out redirect URIs: include your cluster's HTTP endpoint, e.g.
     • https://YOUR-CLUSTER.engflow.com:8080/
   • sign-in redirect URIs: include the path /oidc-redirect (or a wildcard *) for your cluster's HTTP endpoint, e.g.
     • https://YOUR-CLUSTER.engflow.com:8080/oidc-redirect
     • https://YOUR-CLUSTER.engflow.com:8080/*

2. Note down the client_id and client_secret.

3. Locate your IdP's Issuer discovery URI, below referenced as discovery_uri.

   • Google: https://accounts.google.com/.well-known/openid-configuration.
   • okta: Commonly https://{yourOktaDomain}/.well-known/openid-configuration. If you have set up a custom Authorization Server, the URI is listed as the Metadata URI in the details of the Authorization Server, found under Security > API. Then the URI usually has the form https://{yourOktaDomain}/oauth2/{authorizationServerName}/.well-known/oauth-authorization-server.
   • Keycloak: the URI is listed under Endpoints in the Realm Settings > General . Depending on the version of Keycloak you are using, the URI may take on one of the following forms:
     • http://{keycloakhost:keycloakport}/auth/realms/{realm}/.well-known/openid-configuration
     • http://{keycloakhost:keycloakport}/realms/{realm}/.well-known/openid-configuration
   Without Issuer discovery

   If your IdP does not support Issuer discovery, instead locate the following endpoints: authorization_endpoint, token_endpoint and jwks_uri. Then create a JSON file with the following contents, rather than those included in the next step:

   {
     "client_id": "[`client_id` noted down previously]",
     "client_secret": "[`client_secret` noted down previously]",
     "auth_token_endpoint": "[`authorization_endpoint`]",
     "token_endpoint": "[`token_endpoint`]",
     "keys_endpoint": "[`jwks_uri`]"
   }
   

   The token endpoint is only required for the authorization code flow, not the implicit flow.

4. Create a JSON file with the following contents:

   {
     "client_id": "[`client_id` noted down previously]",
     "client_secret": "[`client_secret` noted down previously]",
     "discovery_uri": "[`discovery_uri` noted down previously]",
   }
   

5. Update the EngFlow config.

   Example:

   --http_auth=oidc_login
   --oidc_config=/path/to/json_file
   --principal_based_permissions=*@example.com->admin
   --principal_based_permissions+=*@example-partner.com->user
   

   As the config for the authorization flow includes the client_secret, the JSON file should be stored in a secrets manager. The value for --oidc_config then does not specify an absolute path to the file, but is prepended by secretstore://, followed by the secret's location, e.g.

   --oidc_config=secretstore://arn:aws:secretsmanager:{region}:{accountId}:secret:{secretName}
   

Setup: Implicit flow

The setup is similar to that for the Authorization code flow, with the following differences to the JSON file:

• Omit the entry for the client_secret.
• When not using Issuer discovery, the token_endpoint can also be omitted.

Setup: Implicit flow in Okta

The below gives a more detailed example for setting up the implicit flow in okta. For other IdPs, follow a similar process.

1. Create an OAuth 2.0 Client ID in okta using the App Integration Wizard (AIW).

   In the Applications section, create a app integration.

   For Sign-in method, select OIDC - OpenID Connect.

   For Application Type, select Web Application.

   For Grant type, select Implicit (hybrid).

   For Sign-in redirect URIs, add https://YOUR-CLUSTER.engflow.com:8080/oidc-redirect, adjusting the base URL to the cluster's HTTP endpoint.

   For Sign-out redirect URIs, add the base URL of the cluster's HTTP endpoint, e.g. https://YOUR-CLUSTER.engflow.com:8080/.

   For Assignments choose the appropriate setting for your use case.

   Upon saving, ensure the Grant type setting has Allow ID Token with implicit grant type enabled. Allow Access Token with implicit grant type is not required.

   In the Okta API Scopes of your application grant the scope okta.users.read.self.

   Once you are done, note down the Client ID. You do not need the Client secret.

   Visit okta's Help Center for more information on how to create a OIDC app integrations.

2. Determine the discovery URI.

   By default, the discovery endpoint should be available at the URI https://{yourOktaDomain}/.well-known/openid-configuration.

   Verify this URI loads a JSON file with entries authorization_endpoint, jwks_uri etc. The entry scopes_supported should at least include openid and email.

   If you want to set up a custom Authorization Server, do so in the Security section, subsection API. Ensure the server includes the scopes openid and email. Once done, note down the Metadata URI as the discovery URI. It usually has the form https://{yourOktaDomain}/oauth2/{authorizationServerName}/.well-known/oauth-authorization-server. Verify this URI loads the expected JSON file as described above.

3. Create a JSON file with the following contents:

   {
     "issuer": "OKTA",
     "client_id": "[`client_id` noted down previously]",
     "discovery_uri": "[`discovery URI` noted down previously]",
   }
   

4. Update the EngFlow config.

   Example:

   --http_auth=oidc_login
   --oidc_config=/path/to/okta_config_file
   --principal_based_permissions=*@example.com->admin
   --principal_based_permissions+=*@example-partner.com->user
   

Issuer-specific login buttons

If the --oidc_config does not include an entry for "issuer", the login page shows a generic OIDC login button.

The following issuer-specific buttons are supported: GOOGLE, KEYCLOAK, OKTA.

Example:

{
  "issuer": "GOOGLE",
  "client_id": "[...].apps.googleusercontent.com",
  "discovery_uri": "https://accounts.google.com/.well-known/openid-configuration",
}

Using multiple Identity Providers

EngFlow supports specifying multiple OIDC configurations. The UI login page then shows one login button for each IdP configured. Especially in this case, use issuer-specific login buttons, so users can differentiate the buttons. Reach out to us if you want a custom button for a not-yet listed IdP.

Example configuration:

--http_auth=oidc_login
--oidc_config=/path/to/google_config_file
--oidc_config+=/path/to/okta_config_file

Expiration

You can set the expiry of credentials using --experimental_web_login_expiration.

Version Support

UI Access Permissions

To grant access to the UI --principal_based_permissions needs to be specified.

• v1.54 and older:
  • The default value of --principal_based_permissions is ["*->user"], therefore every client has the user permissions.
  • However, clients need admin access to view the UI.
• v1.55 to v2.14:
  • The default value of --principal_based_permissions is [].
  • Clients need admin access to view the UI.
• v2.15 and newer:
  • The default value of --principal_based_permissions is [].
  • Clients need admin or user access to view the UI.

Note that a user may authenticate even if no permission is set up for their principal. However, they will not be able to access the UI (or any other resources), unless a permission is set up that authorizes them to do so. This means it is safe to use a less restrictive authentication mechanism (e.g. a common Google or okta auth setup), and control access here more strictly.

Multi-value support

For v2.14 and older --http_auth can only have one value.

For v2.15 and newer, you can enable multiple authentication methods for HTTP client authentication, provided they all create a cookie with a JSON Web Token (JWT) for authentication. These include basic, google_login and, since v2.29, oidc_login. Other methods cannot be combined.

Example:

--http_auth=oidc_login
--oidc_config=/path/to/config/file

--http_auth+=google_login
--google_client_id=123456789-ab12cd34.apps.googleusercontent.com

--http_auth+=basic
--basic_auth_htpasswd=/etc/engflow/htpasswd

--principal_based_permissions=*@example.com->admin
--principal_based_permissions+=*@example-partner.com->user

Note the = and += operators:

Both --http_auth and --principal_based_permissions are list-type flags. The = operator drops all previous values, whereas += adds another value.