Skip to content

Remote Build Setup

Following the steps below will:

  • Speed up your builds by running them against the remote execution cluster.

Starting assumptions

  1. You are working on a Linux workstation.
  2. You are following this guide on a fresh workstation.
  3. You can run hermetic Chromium builds locally.

Prepare your environment for remote-execution

Before remote-execution can work, your local build must be hermetic (as described above).

  1. Set some necessary environment variables

    • Set GOMA_SERVER_HOST to the host name of EngFlow's Goma servers (provided during onboarding). Do not include a port or protocol. Example: foo.goma.engflow.com.
    • Set GOMA_SERVER_PORT if the server is running on a port other than 443, the default.
    • Set GOMA_PROVIDE_INFO to true.

    Make sure these variables are always set; for example by adding them to your .profile file (or equivalent).

  2. More optional environment variables.

    • GOMA_USE_SSL controls whether the connection is encrypted. Set to false if not.
    • GOMA_USE_LOCAL controls whether actions are run locally as well as remotely. If set to false, an action is only run locally if it cannot be run remotely, or if it fails to run remotely first.
    • GOMA_FALLBACK controls whether an action should run locally after it fails to run remotely. See How-to: only use remote execution for forcing remote execution.

    As of 2022-08-01, supported envvars are defined in https://chromium.googlesource.com/infra/goma/client/+/refs/heads/main/client/goma_flags.cc without their GOMA_ prefix, in the GOMA_DEFINE_ lines.

  3. Run gn args out/Default (the $EDITOR will open again).

  4. Append use_goma = true into the file; save, then close the $EDITOR.

This has worked when

  1. Your environment has the variables
Bash
1
env | grep GOMA

shows the above environment variables.

  1. out/Default/args.gn includes use_goma = true.

Login to the cluster

  1. Run goma_auth login.

    Note

    The previously supported --no-browser flag, which did an Out-of-Band auth flow, no longer works after 2022-10-03. If you're running goma_auth login over SSH, it will print instructions for port-forwarding.

  2. Accept the PII notice.

  3. Log in with your Google (company, GSuite) account.

This has worked when

There is a Successfully logged in as foo@example.com.

Check your authentication at any time:

Bash
1
goma_auth info

... and you should see logged in as foo@example.com if you are authenticated.

Ensure the compiler-proxy is running with your changes loaded

Inside the command shell where your environment variables are set, run goma_ctl restart.

This has worked when

It prints that it succeeded and you can open http://localhost:8088 in your browser.

Troubleshooting

Authentication

The Goma remoteexec proxy needs to be able to verify your OAuth2 credentials, even if you're running everything on the same host. If you're running the proxy locally for development, you need to ensure that it has access to a GCP service account or user account to do this. For example, to use your GCP user account:

  1. Run gcloud auth login
  2. Set GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/legacy_credentials/foo@example.com when running remoteexec_proxy.

compiler binary hash mismatch errors

If you see compiler binary hash mismatch errors, then the remoteexec proxy (the Goma server) was configured with a set of toolchains that does not include those needed by your Chromium checkout. This usually happens after Google releases a new set of toolchains and you've updated to a very new commit requiring those toolchains.

After a toolchain update, EngFlow builds new container images and updates the configuration of the Goma service. We start this process soon after toolchains are released, and it is ordinarily completed within a day. Please ask questions on Slack or alert our on-call engineer to expedite. Once the service is updated, you shouldn't need to take any other action.

Run a small build

Run autoninja -C out/Default base_unittests again and observe running/completed actions at http://localhost:8088.

This has worked when

autoninja -C out/Default base_unittests succeeds, and ./out/Default/base_unittests{.exe} is executable.

You can see results at http://localhost:8088. This is served by the Goma compiler proxy started with goma_ctl.

Now you can enjoy a fast, full remotely-executed build when running autoninja -C out/Default chrome.

Optimize your day-to-day workflow

  1. (one-time) Set the environment variables mentioned above into your user-profile shell-configuration files or otherwise (for example https://direnv.net if you want to source-control them).

    Bash
    1
2
    export GOMA_SERVER_HOST={cluster-name}.goma.engflow.com
export GOMA_PROVIDE_INFO=true

  2. (every time you want to build) ensure the local compiler-proxy is running and configured:

    Bash
    1
    goma_ctl restart

How-to: only use remote execution

By default, Goma runs each action both locally and remotely, taking the result from the action that finishes first. This can mask problems with remote execution, since the local actions will still succeed.

To force remote execution only, set the following environment variables:

Bash
1
2
GOMA_USE_LOCAL=false
GOMA_FALLBACK=false

See above for explanation. Actions that can only run locally (like linking) will still run locally.