Remote build setup

Steps to configure Chromium (Goma) to run builds against a cluster.

Following the steps below will:

  • Speed up your builds by running them against the remote execution cluster.

Starting assumptions

  1. You are working on a Linux workstation.
  2. You are following this guide on a fresh workstation.
  3. You can run hermetic Chromium builds locally. Refer to how to: set up Chromium (goma) local hermetic build.

Prepare your environment for remote-execution

Before remote-execution can work, your local build must be hermetic (as described above).

  1. Set some necessary environment variables

    • Set GOMA_SERVER_HOST to the host name of EngFlow’s Goma servers (provided during onboarding). Do not include a port or protocol. Example: foo.goma.engflow.com.
    • Set GOMA_SERVER_PORT if the server is running on a port other than 443, the default.
    • Set GOMACTL_USE_PROXY to false.
    • Set GOMA_PROVIDE_INFO to true.

    Make sure these variables are always set; for example by adding them to your .profile file (or equivalent).

  2. More optional environment variables.

    • GOMA_USE_SSL controls whether the connection is encrypted. Set to false if not.
    • GOMA_USE_LOCAL controls whether actions are run locally as well as remotely. If set to false, an action is only run locally if it cannot be run remotely, or if it fails to run remotely first.
    • GOMA_FALLBACK controls whether an action should run locally after it fails to run remotely. See How-to: only use remote execution for forcing remote execution.

    As of 2022-08-01, supported envvars are defined in https://chromium.googlesource.com/infra/goma/client/+/refs/heads/main/client/goma_flags.cc without their GOMA_ prefix, in the GOMA_DEFINE_ lines.

  3. Run gn args out/Default (the $EDITOR will open again).

  4. Append use_goma = true into the file; save, then close the $EDITOR.

This has worked when

  1. Your environment has the variables

    env | grep GOMA
    

    shows the above environment variables.

  2. out/Default/args.gn includes use_goma = true.

Login to the cluster

  1. Run goma_auth login --no-browser.
  2. Accept the PII notice.
  3. Open the URL from the terminal in your browser.
  4. Log in with your Google (company, GSuite) account.
  5. Paste token from browser back into terminal and press Enter.

This has worked when

There is a Successfully logged in as foo@example.com.

Check your authentication at any time:

goma_auth info

… and you should see logged in as foo@example.com if you are authenticated.

Ensure the compiler-proxy is running with your changes loaded

Inside the command shell where your environment variables are set, run goma_ctl restart.

This has worked when

It prints that it succeeded and you can open http://localhost:8088 in your browser.

Troubleshooting

The Goma remoteexec proxy needs to be able to verify your OAuth2 credentials, even if you’re running everything on the same host. If you’re running the proxy locally for development, you need to ensure that it has access to a GCP service account or user account to do this. For example, to use your GCP user account:

  1. Run gcloud auth login
  2. Set GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/legacy_credentials/foo@example.com when running remoteexec_proxy.

If you see compiler binary hash mismatch errors, then the remoteexec proxy was configured with a set of toolchains that does not include those needed by your Chromium checkout. Google updates the Chromium toolchains frequently, so this happens after a very new checkout. See Chromium/Goma toolchains for update instructions.

Run a small build

Run autoninja -C out/Default base_unittests again and observe running/completed actions at http://localhost:8088.

This has worked when

autoninja -C out/Default base_unittests succeeds, and ./out/Default/base_unittests{.exe} is executable.

You can see results at http://localhost:8088. This is served by the Goma compiler proxy started with goma_ctl.

Now you can enjoy a fast, full remotely-executed build when running autoninja -C out/Default chrome.

Optimize your day-to-day workflow

  1. (one-time) Set the environment variables mentioned above into your user-profile shell-configuration files or otherwise (for example https://direnv.net if you want to source-control them).

    export GOMA_SERVER_HOST={cluster-name}.goma.engflow.com
    export GOMACTL_USE_PROXY=false
    export GOMA_PROVIDE_INFO=true
    
  2. (every time you want to build) ensure the local compiler-proxy is running and configured:

    goma_ctl restart
    

How-to: only use remote execution

By default, Goma runs each action both locally and remotely, taking the result from the action that finishes first. This can mask problems with remote execution, since the local actions will still succeed.

To force remote execution only, set the following environment variables:

GOMA_USE_LOCAL=false
GOMA_FALLBACK=false

See above for explanation. Actions that can only run locally (like linking) will still run locally.