Goma FAQ¶

Frequently asked questions about Remote Execution with Chromium (Goma).

What actions run remotely?¶

As of 2022-02-01 only C++ compilation actions run remotely, everything else is built locally.

Can I debug the binaries built remotely?¶

Yes. Goma automatically downloads all build output files from the remote execution cluster.

How can I download all build outputs? How can I download debug symbols?¶

Goma does this automatically: it downloads all build output files from the remote execution cluster.

What are typical build times?¶

As of 2022-02-01, clean build of Chromium for Linux x86-64 on a typical trial cluster is about 1 hour; fully cached build is about 15 minutes.

Can I use a custom CC wrapper?¶

No.

If two developers have slightly different `gn args`, will they be able to share cache hits?¶

Maybe. As long as the differences aren't substantial (e.g. different target_cpu settings), these developers can expect cross-user cache hits.

GN args that modify the Clang command line (e.g. optimization level, debug symbol output) or modify generated header files, will prevent cross-user caching.

Maybe. As long as everything below their source tree looks the same (including the output directory they ran gn args for), and the action is relocatable, these developers can expect cross-user cache hits.

Relocatable actions are independent of the current working directory and they don't depend on the absolute paths of input files. For example C++ compilation actions are usually relocatable, users can expect to share cache hits for them.

The Goma client crashed, how to fix it? Is this normal?¶

Run goma_ctl restart to fix.

This should be rare. It could happen if the compiler_proxy crashes and closes the IPC channels, crashing the gomacc processes.

The build was hanging for a minute, then continued. Is this normal?¶

If you see slow builds, always look at the Active Tasks on the Goma dashboard (http://localhost:8088) to learn more.

Maybe the remote execution cluster was under heavy load, or maybe the build was running non-remoteable slow actions locally (e.g., //third_party/devtools-frontend).

What do the Task Stats mean on the Goma dashboard?¶

In http://localhost:8088/#task-stats you may see:

Text Only
1 2 3 4	`Remote VS Local both run 3534 goma win 23 local win 3511`

Goma runs some of the remotable actions both locally and remotely. The actions race with each other; one finishes first ("wins"), the other is cancelled.

Goma does this to ensure you get the fastest compilation possible. Most actions are so fast that it's quicker to execute them locally than wait for the network overhead of remote execution. (The power of remote execution comes from parallellism and caching.) This is the reason for the "race", and explains the "both run" and "win" lines.

You may wonder why local actions win (almost) all races against remote. The likely culprit is that the remote cache is still cold: perhaps you checked out a new branch and need to recompile base libraries that everything else depends on.

Subsequent builds should be faster and you should see higher "goma win" rates as you get more cache hits.

How do I build inside a Docker container?¶

Goma can work within a container, but some additional configuration is needed.

In the host environment, set GOMA_SERVER_HOST as you would normally and run goma_auth login. This produces the file $HOME/.goma_client_oauth2_config.

Note: Avoid doing this inside the container because it requires forwarding an unknown port after starting the container, but you'll need this file later on. * Build a container image. Make sure to run as a non-root user, ideally with the same uid as your host user so you can share files easily. The Goma compiler proxy will not attempt remote actions if you run it as root. If you want to view the compiler proxy's dashboard, expose the container's port 8088.

Example Dockerfile:

Docker
FROM ubuntu:bionic

RUN apt-get update && \
    apt-get install -y curl git lsof python3

RUN useradd -u 1000 goma
USER goma

ENV PATH=$PATH:/depot_tools

EXPOSE 8088

Build with:

Bash
docker build -t goma-test - <Dockerfile.test

* Run the container. You'll need to bind-mount your source directory, the depot_tools directory, and the .goma_client_oauth2_config file. Make sure .goma_client_oauth2_config is owned by the container user's uid. If you want to view the compiler proxy's dashboard, forward port 8088.

Bash
docker run --name goma-test -it \
  --volume "${HOME}/Code/depot_tools:/depot_tools" \
  --volume "${HOME}/Code/goma:/goma" \
  --volume "${HOME}/.goma_client_oauth2_config:/home/goma/.goma_client_oauth2_config" \
  -p 127.0.0.1:8088:8088 \
  goma-test

* Within the container, set Goma environment variables, start the compiler proxy using goma_ctl start, then run your build. If you want to view the compiler proxy's dashboard, set GOMA_COMPILER_PROXY_LISTEN_ADDR to the empty string. By default, the compiler proxy serves its dashboard on 127.0.0.1:8088, which is not forwarded to the host environment. When GOMA_COMPILER_PROXY_LISTEN_ADDR is set to the empty string, the compiler proxy binds to all addresses.