Frequently asked questions about Remote Execution with Chromium (Goma).
What actions run remotely?¶
As of 2022-02-01 only C++ compilation actions run remotely, everything else is built locally.
Can I debug the binaries built remotely?¶
Yes. Goma automatically downloads all build output files from the remote execution cluster.
How can I download all build outputs? How can I download debug symbols?¶
Goma does this automatically: it downloads all build output files from the remote execution cluster.
What are typical build times?¶
As of 2022-02-01, clean build of Chromium for Linux x86-64 on a typical trial cluster is about 1 hour; fully cached build is about 15 minutes.
Can I use a custom CC wrapper?¶
If two developers have slightly different
gn args, will they be able to share cache hits?¶
Maybe. As long as the differences aren't substantial (e.g. different
target_cpu settings), these developers can expect cross-user cache
GN args that modify the Clang command line (e.g. optimization level, debug symbol output) or modify generated header files, will prevent cross-user caching.
If two developers clone the source tree on different paths, will they be able to share cache hits?¶
Maybe. As long as everything below their source tree looks the same
(including the output directory they ran
gn args for), and the action
is relocatable, these developers can expect cross-user cache hits.
Relocatable actions are independent of the current working directory and they don't depend on the absolute paths of input files. For example C++ compilation actions are usually relocatable, users can expect to share cache hits for them.
The Goma client crashed, how to fix it? Is this normal?¶
goma_ctl restart to fix.
This should be rare. It could happen if the
compiler_proxy crashes and
closes the IPC channels, crashing the
The build was hanging for a minute, then continued. Is this normal?¶
If you see slow builds, always look at the Active Tasks on the Goma dashboard (http://localhost:8088) to learn more.
Maybe the remote execution cluster was under heavy load, or maybe the
build was running non-remoteable slow actions locally (e.g.,
What do the Task Stats mean on the Goma dashboard?¶
In http://localhost:8088/#task-stats you may see:
Goma runs some of the remotable actions both locally and remotely. The actions race with each other; one finishes first ("wins"), the other is cancelled.
Goma does this to ensure you get the fastest compilation possible. Most actions are so fast that it's quicker to execute them locally than wait for the network overhead of remote execution. (The power of remote execution comes from parallellism and caching.) This is the reason for the "race", and explains the "both run" and "win" lines.
You may wonder why local actions win (almost) all races against remote. The likely culprit is that the remote cache is still cold: perhaps you checked out a new branch and need to recompile base libraries that everything else depends on.
Subsequent builds should be faster and you should see higher "goma win" rates as you get more cache hits.
How do I build inside a Docker container?¶
Goma can work within a container, but some additional configuration is needed.
- In the host environment, set
GOMA_SERVER_HOSTas you would normally and run
goma_auth login. This produces the file
Note: Avoid doing this inside the container
because it requires forwarding an unknown port after starting the
container, but you'll need this file later on.
* Build a container image. Make sure to run as a non-root user, ideally with the
uid as your host user so you can share files easily. The Goma compiler
proxy will not attempt remote actions if you run it as root. If you want to
view the compiler proxy's dashboard, expose the container's port 8088.
depot_toolsdirectory, and the
.goma_client_oauth2_configfile. Make sure
.goma_client_oauth2_configis owned by the container user's
uid. If you want to view the compiler proxy's dashboard, forward port
goma_ctl start, then run your build. If you want to view the compiler proxy's dashboard, set
GOMA_COMPILER_PROXY_LISTEN_ADDRto the empty string. By default, the compiler proxy serves its dashboard on
127.0.0.1:8088, which is not forwarded to the host environment. When
GOMA_COMPILER_PROXY_LISTEN_ADDRis set to the empty string, the compiler proxy binds to all addresses.