Getting Started

This tutorial guides you through building our EngFlow/example repository to test out Remote Caching and Remote Execution on an EngFlow cluster. By the end of this exercise, you'll have the tools and know-how to build your own project using EngFlow.

Info

At this time, this walkthrough is only compatible with the Bazel build system. Future documentation will provide guidance for additional build systems that are compatible with the Remote Execution API, such as Buck2 or the Chromium build system (Goma).

Intro to EngFlow

Already signed up? Jump right in

It looks like you're ready to start building with EngFlow. After completing this exercise, there are a few other guides you may find useful for building your own project with EngFlow:

• Already using Bazel: Check out the First-Time Setup instructions for a detailed guide.
• Migrating to Bazel: EngFlow provides a collection of Migration resources and an Advanced Bazel Migration Toolkit that you're welcome to use!
• Using CMake: Follow our guide on CMake configuration.

Explore our free tools

Use MyEngFlow to run EngFlow locally in a Docker container. This lets you explore our Remote Caching and Build & Test UI, without Remote Execution.

Once MyEngFlow is running, continue following most of the exercise steps here, but with the following changes:

• Skip the Configure the project using the trial cluster section.
• Follow the instructions in the MyEngFlow user interface for defining your own .bazelrc file.
• Make sure to use bazel build --config=engflow wherever bazel build appears.
• Replace all instances of https://CLUSTER_NAME.cluster.engflow.com with http://127.0.0.1:8080.

Use the Inspect results in the UI section at the end of this page as a reference while inspecting your build and test results.

If you encounter any problems, we're here to help troubleshoot. Just click on the Get In Touch button in the upper right-hand corner of the MyEngFlow page.

Request a trial

Want to check out what it's like to build using a full EngFlow instance? Visit our Demo page and click the Request Trial button! While waiting to receive the trial details, follow the Prepare your environment steps below.

Once you receive the trial information, follow the rest of this guide. If you have any problems or questions, reach out to your Sales Engineer or Developer Support Engineer.

Prepare your environment

Install Bazel

The example project uses the Bazel build system. Please install Bazel on your machine if you haven't done so already.

Clone the example repository

The EngFlow/example repository contains examples in various languages that demonstrate how to build and test software with the EngFlow Remote Execution service:

git clone https://github.com/EngFlow/example.git

Review any language-specific guidance

The README file in the EngFlow/example repository provides guidance on how to build the code and run the tests for various languages. Note which languages contain available bazel build and bazel test targets.

Perform any special setup required for the language of your choice as described in the README file. Then run the local bazel build and bazel test commands to ensure they complete successfully, but skip the remote execution commands. Follow along with the rest of the steps on this page instead.

Build the example project using EngFlow

Configure the project using the trial cluster

Once you've received your trial access, proceed through the following steps, replacing CLUSTER_NAME with the name of your EngFlow trial cluster.

1. Log into the cluster UI at https://CLUSTER_NAME.cluster.engflow.com

2. If your cluster has mutual authentication enabled, download a Mutual Transport Layer Security (mTLS) certificate from:

   https://CLUSTER_NAME.cluster.engflow.com/gettingstarted
   

   This downloads a ZIP file that you need to unpack. We recommend unpacking it into a dedicated .engflow directory in your home directory, i.e., ~/.engflow.

   The getting started guide link on this cluster page is a link to the Bazel First-Time Setup documentation. We'll return to this documentation in the last step of this exercise.

   

3. If you're curious, you can view the current state of your trial cluster on the Cluster Status page:

4. Create a .bazelrc.user file in your EngFlow/example working directory. The .bazelrc from EngFlow/example imports this file to pick up your local configurations. Add lines like the following, replacing PATH_TO_CERTIFICATE_DIR with the absolute path into which you unpacked the certificate files (e.g., /Users/foobar/.engflow):

   # Incorporate configurations from .bazelrc, which sources this file.
   #
   # - Depending on your cluster configuration, replace remote_linux_64 with
   #   remote_macos_x64 or remote_windows_x64.
   build:engflow --config=engflow_common
   build:engflow --config=remote_linux_x64
   
   # Configuration for your trial cluster
   build:engflow --remote_executor=grpcs://CLUSTER_NAME.cluster.engflow.com/
   build:engflow --bes_backend=grpcs://CLUSTER_NAME.cluster.engflow.com/
   build:engflow --bes_results_url=https://CLUSTER_NAME.cluster.engflow.com/invocation/
   build:engflow --nogoogle_default_credentials
   build:engflow --bes_lifecycle_events
   
   # Configuration for your mTLS certificates (if required)
   build:engflow --tls_client_certificate=PATH_TO_CERTIFICATE_DIR/engflow.crt
   build:engflow --tls_client_key=PATH_TO_CERTIFICATE_DIR/engflow.key
   
   # This ensures _all_ build and test runs will be remote. Comment this out
   # when building locally.
   build --config=engflow
   

5. EngFlow uses Linux x64 by default (--config=remote_linux_x64). If you set up a trial for a different architecture, you will need to change this configuration parameter accordingly. See the .bazelrc file at the top level of the EngFlow/example repo for the correct configuration label for your architecture.

Build targets remotely

1. Pick one of the language-specific project directories from EngFlow/example that you'd like to build, and build all targets within that directory. For example, when choosing java:

   bazel build //java/...
   

2. You should see output similar to the following in the results (where INVOCATION_ID is a globally unique identifier containing hexadecimal digits and dashes):

   $ bazel build //java/...
   
   INFO: Invocation ID: INVOCATION_ID
   INFO: Streaming build results to: https://CLUSTER_NAME.cluster.engflow.com/invocation/INVOCATION_ID
   INFO: Analyzed 10 targets (5 packages loaded, 5608 targets configured).
   [ ...snip... ]
   INFO: Found 10 targets...
   INFO: Elapsed time: 159.513s, Critical Path: 87.38s
   INFO: 872 processes: 385 internal, 487 remote.
   INFO: Build completed successfully, 872 total actions
   INFO: Streaming build results to: https://CLUSTER_NAME.cluster.engflow.com/invocation/INVOCATION_ID
   

Inspect results in the UI

1. Follow the build results URL in the final line of the output above to view the invocation details page:

   https://CLUSTER_NAME.cluster.engflow.com/invocation/INVOCATION_ID
   

   

2. Explore this page to get a feel for how the EngFlow UI enables visibility into build results, details, and performance. Note the build summary at the top and the content within each of the tabs (Highlights, Target Tree, etc.).

3. Read the Invocation Details Page documentation for more details about the information presented on this page.

Execute tests remotely

1. Run all the test targets within the chosen directory and ensure that they all PASSED:

   $ bazel test //java/...
   
   INFO: Invocation ID: INVOCATION_ID
   INFO: Streaming build results to: https://CLUSTER_NAME.cluster.engflow.com/invocation/INVOCATION_ID
   INFO: Analyzed 10 targets (159 packages loaded, 5608 targets configured).
   [ ...snip... ]
   INFO: Found 9 targets and 1 test target...
   INFO: Elapsed time: 187.185s, Critical Path: 183.46s
   INFO: 501 processes: 487 remote cache hit, 13 internal, 1 remote.
   INFO: Build completed successfully, 501 total actions
   //java/com/engflow/example:ExampleTest                 PASSED in 0.9s
   
   Executed 1 out of 1 test: 1 test passes.
   INFO: Streaming build results to: https://CLUSTER_NAME.cluster.engflow.com/invocation/INVOCATION_ID
   

2. Run the tests one more time to ensure that Bazel uses the cached results, indicated by (cached) PASSED:

   $ bazel test //java/...
   
   INFO: Invocation ID: INVOCATION_ID
   INFO: Streaming build results to: https://CLUSTER_NAME.cluster.engflow.com/invocation/INVOCATION_ID
   INFO: Analyzed 10 targets (0 packages loaded, 0 targets configured).
   INFO: Found 9 targets and 1 test target...
   INFO: Elapsed time: 0.109s, Critical Path: 0.00s
   INFO: 1 process: 1 internal.
   INFO: Build completed successfully, 1 total action
   //java/com/engflow/example:ExampleTest        (cached) PASSED in 0.9s
   
   Executed 0 out of 1 test: 1 test passes.
   INFO: Streaming build results to: https://CLUSTER_NAME.cluster.engflow.com/invocation/INVOCATION_ID
   

3. Note the differences in the build summaries and test target details between the two invocations (Target Tree > //java/com/engflow > /example > :ExampleTest). The first invocation runs a bit longer, to fetch necessary infrastructure components, build the targets, and run the tests: Compare this to the second invocation, which shows the entire build and test took only a few seconds, thanks to caching. You can also see that the second invocation reports the exact same test results from the first invocation:

Forge your own project using EngFlow

Now that you've successfully built and tested the example project, you're ready to configure your own project to build using the EngFlow cluster! See the Already signed up? Jump right in section at the top of this page for further guidance.