ef command line tool

ef is a command line tool that serves as an interface for several EngFlow services. You can use it to run commands remotely (including entire builds), to download and upload files to content addressable storage (CAS), and to manage secrets.

This page documents how to install, configure, and run commands remotely using ef.

Install ef

To install ef, run the command below:

LinuxmacOSWindows

curl -fsSL https://docs.engflow.com/install-ef.sh | bash

This installs ef into /usr/local/bin/ and requires sudo access. You can change the install directory by running the script with --install-dir /path/to/install/dir.

curl -fsSL https://docs.engflow.com/install-ef.sh | bash

This installs ef into /usr/local/bin/ and requires sudo access. You can change the install directory by running the script with --install-dir /path/to/install/dir.

irm https://docs.engflow.com/install-ef.ps1 | iex

This installs ef into $home\AppData\Local\Programs\ef\. You may need to add this to your PATH. You can change the install directory by running the script with -InstallDir c:\path\to\install\dir.

Log in with ef

Before you can run commands remotely, you need to obtain a credential with the ef login command below, replacing the last argument with your cluster's hostname.

ef login example.cluster.engflow.com

This opens a browser window. Sign in with your SSO provider, if you haven't already. Once confirmed, ef obtains a credential. On most clusters, the credential is valid for 90 days.

By default, the credential is stored on your system's encrypted keyring. If no encrypted keyring is available, for example, in a remote environment that doesn't require a password to log in, you can use the -store=file flag to store the credential in a plaintext file instead.

Use ef as a credential helper

ef serves as a Bazel credential helper. This means Bazel can use the credential obtained with ef login, so you don't need to manage mTLS certificates or --remote_header flags. To use ef as a credential helper, add this flag to .bazelrc in your home directory, replacing the cluster name with your own:

.bazelrc

common:engflow --credential_helper=example.cluster.engflow.com=ef

More authentication commands

To remove a credential from the current machine:

ef logout example.cluster.engflow.com

To copy a credential from one machine to another, for example, when installing a credential into a non-interactive environment or installing the same credential onto multiple machines:

ef export example.cluster.engflow.com >credential-file
ef import <credential-file

To print the credential, for use with tools that don't directly support the Bazel credential helper protocol:

ef get <<<'{"uri":"https://example.cluster.engflow.com"}'

Configuration

ef accepts most flags either as command line arguments or as settings in an ef.toml configuration file. You'll want to create an ef.toml file so that you don't need to set frequently used flags like -server with every command.

The default file location depends on your operating system.

LinuxmacOSWindows

$HOME/.config/ef.toml

$HOME/Library/Application Support/ef.toml

$env:HOME\AppData\Roaming\ef.toml

You can use the -config_file=/path/to/file flag to use a different configuration file.

The configuration file is divided into sections. Each section is a separate configuration with its own list of settings. The settings use the same names as command line flags. You'll want a [default] configuration section, but you may add additional configurations if you work with multiple servers, platforms, or instances. You can use a different configuration with the -config=section-name flag. For example, if you have a configuration section named [foo], you can use this configuration with the -config=foo flag.

A minimal ef.toml file looks like this:

[default]
server = "grpcs://example.cluster.engflow.com"
git_bearer_token_name = "GITHUB_TOKEN"
exec_property = [
  "Pool=ef_run_linux_x64",
  "container-image=docker://docker.example.com/registry/image@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
]

Replace the server flag with your server URL.

git_bearer_token_name is needed to run commands in git mode when using a private repository. See Repository access for instructions.

The container-image property should typically point to the same image you use for CI. If you're using EngFlow CI Runners already, you can use the same image your Bento is configured with. See Create a container image for remote actions for requirements and instructions. Specifically, the requirements for ef run are:

• A user engflow with UID 108 and group engflow with GID 114 configured in /etc/passwd and /etc/group
• HOME set to a directory writable by that user.
• A complete shell environment with Bash, coreutils, and other tools (most Linux base images have this by default)
• git, for git mode
• bazel and any other tools you wish to invoke via ef run

Run commands remotely

You can run commands remotely with ef run. These may be small build steps or end-to-end builds.

ef run bazel build --config=engflow //...

git mode

By default, when ef run is invoked inside a Git repository or worktree, it runs with -mode=git. The alternative is the raw mode.

In git mode, ef run attempts to simulate the local environment by checking out the Git repository remotely. ef run syncs any uncommitted files and unpushed commits to the remote environment through the CAS, so there's no need to push changes in progress first.

ef run also syncs changes in the remote environment back to your local environment.

ef run syncs changes

# Write a local file
$ echo hello >file

# Read the file remotely
$ ef run cat file
hello

# Update the file remotely
$ ef run sh -c 'echo world >>file'

# Read the file locally
$ cat file
hello
world

Repository access

For git mode to work, your origin repository URL must be accessible from the remote environment. This typically requires an https:// URL and internet access on the remote execution pool.

If your Git repository is private, you can configure ef run to authenticate with your Git host using a bearer token, like a GitHub personal access token. The token only requires read access to repository contents, though you may wish to configure additional permissions depending on what commands you want to run.

1. Get a bearer token (personal access token) from your Git host.

2. Store the token securely:

   ef user-secret set -stdin GITHUB_TOKEN <<'EOF'
   your token here
   EOF
   

3. Configure ef to use the token in ef.toml:

   ef.toml

   [default]
   git_bearer_token_name = "GITHUB_TOKEN"
   

ef stores secrets on your system's encrypted keyring. If no encrypted keyring is available, you can add the -store=file flag to store the secret in a plaintext file instead. See User secrets for more.

Once configured, the GITHUB_TOKEN environment variable is set to the secret's value in the remote environment. ef run configures Git to use this token automatically. The token is not written to unencrypted storage including the CAS at any point.

Remote execution

In git mode, your remote build commands can make use of remote execution on the same cluster. The server automatically provisions a short-lived credential, made available in the environment variable ENGFLOW_RPC_CREDENTIALS. It also writes a .bazelrc file in the worker's home directory that configures a Bazel credential helper to use this.

If you have an engflow configuration already set up for remote execution from your local environment, you can use it with ef run.

.bazelrc

common:engflow --platforms=//platform/linux_x64
common:engflow --remote_executor=grpcs://example.cluster.engflow.com
common:engflow --bes_backend=grpcs://example.cluster.engflow.com
common:engflow --bes_results_url=https://example.cluster.engflow.com/invocations/default/

Assuming the above configuration exists, you can run:

ef run bazel build --config=engflow //...

raw mode

To run a command remotely without a Git repository, use -mode=raw or invoke ef run outside of a Git repository or worktree. In this mode, ef run simply executes a command without wrapping or special handling.

Flags and features

• -e KEY=VALUE or -e KEY: Set environment variables. Use -e KEY to copy the value from the local environment.
• -input=PATH or -input=SRC=DST: Make input files available in the remote environment. SRC is the local path to a file or directory, and DST is the remote path where it should be copied.
• -input-file-digest=DST=DIGEST or -input-dir-digest=DST=DIGEST: Use a file or directory already uploaded to the CAS if you know its SHA-256 digest and size. The digest is written as HASH/SIZE like 0395d52da42a286ec93550ff66b67a53ccedeb284b81ed685462637c477aad49/1234.
• -output=PATH or -output=SRC=DST: Download an output file from the remote environment. SRC is the remote path, and DST is the local path.
• -cache: Enable the action cache. When enabled, if the command completes successfully (exits 0), the result will be cached. If an identical command was cached earlier, its result will be served, and the command won't be executed again. This also allows concurrent executions of identical commands to be coalesced.

Cancellation

To cancel execution of a remote command, press ^C. ef run will call an API that requests cancellation.

To stop ef run without canceling the remote command, kill it with any signal other than SIGINT.

Manage secrets

ef can manage secrets for use with ef run. See Secrets for more details on how EngFlow manages secrets.

User secrets

User secrets are key-value pairs specific to one user. ef stores user secrets locally on the system's encrypted keyring (unless you opt to use unencrypted files with -store=file). ef passes user secrets to remote actions when requested, but never stores them unencrypted on EngFlow servers.

To create or update a secret:

ef user-secret set -stdin SECRET_NAME <<<'EOF'
your secret value
EOF

To print the value of a secret:

ef user-secret get SECRET_NAME

To list names of secrets:

ef user-secret list

To delete a secret:

ef user-secret delete SECRET_NAME

To use a secret as an environment variable in a remote action:

ef run -user_secret=SECRET_NAME ...

Secrets written to stdout or stderr are redacted to prevent accidental disclosure.

Instance secrets

Instance secrets are key-value pairs shared by all users of an EngFlow instance. These may be used for things like container registry keys or bot account credentials. ef stores instance secrets on EngFlow servers. Secret values are never written unencrypted to the CAS, but secret names may be stored and logged unencrypted.

To create or update a secret:

ef instance-secret set -stdin SECRET_NAME <<<'EOF'
your secret value
EOF

To print the value of a secret:

ef instance-secret get SECRET_NAME

To list names of secrets:

ef instance-secret list

To delete a secret:

ef instance-secret delete SECRET_NAME

To use a secret as an environment variable in a remote action:

ef run -instance_secret=SECRET_NAME ...

Secrets written to stdout or stderr are redacted to prevent accidental disclosure.

Content addressable storage (CAS)

You can use ef to read and write files and directories in the CAS.

To write a file or directory:

ef write PATH...

ef write prints the SHA-256 digest for each uploaded file.

To print a file to stdout:

ef read DIGEST

To copy a file to a local path:

ef read -o=LOCAL_PATH DIGEST

To copy a directory to a local path:

ef read -o=LOCAL_PATH -dir DIGEST