FAQ

Why do I get PERMISSION_DENIED when trying to write to the cache?

The default service configuration allows remote execution but not remote caching. This is more secure.

A user with write access to the cache can write anything, including malicious code and binaries, which can then be returned to other users on cache lookups.

By comparison, remotely executed actions are typically sandboxed on a remote machine, which the user does not have direct control over. Since the cache key is a cryptographic hash of all input files, the command line, and the environment variables, it’s significantly harder to inject malicious data into the action cache.

In order to allow remote cache access, you need to adjust the permissions settings for the service depending on your authentication configuration (--client_auth ).

If you are using --client_auth=gcp_rbe , then you need to adjust permissions in the GCP IAM console.

Otherwise use --principal_based_permissions to configure per-user permissions.

If you disable client authentication (--client_auth=none ), you can add the following line to your configuration:

--principal_based_permissions=*->admin

When running on GCP, why can’t it pull my image from gcr.io?

While the GCP workers are authenticated with gcloud out of the box, images uploaded to gcr.io are not world-readable by default. You should check that the EngFlow RE role account has access to the image (or give it access if necessary).

See the Google Container Registry documentation for more details: https://cloud.google.com/container-registry/docs/access-control

The EngFlow RE role account is typically named: engflow-re-bot@<project>.iam.gserviceaccount.com

What if I get "clone": Operation not permitted from the sandbox?

Sandboxed execution uses clone(2), and may fail if the current user has insufficient privileges to use user namespaces.

If you run the Remote Execution service on Kubernetes or in Docker containers, or on a host where unprivileged user namespaces are disabled, sandboxed actions may fail with this error:

external/bazel/src/main/tools/linux-sandbox.cc:153: "clone": Operation not permitted

If you run the service in a container, you can try running it in privileged mode (--privileged).

You can also try enabling unprivileged user namespaces in the kernel (Debian):

sysctl -w kernel.unprivileged_userns_clone=1

As a last alternative, you can enable local execution on the server side (--allow_local ) and disable sandboxed execution on the client side (sandboxAllowed ).

Why do actions hang for 5 minutes then fail with RESOURCE_EXHAUSTED: Max queue time exhausted?

The rule was probably requesting an Executor Pool that didn’t exist. You can verify this theory if you override --max_queue_time_in_empty_pool to 30s for example, retry the build, and check if the same action fails exactly after that timeout.

How can I force a full rebuild to measure clean build performance?

Run bazel clean, then build again with --noremote_accept_cached.

Why do I get “403 Forbidden” errors from S3?

If you see the error on the client side:

/home/foo/.cache/bazel/_bazel_foo/84bdc474e377f556da900f3f344494fb/external/com_google_protobuf/BUILD:161:11: C++ compilation of rule '@com_google_protobuf//:protobuf_lite' failed (Exit 34): java.io.IOException: io.grpc.StatusRuntimeException: INTERNAL: Permission error while looking for 'blobs/ac/12d38991349d6297f807262fbf301ff178fdd178a4eed14c7d7df1fdbb955f89' in bucket '<BUCKET-NAME>' in region 'eu-central-1' (status 403). Details: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 29BB9C279A99CFFD; S3 Extended Request ID: <REDACTED>; Proxy: null), S3 Extended Request ID: <REDACTED>
--- cut here ---8<--------8<--------8<--------8<--- cut here ---
java.nio.file.AccessDeniedException: Permission error while looking for 'blobs/ac/12d38991349d6297f807262fbf301ff178fdd178a4eed14c7d7df1fdbb955f89' in bucket '<BUCKET-NAME>' in region 'eu-central-1' (status 403). Details: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 29BB9C279A99CFFD; S3 Extended Request ID: <REDACTED>; Proxy: null), S3 Extended Request ID: <REDACTED>
        at com.engflow.re.storage.aws.S3Client$RealApiWrapper.permissionError(S3Client.java:566)
        at com.engflow.re.storage.aws.S3Client$RealApiWrapper.download(S3Client.java:507)
        at com.engflow.re.storage.aws.S3Client.getDownloadStream(S3Client.java:257)
        at com.engflow.re.storage.ExternalStorage.lambda$getActionResult$5(ExternalStorage.java:492)
        <abbreviated>
--- cut here ---8<--------8<--------8<--------8<--- cut here ---

then you need to add s3 permissions to the IAM role policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:List*"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<BUCKET-NAME>",
                "arn:aws:s3:::<BUCKET-NAME>/*"
            ]
        }
    ]
}

What if my C++ compilation fails with missing dependency declarations errors?

This error:

ERROR: /home/foo/stuff/bazel/third_party/zlib/BUILD:25:19: undeclared inclusion(s) in rule '//third_party/zlib:zlib_checked_in': this rule is missing dependency declarations for the following files included by 'third_party/zlib/trees.c':
  '/usr/lib/gcc/x86_64-linux-gnu/8/include-fixed/limits.h'
  '/usr/lib/gcc/x86_64-linux-gnu/8/include-fixed/syslimits.h'

The culprit is that the selected C++ toolchain’s cc_toolchain_config.cxx_builtin_include_directories is missing that directory. Add /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed/ to that list.

What if my Java tests fail with SecurityException: Can't read cryptographic policy directory?

Bazel had a bug (https://github.com/bazelbuild/bazel/issues/9189) before release 4.0.0 that caused the security configuration files be excluded from the uploaded JDK when using --javabase=@remotejdk11_linux//:jdk, --javabase=@remotejdk15_macos//:jdk, or similar. There are several options to fix this:

• Upgrade to Bazel 4.0.0 or later.
• If you use Docker: use a Docker image with an installed JDK 11, and configure rbe_autoconfig to use that, either by setting JAVA_HOME in the Docker image or by setting java_home in your rbe_autoconfig configuration.
• Use your own Java JDK toolchain definition instead of the one included with Bazel.