Skip to content

Cache & Carry

EngFlow in a Box — Full Documentation

Everything that is not in the quick start. Nothing here is needed to get a build running: that takes one cable and three steps, and they are all in that file. Come here when you want one particular thing.

Each section stands alone, so read the one you need and ignore the rest. If something is not working, start with Troubleshooting in the quick start.


Giving it internet

The box needs a network connection to fetch its license — see It needs internet each time it starts. Any one of these is enough. Your builds always go over the USB-C cable regardless of which you pick.

If you were given this box at a conference, check here first — it may already be done. Boxes handed out at an event are usually set up with the venue's own Wi-Fi, so they connect by themselves and there is nothing for you to do while you are there. To see whether yours is one of them plug in your USB-C cable and run:

Bash
ssh myefbox@10.55.0.1                       # password: admin
nmcli connection show | grep -i wifi        # a "myEFbox-conference" line means yes

Two things follow from that. It only works at the venue — once you are home, that network is out of range and you need one of the options below. And on those boxes the Wi-Fi radio ships on rather than off, so you can skip the nmcli radio wifi on step. If you would rather it not reconnect to the venue network at all, remove it:

Bash
sudo nmcli connection delete myEFbox-conference

Ethernet — the easiest

Plug a network cable into the Pi's Ethernet port. There is nothing to configure and no password to type; it picks up an address by DHCP and the box starts serving within about five minutes. This is the option to prefer — it leaves no stored credential on the box, and it also lets other machines reach it (see below).

Wi-Fi

The Wi-Fi radio is switched off when the box ships — it does not reach for a network on your desk unless you ask it to — so turn it on first. (On a conference box it is already on, and nmcli radio wifi on simply does nothing.)

Bash
1
2
3
4
5
ssh myefbox@10.55.0.1                    # password: admin
sudo nmcli radio wifi on                 # off by default; without this the next
                                         # command says "No network with SSID ... found"
sudo nmcli device wifi connect "<your-ssid>" password "<your-password>"
# sudo asks for the same password: admin

Check it worked, and see the address it was given:

Bash
nmcli -t -f NAME,DEVICE connection show --active
hostname -I

Two things to watch for:

  • Copy your network name exactly, including any leading or trailing space. Those are legal in a network name and some phone hotspots have one ("Pixel 7 "). A name trimmed by hand is a different network and will not associate.
  • The passphrase is stored on the box (root-only, not world-readable) so it reconnects by itself after a power-cycle. If that is not what you want, use Ethernet instead, or remove it later with sudo nmcli connection delete <name>.

Only 2.4 GHz and 5 GHz home/office networks work. A network that needs a browser sign-in page (hotel or guest Wi-Fi, "captive portal") will associate but the box will not get its license, because it cannot click through the page. Use Ethernet or a phone hotspot in that situation.

A phone hotspot

Works exactly like Wi-Fi above, and is a good fallback on a locked-down corporate network. The license check is the only traffic that leaves the box, so it uses a negligible amount of data.

If you cannot give it a network at all

Ask your EngFlow contact. There is an operator tool that shares a Mac's own internet connection with the box over the USB-C cable, so no network setup is needed on the box — it is not part of this quick start because it needs administrator rights on your laptop and some corporate laptops block it.

Using it from another machine on your network

The USB-C cable serves exactly one machine — the one it is plugged into. Once the box is on Ethernet or Wi-Fi, any machine on the same network can use it too, and the box will happily serve several at once (they share one cache, which is usually what a team wants).

First find the box's network address, from the machine with the cable:

Bash
ssh myefbox@10.55.0.1 hostname -I        # password: admin

That prints the box's addresses. 10.55.0.1 is the USB cable; the other one (something like 192.168.1.42) is its address on your network. Then on the other machine, use that address in place of 10.55.0.1:

Text Only
1
2
3
4
build:efcache --remote_cache=grpc://192.168.1.42:8080
build:efcache --bes_backend=grpc://192.168.1.42:8080
build:efcache --bes_results_url=http://192.168.1.42:8088/invocation/
build:efcache --remote_timeout=600

Check it from that machine before building:

Bash
curl -s -o /dev/null -w '%{http_code}\n' --http2-prior-knowledge \
    http://192.168.1.42:8080/healthz          # 200 = reachable

Three things worth knowing:

  • Expect it to be slower than the cable. On the reference project the same build took 3.8 s over USB-C and 52 s over this box's Wi-Fi. Over wired Ethernet it should land between the two. The machine with the cable should keep using 10.55.0.1.
  • The address can change. Most routers hand out addresses by DHCP and may give the box a different one after a reboot. If builds suddenly cannot reach the cache, re-check hostname -I. To make it stable, either reserve an address for the box in your router's settings (usually called "DHCP reservation" or "static lease"), or use the .local name described in the next section.
  • This works because the cache and UI listen on all interfaces, deliberately, so that a team can share one box. That is also why the box belongs on a network you trust — see Security and privacy.

A friendlier name than an IP address

If 10.55.0.1 in your ~/.bazelrc bothers you, or you want one file that works on several machines, give the box a name.

The name it already has, no setup needed

The box announces itself on the local network as <its-name>.local — typically myEFbox-001.local, where the number is your unit's. If you are not sure of the name, it is in myEFbox-info.txt on the SD card, and ssh myefbox@10.55.0.1 hostname prints it. On macOS and most Linux desktops this works immediately, with nothing to configure:

Bash
ping -c2 myEFbox-001.local
open http://myEFbox-001.local:8088

This works over the USB-C cable, and over Ethernet or Wi-Fi once the box is on your network. Because the name follows the box rather than its address, it keeps working when your router hands the box a different one — which makes it the better choice for a ~/.bazelrc you use on more than one machine:

Text Only
1
2
3
build:efcache --remote_cache=grpc://myEFbox-001.local:8080
build:efcache --bes_backend=grpc://myEFbox-001.local:8080
build:efcache --bes_results_url=http://myEFbox-001.local:8088/invocation/

If the name does not resolve, your machine or network does not do mDNS (common on Windows, and some corporate networks block it). Use the /etc/hosts method below, which always works.

Your own name via /etc/hosts

To use a name of your choosing — say my.engflow.mini — add one line to your own machine's hosts file. This is a change on your machine, not on the box, and it affects only that machine.

On macOS and Linux the file is /etc/hosts:

Bash
sudo sh -c 'echo "10.55.0.1  my.engflow.mini" >> /etc/hosts'

On Windows it is C:\Windows\System32\drivers\etc\hosts, edited as Administrator, with the same content.

Use the box's network address instead of 10.55.0.1 on a machine that reaches it over the network rather than the cable. Then:

Text Only
1
2
3
4
build:efcache --remote_cache=grpc://my.engflow.mini:8080
build:efcache --bes_backend=grpc://my.engflow.mini:8080
build:efcache --bes_results_url=http://my.engflow.mini:8088/invocation/
build:efcache --remote_timeout=600

and the UI is at http://my.engflow.mini:8088.

Notes:

  • Keep the port numbers. A hosts file maps a name to an address and cannot change ports, so it is my.engflow.mini:8088, not my.engflow.mini. Ports 8080 and 8088 stay as they are.
  • No configuration on the box is required. Both ports answer to whatever name you point at them — we tested my.engflow.mini against both and got 200.
  • Pick a name with a dot in it, as above. A single-word name can collide with your network's own search domain.
  • The link Bazel prints comes from your --bes_results_url, so it will use your new name too. One place still shows the raw address: links generated by the box itself inside the UI use 10.55.0.1:8088. They work fine from the cabled machine; from another machine, navigate from the UI's home page instead.
  • To undo it, delete the line you added from the same file.

Renewing the license

Normally there is nothing to do. The box fetches a license every time it starts, and re-checks every five minutes while running, so a box that has a network connection keeps itself licensed with no action from you.

You would notice a lapse as the UI showing an error page and Bazel reporting the cache as unreachable. In order, try:

Bash
1
2
3
4
5
6
7
8
9
# 1. Confirm the box can reach the internet at all.
ssh myefbox@10.55.0.1                    # password: admin
getent hosts my.engflow.com              # prints addresses = network is fine

# 2. Make it re-fetch immediately instead of waiting for the 5-minute check.
sudo systemctl restart myengflow-mini

# 3. Confirm it is serving again (give it ~40 seconds first).
curl -s --http2-prior-knowledge http://127.0.0.1:8080/healthz    # HEALTHY

Step 1 is the one that usually explains it: a lapsed license is nearly always a box that lost its network, not a licensing problem. If getent prints nothing, fix the network — see Giving it internet — and the box recovers on its own within five minutes.

Restarting never costs you your cache. Everything already stored stays stored.

These are demonstration licenses and they are time-limited — measured in hours, not months. The box renewing itself is what makes that invisible in normal use. If /healthz says HEALTHY you are licensed and there is nothing to renew.

If the box has a working network connection and still will not serve after a restart, that is the point to contact your EngFlow contact rather than to keep retrying — include the output of:

Bash
ssh myefbox@10.55.0.1 'docker logs --tail 50 engflow-cache'

which contains the licensing messages and the validity window.

Using your own cluster UUID

Your box came with a cluster UUID already in it — that is what it presents to EngFlow to get its license, and it is why the box worked without you signing up for anything. You might want to replace it with one of your own:

  • you started your own trial or have your own EngFlow account, and want this box's builds to appear under it;
  • the UUID it shipped with has expired and you would rather manage renewals yourself than come back to us;
  • you are moving the box to a different team, project, or account.

None of this is required. If the UI works and your builds are fast, there is nothing here you need.

Get a UUID

  1. Go to https://my.engflow.com and sign in (or create an account).
  2. Start a Local Trial.
  3. Copy the Cluster UUID — it looks like 7b2c1f04-9a3e-4c81-b6d5-2e8f0a1b3c4d.

Put it on the box

One line to change, then a restart:

Bash
ssh myefbox@10.55.0.1                    # password: admin
cd ~/myengflow

# See what it is now.
grep CLUSTER_UUID compose.yaml

# Replace it. Paste your own UUID between the quotes.
NEW=00000000-0000-0000-0000-000000000000
sed -i.bak "s/^\( *- CLUSTER_UUID=\).*/\1$NEW/" compose.yaml

# Confirm the change looks right BEFORE restarting.
grep CLUSTER_UUID compose.yaml

# Pick up the new UUID.
sudo systemctl restart myengflow-mini

Give it about 40 seconds, then check it worked:

Bash
curl -s --http2-prior-knowledge http://127.0.0.1:8080/healthz    # HEALTHY

HEALTHY means the new UUID was accepted and the box is licensed and serving. sed -i.bak left a copy of the previous file as compose.yaml.bak, so you can put the old UUID back by hand if you need to.

If it does not come back

The box will not serve on a UUID that EngFlow rejects, so this is the one place where a typo shows up as a dead box rather than a warning. The logs say which:

Bash
docker logs --tail 50 engflow-cache | grep -i -e license -e uuid
  • Invalid cluster UUID, or a licensing failure naming your UUID — the UUID is wrong or its trial is not active. Check it against my.engflow.com, character for character; it is easy to lose a character when copying by hand.
  • Locating a license failed, with no mention of the UUID — the box cannot reach EngFlow at all. That is a network problem, not a UUID problem; see Giving it internet.
  • To go back to how it shipped:
    Bash
    cd ~/myengflow && cp compose.yaml.bak compose.yaml
    sudo systemctl restart myengflow-mini
    

Two things worth knowing:

  • Your cache survives the change. The UUID controls licensing, not storage. Everything already cached stays cached, and your build history stays in the box's database.
  • A UUID is not a secret in the sense a password is, but it identifies your account to EngFlow. Treat it the way you would an account ID: no need to encrypt it, no reason to publish it.

Renaming the box and changing the password

Both are optional. The box works as shipped, and neither change affects your builds or your cache.

The box ships with a known password, admin, which is the same for every unit — so anyone who can reach it over the network and knows that can log in. Changing it takes one command:

Bash
ssh myefbox@10.55.0.1                    # password: admin
passwd                                   # asks for 'admin', then your new password twice

That is the whole job: the same password is used for logging in and for sudo, so both change together. Nothing else needs updating — the cache, the UI and Bazel never use it.

Write it down somewhere. There is no recovery if you forget it, short of re-flashing the card, which loses the cache.

Rename the box

The name myEFbox-001 appears in your shell prompt, in the box's logs, and as the .local name it advertises. To change it:

Bash
1
2
3
4
ssh myefbox@10.55.0.1                    # password: admin
sudo hostnamectl set-hostname mycache
sudo sed -i 's/^127\.0\.1\.1.*/127.0.1.1\tmycache/' /etc/hosts
sudo reboot

The second command matters: if the new name is not in /etc/hosts, every later sudo prints unable to resolve host before doing what you asked. It still works, but the warning is alarming and permanent.

After the reboot the box answers to mycache.local and the old myEFbox-001.local stops resolving, so update any .bazelrc that used it. Numeric addresses (10.55.0.1 and its network address) are unaffected.

The login name stays myefbox. That is the user account, which is separate from the box's name — you still connect with ssh myefbox@mycache.local. Renaming the account is possible but has to be done from a different login and breaks the paths the cache is configured with, so we do not recommend it.

If the cache fills up

Nothing breaks. The box holds a fixed amount of cache data, and once it's full it keeps serving everything it has but stops accepting anything new. Your builds still complete correctly — they just stop getting faster. Nothing is corrupted and nothing is lost.

Most people will never see this. Rebuilding the same project adds almost nothing to the cache: in our testing, four rebuilds in a row added zero new data because every result was already stored. You only add to the cache when you change code.

How to tell

The clearest sign is in your own build output. Bazel prints a warning but still reports success:

Text Only
1
2
3
WARNING: Remote Cache: io.grpc.StatusRuntimeException:
    RESOURCE_EXHAUSTED: Not enough space to create a replica
INFO: Build completed successfully, 1824 total actions

Note the build succeeded. If you see Build completed successfully, you have working results — this is not a failed build.

The other sign is builds quietly losing their speed advantage: something that took 4 seconds starts taking closer to its no-cache time. Combined with the warning above, that's a full cache.

To check the box directly:

Bash
ssh myefbox@10.55.0.1                    # password: admin

# How many times the box has refused new cache data.
# 0 means the cache is not full. Any number above 0 means it is.
# (The -a matters: the log can contain binary bytes, and without it grep may
#  print nothing at all instead of a count.)
grep -ac "Not enough space" ~/.cache/myengflow_mini/engflow.log.0

# How much room the card has, and how much the cache is using
df -h /
du -sh ~/.cache/myengflow_mini/cas

Neither command needs sudo.

Don't rely on the health check for this. http://10.55.0.1:8080/healthz still returns 200 on a full cache — correctly, because the box is healthy. A full cache is not a broken box, so the health check doesn't flag it.

The fix: restart it

One command, and it keeps everything already cached:

Bash
ssh myefbox@10.55.0.1                    # password: admin
cd ~/myengflow && docker compose restart

Give it about 40 seconds to come back, then build as usual. We tested this on a genuinely full cache: writes resumed at full speed and you do not lose anything already cached.

If you'd rather use the service manager, this does the same thing and asks for the admin password:

Bash
sudo systemctl restart myengflow-mini

If it fills up again quickly

Restarting reclaims the box's write budget without deleting anything, so if you're doing genuinely new work each build you may hit the limit again. To start from an empty cache instead:

Bash
1
2
3
4
ssh myefbox@10.55.0.1
sudo systemctl stop myengflow-mini
sudo rm -rf ~/.cache/myengflow_mini/cas
sudo systemctl start myengflow-mini

That throws away all cached results — your next build will be a normal cold build, and everything after it gets fast again. We measured it: the build right after a wipe took 97 s, the one after that 5.6 s with 1157 cache hits. The box repopulates itself.

One thing not to do: don't delete individual cache files by hand (for example with find ~/.cache/myengflow_mini/cas -delete). The box tracks what it holds in a database, so that frees disk space but does not let it accept new data, and leaves its records pointing at files that are gone. Restart instead, or remove the whole cas directory as shown above.


Powering it separately (optional)

You don't need this — one cable to your laptop is the intended setup and what we test. But the Pi 4 has a single USB-C port, so that port cannot take a power adapter and your laptop at the same time. If you want the box powered independently, use one of these:

  • A powered USB-C hub or dock between laptop and box. The hub supplies the power, and data still reaches your laptop over the same link. Confirm the hub passes the connection through — some don't (see Troubleshooting).
  • The Ethernet port for a box you want to leave running on a desk. Then the official 5 V / 3 A USB-C supply goes in the USB-C port and your builds go over the network instead of the cable. Note this is the slower path: over the network the same build took 52 s against 3.8 s on the cable.

Do it if your laptop port limits current, or to keep the box up while it sleeps.

Switching it off

When you're done with the box, shut it down before unplugging it:

Text Only
ssh myefbox@10.55.0.1 'systemctl poweroff'

Wait a few seconds for the green LED to stop flickering, then pull the cable.

Pulling the cable while it's running is usually harmless, but not always — if the box is writing to its card it can corrupt data, and the fix is a reflash. Nothing is lost by shutting down first.

It needs internet again after each power-up — see Give it internet once.


Building your own box from scratch

This section is for someone who does not have one of these boxes. If you are holding one, the rest of the guide already applies and you can ignore this. Pass this section to a colleague who saw yours and wants their own — it needs nothing from your box, no files from us, and no EngFlow contact. Everything below is either a public download or something you type.

It takes about an hour, most of it waiting on downloads.

What you need to buy

Raspberry Pi 4 Model B, 4 GB or 8 GB 8 GB is nicer but 4 GB is what we benchmarked and it is plenty. A Pi 5 also works for everything except the single-cable trick — see the note below
MicroSD card, 32 GB minimum 64 GB recommended. A U3 / A2-rated card is worth the extra couple of dollars; the cache is I/O-heavy
A USB-C data cable To your laptop. Not a charge-only cable — that is the single most common thing that goes wrong
A card reader If your computer has no SD slot

Optional: a passive heatsink is a cheap yes. A fan is not worth it — we measured it. Serving a cache is I/O-bound, so the board peaked at 67.6 °C during our benchmarks, 12 °C below where it would start throttling. (It does throttle hard under sustained all-core compute, but that is not this workload.)

You do not need a power supply. One USB-C cable from your laptop powers the board and carries the build traffic at the same time.

1. Flash Raspberry Pi OS

  1. Install Raspberry Pi Imager from https://www.raspberrypi.com/software/.
  2. Choose Raspberry Pi OS Lite (64-bit). Lite means no desktop, which is what you want — this box has no screen. 64-bit is required; the container images are ARM64-only.
  3. Open the imager's settings (the gear, or "Edit Settings" when it asks) and set:
  4. Hostname: myengflow — you will use myengflow.local to reach it
  5. Enable SSH, with password authentication
  6. Username and password — pick your own; this guide writes pi
  7. Wi-Fi, with your network and your country code. Do configure it here: it is much easier than doing it later on a machine with no screen
  8. Write the card.

2. Make the USB-C cable work as a network connection

Do this now, on the card, before the first boot — it is two edits on the card and much more awkward afterwards. This is what turns one cable into a working network link, and it is the step the official guide does not cover.

Put the card back in your computer. It mounts a small partition called bootfs. In it:

Add two lines to the end of config.txt:

Text Only
[all]
dtoverlay=dwc2,dr_mode=peripheral

Add one thing to cmdline.txt. That file is a single long line — do not add a newline anywhere in it. Append this to the end of the existing line, separated by a space:

Text Only
modules-load=dwc2,g_ether

dwc2 is the USB controller in device mode and g_ether makes the Pi appear to your laptop as a USB network adapter. Eject the card.

3. First boot and Docker

Put the card in the Pi and power it up. Give it two minutes, then connect over your network — this first connection is over Wi-Fi or Ethernet, not the cable:

Bash
ssh pi@myengflow.local

Then:

Bash
# Update, and install the one package the browser UI needs (see step 6).
sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y --no-install-recommends nghttp2-proxy curl

# Docker, from Docker's own installer.
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Run docker without sudo. Requires logging out and back in.
sudo usermod -aG docker $USER
exit

Reconnect and confirm Docker works:

Bash
ssh pi@myengflow.local
docker run --rm hello-world

The cable gives you a network interface but no address on it. This unit file sets one at every boot:

Bash
sudo tee /etc/systemd/system/myengflow-usb0.service > /dev/null <<'EOF'
[Unit]
Description=USB-C gadget link (10.55.0.1/24 on usb0)
After=systemd-modules-load.service
[Service]
Type=oneshot
RemainAfterExit=yes
# usb0 can appear a moment after this unit starts, so wait for it.
ExecStart=/bin/sh -c 'for i in $(seq 30); do [ -e /sys/class/net/usb0 ] && break; sleep 1; done; \
    ip link set usb0 up; ip addr replace 10.55.0.1/24 dev usb0'
[Install]
WantedBy=multi-user.target
EOF

# Keep NetworkManager's hands off usb0, or the two fight over the interface.
sudo tee /etc/NetworkManager/conf.d/99-usb0-unmanaged.conf > /dev/null <<'EOF'
[keyfile]
unmanaged-devices=interface-name:usb0
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now myengflow-usb0.service
sudo systemctl restart NetworkManager

Now plug the USB-C cable from your laptop into the Pi's USB-C port (the one that normally takes power). From your laptop:

Bash
ping -c 3 10.55.0.1

If that answers, the hard part is done. If it does not, it is almost certainly a charge-only cable — see Troubleshooting, which applies here too.

Why a fixed address and not DHCP? Because 10.55.0.1 is then the same on every box and can be written into a .bazelrc that just works. Your laptop gives itself a 169.254.x.x link-local address on that interface, which is enough to reach the Pi. Our boxes also run dnsmasq on usb0 to hand the laptop a proper 10.55.0.x, which is tidier but not needed.

5. Get a cluster UUID

  1. Go to https://my.engflow.com and sign in or create an account.
  2. Start a Local Trial.
  3. Copy the Cluster UUID.

These trial licenses are short-lived — hours, not months. That is fine, because the box re-fetches its license every time it starts and every five minutes while running (step 7 sets that up). Restart the trial on my.engflow.com when it lapses.

6. The one thing you would otherwise get stuck on

Left alone, MyEngFlow Mini starts a bundled envoy proxy in front of itself. On a Raspberry Pi 4, envoy cannot start at all. Its memory allocator asks the kernel for addresses around 23 TB, and Raspberry Pi OS builds its 64-bit kernel with a 512 GB address space, so those addresses do not exist and envoy aborts:

Text Only
tcmalloc/system-alloc.cc:625] MmapAligned() failed - unable to allocate with tag
tcmalloc/arena.cc:58] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data

This is not a RAM problem — it happens with 3 GB free — and no environment variable fixes it, because the address is baked into the envoy binary. Do not spend an evening on it as we did.

Mini only needs two things from that process, so you can replace it with about 250 lines of C. Create envoy_shim.c:

Bash
mkdir -p ~/myengflow/bin && cd ~/myengflow/bin
nano envoy_shim.c

You need a program that does exactly two things:

  1. answers HTTP 200 on 127.0.0.1:9902/ready — Mini polls this and refuses to start without it;
  2. accepts connections on 0.0.0.0:8080 and relays them, byte for byte, to 127.0.0.1:8081, where Mini's own server is listening.

That is the whole contract. A straightforward accept() / fork() TCP relay does it. Two details are worth getting right, both of which bit us:

  • Build it statically — gcc -O2 -static -Wall -o envoy_shim envoy_shim.c — because the Mini container image is distroless and has no shared libraries for a dynamically-linked binary to find.
  • Make the relay non-blocking, with a separate buffer per direction. Our first version used blocking writes, and it was crawlingly slow: this traffic is HTTP/2, and a stalled write in one direction blocks the flow-control frames travelling the other way. Poll each direction for reading only when its buffer has room and for writing only when it has bytes pending, and propagate half-close with shutdown(SHUT_WR).

Ours relays at ~480–519 MB/s, far above any link it will ever carry, so a simple implementation is not the bottleneck. If you would rather not write this, ask your EngFlow contact for envoy_shim.c — it is the one piece of this that is ours rather than public.

Then the second half of the same problem. With envoy gone, nothing translates protocols any more: Mini's server speaks only h2c (HTTP/2 without TLS), and browsers refuse to speak h2c. So the cache works for Bazel but the UI will not load in a browser. nghttpx (installed in step 3) fixes it by terminating ordinary HTTP on port 8088 and forwarding as h2c:

Bash
sudo mkdir -p /etc/nghttpx
sudo tee /etc/nghttpx/nghttpx.conf > /dev/null <<'EOF'
frontend=0.0.0.0,8088;no-tls
backend=127.0.0.1,8080;;proto=h2;no-tls
workers=1
errorlog-syslog=yes
# The UI streams invocation data; don't cut long-lived responses short.
stream-read-timeout=0
stream-write-timeout=0
backend-read-timeout=300
backend-write-timeout=300
EOF
sudo systemctl enable nghttpx
sudo systemctl restart nghttpx      # restart, not just enable: installing the
                                    # package already started it on its own config

So: Bazel talks to 8080, browsers to 8088. That split is why this guide's .bazelrc has 8080 for the cache and 8088 for the results URL.

7. Start it

Bash
1
2
3
mkdir -p ~/myengflow && cd ~/myengflow
mkdir -p ~/.cache/myengflow_mini
nano compose.yaml

Paste this, and put your cluster UUID on the CLUSTER_UUID line. The /home/pi paths must match your actual username:

YAML
services:
  myengflow_mini:
    container_name: engflow-cache
    depends_on:
      db:
        condition: service_healthy
    environment:
      - DATA_DIR=/usr/share/myengflow_mini
      # The USB-C address, so the UI's own links work from the cabled laptop.
      - UI_URL=http://10.55.0.1:8088
      - CLUSTER_UUID=PUT_YOUR_CLUSTER_UUID_HERE
      - DB_JDBC=jdbc:postgresql://db:5432/engflow
      - TZ=UTC
      # 4 GB board. On an 8 GB board use -Xmx3072m -Xms1024m.
      - JAVA_TOOL_OPTIONS=-Xmx1536m -Xms512m -XX:+UseG1GC -XX:MaxGCPauseMillis=200
      # Replace envoy with the shim from step 6.
      - ENVOY_BINARY=/usr/local/bin/envoy_shim
    ports:
      - "8080:8080"
    image: ghcr.io/engflow/myengflow_mini:latest
    platform: linux/arm64
    volumes:
      - /home/pi/.cache/myengflow_mini:/usr/share/myengflow_mini
      - /home/pi/myengflow/bin/envoy_shim:/usr/local/bin/envoy_shim:ro
    restart: unless-stopped

  db:
    container_name: engflow-db
    image: postgres:16-alpine
    platform: linux/arm64
    restart: unless-stopped
    environment:
      POSTGRES_DB: engflow
      POSTGRES_USER: root
      POSTGRES_PASSWORD: password
      POSTGRES_HOST_AUTH_METHOD: trust
      TZ: UTC
    command:
      - postgres
      - -c
      - shared_buffers=128MB
      - -c
      - max_connections=20
      - -c
      - work_mem=4MB
      - -c
      - maintenance_work_mem=64MB
      - -c
      - effective_cache_size=512MB
      - -c
      - random_page_cost=1.1
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U root"]
      interval: 2s
      timeout: 5s
      retries: 10
    volumes:
      - db-data:/var/lib/postgresql/data

volumes:
  db-data:
    driver: local

Start it and watch:

Bash
1
2
3
docker compose pull            # a few minutes; the images are large
docker compose up -d
docker compose logs -f myengflow_mini

Wait for the EngFlow logo and a line naming the license validity window. Ctrl-C stops following the logs; the containers keep running. Then:

Bash
curl -s --http2-prior-knowledge http://127.0.0.1:8080/healthz

HEALTHY means you are done. Note the --http2-prior-knowledge flag: it is required. Port 8080 does not speak ordinary HTTP, so a plain curl reports nothing at all and exit code 000 on a perfectly healthy box. That flag cost us real debugging time.

Start it at boot, so the box works after a power cut with nobody logged in:

Bash
sudo tee /etc/systemd/system/myengflow-mini.service > /dev/null <<EOF
[Unit]
Description=MyEngFlow Mini Cache Server
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=$HOME/myengflow
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
User=$USER
Group=$USER
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable myengflow-mini

And add the license retry timer. Do not skip this one. Mini tries five times to get a license and then gives up permanently — the container stays up, serving errors, so nothing restarts it. If the box ever starts without a working network, it will sit there dead until someone restarts it by hand. We found this the slow way: connecting Wi-Fi hours later changed nothing.

Bash
sudo tee /usr/local/sbin/myengflow-license-retry.sh > /dev/null <<'EOF'
#!/bin/sh
# Restart the stack if it is unhealthy but the license server is now reachable.
# Does nothing at all on a healthy box, so it is safe to run every 5 minutes.
set -e
if curl -fsS -m 10 --http2-prior-knowledge -o /dev/null \
        http://127.0.0.1:8080/healthz 2>/dev/null; then
    exit 0    # healthy
fi
# Only act if EngFlow is actually reachable, so we never restart in a loop
# on a box that simply has no network.
if ! getent hosts my.engflow.com >/dev/null 2>&1; then
    exit 0
fi
logger -t myengflow-license-retry "unhealthy and my.engflow.com resolves; restarting"
systemctl restart myengflow-mini.service
EOF
sudo chmod 755 /usr/local/sbin/myengflow-license-retry.sh

sudo tee /etc/systemd/system/myengflow-license-retry.service > /dev/null <<'EOF'
[Unit]
Description=MyEngFlow Mini license self-recovery check
After=myengflow-mini.service
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/myengflow-license-retry.sh
EOF

sudo tee /etc/systemd/system/myengflow-license-retry.timer > /dev/null <<'EOF'
[Unit]
Description=Check periodically whether the cache can get its license
[Timer]
OnBootSec=3min
OnUnitActiveSec=5min
Persistent=true
[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now myengflow-license-retry.timer

With that timer, a box you plug into a network starts serving within about five minutes on its own. Measured on ours: 90 seconds, with nothing typed.

8. Point Bazel at it

Exactly as in step 2 of the quick start:

Text Only
1
2
3
4
build:efcache --remote_cache=grpc://10.55.0.1:8080
build:efcache --bes_backend=grpc://10.55.0.1:8080
build:efcache --bes_results_url=http://10.55.0.1:8088/invocation/
build:efcache --remote_timeout=600

Build twice with bazel build --config=efcache //.... The first build fills the cache; the second should be dramatically faster and report remote cache hit for most actions. The UI is at http://10.55.0.1:8088.

Everything else in this guide now applies to your box too — the UI, the .local name, what to do when the cache fills up, changing the password, the security notes. Substitute your own username for myefbox and your own hostname for myEFbox-001.

Things we got wrong, so you don't have to

Each of these cost us hours:

  • A charge-only USB-C cable. By far the most common failure. The board powers up and its LED lights, so it looks fine, but no network interface ever appears. Use a cable you have moved data with.
  • curl http://…:8080/healthz returning nothing. Not a broken box. Port 8080 is HTTP/2-only; you need --http2-prior-knowledge.
  • The UI not loading in a browser while Bazel works perfectly. That is the missing h2c translation — step 6's nghttpx.
  • Freeing space by deleting cache files. Do not. find …/cas -delete frees disk but the server tracks what it holds in a database, so it still refuses new writes and now has records pointing at files that are gone. We measured this: 300 MB deleted, filesystem 52% down to 36%, writes still refused. Restart instead — it reconciles the accounting and keeps every blob. See If the cache fills up.
  • Trying to give envoy the address space it wants. We built two 16 KB-page kernels to raise the address limit. Neither boots on a Pi 4 — no console, no network, dead. The shim is the answer; the kernel is not.
  • Sizing the Java heap by RAM alone. Leave most of the RAM to the page cache rather than the heap. A cache server is fast because blobs are served from RAM instead of read back off the SD card, and that is the page cache's job, not the JVM's.
  • Expecting the container memory limits to apply. Raspberry Pi OS ships with cgroup memory accounting off, so Docker silently discards memory: limits. JAVA_TOOL_OPTIONS -Xmx is the ceiling that actually takes effect. (We left the limits in the file above anyway; they start working if you ever enable accounting.)

What to expect from it

Measured on a Pi 4 (4 GB) with a MicroSD card, building a real 1804-action open-source Bazel project:

Build time
No cache 85 s
Cached, over the USB-C cable 3.8 s
Cached, over the Pi's Wi-Fi 52 s

Use the cable. It is about 13× faster than this board's Wi-Fi, which is the bottleneck for anything large. Wired Ethernet lands between the two.

An SSD over USB 3.0 instead of a MicroSD card is the usual advice, and it will help writes. But on our measurements the card was not the limit for cached builds — the USB 2.0 gadget link (~31–37 MB/s) was, and the CPU peaked at only ~54%. Start with a good card; add an SSD if you find yourself waiting on cold builds.

About the Pi 5

A Pi 5 works fine as a cache — it boots, provisions, licenses and serves, and we verified all of that on real hardware. But we could not get the single-USB-C-cable link working on it. On a Pi 5 connected to a MacBook Pro with the same cable and port that work on a Pi 4, the interface never appears and has carried zero packets. Raspberry Pi's own documentation says the Pi 5 supports device mode on that port, so this may be a cable or host issue rather than a hard limitation — but we could not resolve it, and it is why we build these boxes on Pi 4s.

If you use a Pi 5, plan on Ethernet or Wi-Fi and use the box's network address everywhere this guide says 10.55.0.1. Everything else applies unchanged. The Pi 5 also draws more power: check vcgencmd get_throttled returns 0x0 on a laptop-powered board, and if it does not, power it from a proper supply.

What is not in this guide

The boxes we hand out are made from a prebuilt card image that also does a few things this guide leaves out, because they matter for producing a hundred identical units and not for making one:

  • provisioning entirely offline, with Docker and both container images baked into the image, so no unit needs Wi-Fi while it is being made;
  • sizing the heap to the board automatically at every start, so one image works on 2 / 4 / 8 GB boards and corrects itself if the card is moved;
  • dnsmasq on the USB link, avahi for the .local name, UTC clock, and a per-unit hostname;
  • a verification tool that runs ~50 checks on a finished unit.

None of that changes how the box performs. If you want it, that tooling is what your EngFlow contact hands to whoever is building units in quantity.


Credits

The box these are given away in is a remix of https://makerworld.com/en/models/2311113, used under the Creative Commons Attribution 4.0 licence (https://creativecommons.org/licenses/by/4.0/).


© 2020–2026 EngFlow