Cache & Carry¶
EngFlow in a Box — Full Documentation¶
Everything that is not in the quick start. Nothing here is needed to get a build running: that takes one cable and three steps, and they are all in that file. Come here when you want one particular thing.
Each section stands alone, so read the one you need and ignore the rest. If something is not working, start with Troubleshooting in the quick start.
Giving it internet¶
The box needs a network connection to fetch its license — see It needs internet each time it starts. Any one of these is enough. Your builds always go over the USB-C cable regardless of which you pick.
If you were given this box at a conference, check here first — it may already be done. Boxes handed out at an event are usually set up with the venue's own Wi-Fi, so they connect by themselves and there is nothing for you to do while you are there. To see whether yours is one of them plug in your USB-C cable and run:
Bash Two things follow from that. It only works at the venue — once you are home, that network is out of range and you need one of the options below. And on those boxes the Wi-Fi radio ships on rather than off, so you can skip the
nmcli radio wifi onstep. If you would rather it not reconnect to the venue network at all, remove it:
Bash
Ethernet — the easiest¶
Plug a network cable into the Pi's Ethernet port. There is nothing to configure and no password to type; it picks up an address by DHCP and the box starts serving within about five minutes. This is the option to prefer — it leaves no stored credential on the box, and it also lets other machines reach it (see below).
Wi-Fi¶
The Wi-Fi radio is switched off when the box ships — it does not reach for a
network on your desk unless you ask it to — so turn it on first. (On a conference
box it is already on, and nmcli radio wifi on simply does nothing.)
| Bash | |
|---|---|
Check it worked, and see the address it was given:
Two things to watch for:
- Copy your network name exactly, including any leading or trailing space.
Those are legal in a network name and some phone hotspots have one (
"Pixel 7 "). A name trimmed by hand is a different network and will not associate. - The passphrase is stored on the box (root-only, not world-readable) so it
reconnects by itself after a power-cycle. If that is not what you want, use
Ethernet instead, or remove it later with
sudo nmcli connection delete <name>.
Only 2.4 GHz and 5 GHz home/office networks work. A network that needs a browser sign-in page (hotel or guest Wi-Fi, "captive portal") will associate but the box will not get its license, because it cannot click through the page. Use Ethernet or a phone hotspot in that situation.
A phone hotspot¶
Works exactly like Wi-Fi above, and is a good fallback on a locked-down corporate network. The license check is the only traffic that leaves the box, so it uses a negligible amount of data.
If you cannot give it a network at all¶
Ask your EngFlow contact. There is an operator tool that shares a Mac's own internet connection with the box over the USB-C cable, so no network setup is needed on the box — it is not part of this quick start because it needs administrator rights on your laptop and some corporate laptops block it.
Using it from another machine on your network¶
The USB-C cable serves exactly one machine — the one it is plugged into. Once the box is on Ethernet or Wi-Fi, any machine on the same network can use it too, and the box will happily serve several at once (they share one cache, which is usually what a team wants).
First find the box's network address, from the machine with the cable:
| Bash | |
|---|---|
That prints the box's addresses. 10.55.0.1 is the USB cable; the other one
(something like 192.168.1.42) is its address on your network. Then on the other
machine, use that address in place of 10.55.0.1:
| Text Only | |
|---|---|
Check it from that machine before building:
| Bash | |
|---|---|
Three things worth knowing:
- Expect it to be slower than the cable. On the reference project the same
build took 3.8 s over USB-C and 52 s over this box's Wi-Fi. Over wired
Ethernet it should land between the two. The machine with the cable should keep
using
10.55.0.1. - The address can change. Most routers hand out addresses by DHCP and may give
the box a different one after a reboot. If builds suddenly cannot reach the
cache, re-check
hostname -I. To make it stable, either reserve an address for the box in your router's settings (usually called "DHCP reservation" or "static lease"), or use the.localname described in the next section. - This works because the cache and UI listen on all interfaces, deliberately, so that a team can share one box. That is also why the box belongs on a network you trust — see Security and privacy.
A friendlier name than an IP address¶
If 10.55.0.1 in your ~/.bazelrc bothers you, or you want one file that works
on several machines, give the box a name.
The name it already has, no setup needed¶
The box announces itself on the local network as <its-name>.local — typically
myEFbox-001.local, where the number is your unit's. If you are not sure of the
name, it is in myEFbox-info.txt on the SD card, and ssh myefbox@10.55.0.1
hostname prints it. On macOS and most Linux desktops this works immediately, with
nothing to configure:
This works over the USB-C cable, and over Ethernet or Wi-Fi once the box is on your
network. Because the name follows the box rather than its address, it keeps working
when your router hands the box a different one — which makes it the better choice
for a ~/.bazelrc you use on more than one machine:
| Text Only | |
|---|---|
If the name does not resolve, your machine or network does not do mDNS (common on
Windows, and some corporate networks block it). Use the /etc/hosts method below,
which always works.
Your own name via /etc/hosts¶
To use a name of your choosing — say my.engflow.mini — add one line to your own
machine's hosts file. This is a change on your machine, not on the box, and it
affects only that machine.
On macOS and Linux the file is /etc/hosts:
| Bash | |
|---|---|
On Windows it is C:\Windows\System32\drivers\etc\hosts, edited as Administrator,
with the same content.
Use the box's network address instead of 10.55.0.1 on a machine that reaches
it over the network rather than the cable. Then:
| Text Only | |
|---|---|
and the UI is at http://my.engflow.mini:8088.
Notes:
- Keep the port numbers. A hosts file maps a name to an address and cannot
change ports, so it is
my.engflow.mini:8088, notmy.engflow.mini. Ports 8080 and 8088 stay as they are. - No configuration on the box is required. Both ports answer to whatever name
you point at them — we tested
my.engflow.miniagainst both and got200. - Pick a name with a dot in it, as above. A single-word name can collide with your network's own search domain.
- The link Bazel prints comes from your
--bes_results_url, so it will use your new name too. One place still shows the raw address: links generated by the box itself inside the UI use10.55.0.1:8088. They work fine from the cabled machine; from another machine, navigate from the UI's home page instead. - To undo it, delete the line you added from the same file.
Renewing the license¶
Normally there is nothing to do. The box fetches a license every time it starts, and re-checks every five minutes while running, so a box that has a network connection keeps itself licensed with no action from you.
You would notice a lapse as the UI showing an error page and Bazel reporting the cache as unreachable. In order, try:
Step 1 is the one that usually explains it: a lapsed license is nearly always a
box that lost its network, not a licensing problem. If getent prints nothing,
fix the network — see Giving it internet — and the box
recovers on its own within five minutes.
Restarting never costs you your cache. Everything already stored stays stored.
These are demonstration licenses and they are time-limited — measured in hours,
not months. The box renewing itself is what makes that invisible in normal use. If
/healthz says HEALTHY you are licensed and there is nothing to renew.
If the box has a working network connection and still will not serve after a restart, that is the point to contact your EngFlow contact rather than to keep retrying — include the output of:
| Bash | |
|---|---|
which contains the licensing messages and the validity window.
Using your own cluster UUID¶
Your box came with a cluster UUID already in it — that is what it presents to EngFlow to get its license, and it is why the box worked without you signing up for anything. You might want to replace it with one of your own:
- you started your own trial or have your own EngFlow account, and want this box's builds to appear under it;
- the UUID it shipped with has expired and you would rather manage renewals yourself than come back to us;
- you are moving the box to a different team, project, or account.
None of this is required. If the UI works and your builds are fast, there is nothing here you need.
Get a UUID¶
- Go to https://my.engflow.com and sign in (or create an account).
- Start a Local Trial.
- Copy the Cluster UUID — it looks like
7b2c1f04-9a3e-4c81-b6d5-2e8f0a1b3c4d.
Put it on the box¶
One line to change, then a restart:
Give it about 40 seconds, then check it worked:
| Bash | |
|---|---|
HEALTHY means the new UUID was accepted and the box is licensed and serving.
sed -i.bak left a copy of the previous file as compose.yaml.bak, so you can put
the old UUID back by hand if you need to.
If it does not come back¶
The box will not serve on a UUID that EngFlow rejects, so this is the one place where a typo shows up as a dead box rather than a warning. The logs say which:
| Bash | |
|---|---|
Invalid cluster UUID, or a licensing failure naming your UUID — the UUID is wrong or its trial is not active. Check it against my.engflow.com, character for character; it is easy to lose a character when copying by hand.Locating a license failed, with no mention of the UUID — the box cannot reach EngFlow at all. That is a network problem, not a UUID problem; see Giving it internet.- To go back to how it shipped:
Two things worth knowing:
- Your cache survives the change. The UUID controls licensing, not storage. Everything already cached stays cached, and your build history stays in the box's database.
- A UUID is not a secret in the sense a password is, but it identifies your account to EngFlow. Treat it the way you would an account ID: no need to encrypt it, no reason to publish it.
Renaming the box and changing the password¶
Both are optional. The box works as shipped, and neither change affects your builds or your cache.
Change the password (recommended if it will live on a shared network)¶
The box ships with a known password, admin, which is the same for every unit —
so anyone who can reach it over the network and knows that can log in. Changing it
takes one command:
| Bash | |
|---|---|
That is the whole job: the same password is used for logging in and for sudo, so
both change together. Nothing else needs updating — the cache, the UI and Bazel
never use it.
Write it down somewhere. There is no recovery if you forget it, short of re-flashing the card, which loses the cache.
Rename the box¶
The name myEFbox-001 appears in your shell prompt, in the box's logs, and as the
.local name it advertises. To change it:
| Bash | |
|---|---|
The second command matters: if the new name is not in /etc/hosts, every later
sudo prints unable to resolve host before doing what you asked. It still works,
but the warning is alarming and permanent.
After the reboot the box answers to mycache.local and the old
myEFbox-001.local stops resolving, so update any .bazelrc that used it. Numeric
addresses (10.55.0.1 and its network address) are unaffected.
The login name stays
myefbox. That is the user account, which is separate from the box's name — you still connect withssh myefbox@mycache.local. Renaming the account is possible but has to be done from a different login and breaks the paths the cache is configured with, so we do not recommend it.
If the cache fills up¶
Nothing breaks. The box holds a fixed amount of cache data, and once it's full it keeps serving everything it has but stops accepting anything new. Your builds still complete correctly — they just stop getting faster. Nothing is corrupted and nothing is lost.
Most people will never see this. Rebuilding the same project adds almost nothing to the cache: in our testing, four rebuilds in a row added zero new data because every result was already stored. You only add to the cache when you change code.
How to tell¶
The clearest sign is in your own build output. Bazel prints a warning but still reports success:
| Text Only | |
|---|---|
Note the build succeeded. If you see Build completed successfully, you have
working results — this is not a failed build.
The other sign is builds quietly losing their speed advantage: something that took 4 seconds starts taking closer to its no-cache time. Combined with the warning above, that's a full cache.
To check the box directly:
Neither command needs sudo.
Don't rely on the health check for this.
http://10.55.0.1:8080/healthzstill returns200on a full cache — correctly, because the box is healthy. A full cache is not a broken box, so the health check doesn't flag it.
The fix: restart it¶
One command, and it keeps everything already cached:
Give it about 40 seconds to come back, then build as usual. We tested this on a genuinely full cache: writes resumed at full speed and you do not lose anything already cached.
If you'd rather use the service manager, this does the same thing and asks for the
admin password:
| Bash | |
|---|---|
If it fills up again quickly¶
Restarting reclaims the box's write budget without deleting anything, so if you're doing genuinely new work each build you may hit the limit again. To start from an empty cache instead:
| Bash | |
|---|---|
That throws away all cached results — your next build will be a normal cold build, and everything after it gets fast again. We measured it: the build right after a wipe took 97 s, the one after that 5.6 s with 1157 cache hits. The box repopulates itself.
One thing not to do: don't delete individual cache files by hand (for example with
find ~/.cache/myengflow_mini/cas -delete). The box tracks what it holds in a database, so that frees disk space but does not let it accept new data, and leaves its records pointing at files that are gone. Restart instead, or remove the wholecasdirectory as shown above.
Powering it separately (optional)¶
You don't need this — one cable to your laptop is the intended setup and what we test. But the Pi 4 has a single USB-C port, so that port cannot take a power adapter and your laptop at the same time. If you want the box powered independently, use one of these:
- A powered USB-C hub or dock between laptop and box. The hub supplies the power, and data still reaches your laptop over the same link. Confirm the hub passes the connection through — some don't (see Troubleshooting).
- The Ethernet port for a box you want to leave running on a desk. Then the official 5 V / 3 A USB-C supply goes in the USB-C port and your builds go over the network instead of the cable. Note this is the slower path: over the network the same build took 52 s against 3.8 s on the cable.
Do it if your laptop port limits current, or to keep the box up while it sleeps.
Switching it off¶
When you're done with the box, shut it down before unplugging it:
| Text Only | |
|---|---|
Wait a few seconds for the green LED to stop flickering, then pull the cable.
Pulling the cable while it's running is usually harmless, but not always — if the box is writing to its card it can corrupt data, and the fix is a reflash. Nothing is lost by shutting down first.
It needs internet again after each power-up — see Give it internet once.
Building your own box from scratch¶
This section is for someone who does not have one of these boxes. If you are holding one, the rest of the guide already applies and you can ignore this. Pass this section to a colleague who saw yours and wants their own — it needs nothing from your box, no files from us, and no EngFlow contact. Everything below is either a public download or something you type.
It takes about an hour, most of it waiting on downloads.
What you need to buy¶
| Raspberry Pi 4 Model B, 4 GB or 8 GB | 8 GB is nicer but 4 GB is what we benchmarked and it is plenty. A Pi 5 also works for everything except the single-cable trick — see the note below |
| MicroSD card, 32 GB minimum | 64 GB recommended. A U3 / A2-rated card is worth the extra couple of dollars; the cache is I/O-heavy |
| A USB-C data cable | To your laptop. Not a charge-only cable — that is the single most common thing that goes wrong |
| A card reader | If your computer has no SD slot |
Optional: a passive heatsink is a cheap yes. A fan is not worth it — we measured it. Serving a cache is I/O-bound, so the board peaked at 67.6 °C during our benchmarks, 12 °C below where it would start throttling. (It does throttle hard under sustained all-core compute, but that is not this workload.)
You do not need a power supply. One USB-C cable from your laptop powers the board and carries the build traffic at the same time.
1. Flash Raspberry Pi OS¶
- Install Raspberry Pi Imager from https://www.raspberrypi.com/software/.
- Choose Raspberry Pi OS Lite (64-bit). Lite means no desktop, which is what you want — this box has no screen. 64-bit is required; the container images are ARM64-only.
- Open the imager's settings (the gear, or "Edit Settings" when it asks) and set:
- Hostname:
myengflow— you will usemyengflow.localto reach it - Enable SSH, with password authentication
- Username and password — pick your own; this guide writes
pi - Wi-Fi, with your network and your country code. Do configure it here: it is much easier than doing it later on a machine with no screen
- Write the card.
2. Make the USB-C cable work as a network connection¶
Do this now, on the card, before the first boot — it is two edits on the card and much more awkward afterwards. This is what turns one cable into a working network link, and it is the step the official guide does not cover.
Put the card back in your computer. It mounts a small partition called
bootfs. In it:
Add two lines to the end of config.txt:
Add one thing to cmdline.txt. That file is a single long line — do not
add a newline anywhere in it. Append this to the end of the existing line,
separated by a space:
| Text Only | |
|---|---|
dwc2 is the USB controller in device mode and g_ether makes the Pi appear to
your laptop as a USB network adapter. Eject the card.
3. First boot and Docker¶
Put the card in the Pi and power it up. Give it two minutes, then connect over your network — this first connection is over Wi-Fi or Ethernet, not the cable:
| Bash | |
|---|---|
Then:
Reconnect and confirm Docker works:
4. Give the USB link an address¶
The cable gives you a network interface but no address on it. This unit file sets one at every boot:
Now plug the USB-C cable from your laptop into the Pi's USB-C port (the one that normally takes power). From your laptop:
| Bash | |
|---|---|
If that answers, the hard part is done. If it does not, it is almost certainly a charge-only cable — see Troubleshooting, which applies here too.
Why a fixed address and not DHCP? Because
10.55.0.1is then the same on every box and can be written into a.bazelrcthat just works. Your laptop gives itself a169.254.x.xlink-local address on that interface, which is enough to reach the Pi. Our boxes also rundnsmasqonusb0to hand the laptop a proper10.55.0.x, which is tidier but not needed.
5. Get a cluster UUID¶
- Go to https://my.engflow.com and sign in or create an account.
- Start a Local Trial.
- Copy the Cluster UUID.
These trial licenses are short-lived — hours, not months. That is fine, because the box re-fetches its license every time it starts and every five minutes while running (step 7 sets that up). Restart the trial on my.engflow.com when it lapses.
6. The one thing you would otherwise get stuck on¶
Left alone, MyEngFlow Mini starts a bundled envoy proxy in front of itself.
On a Raspberry Pi 4, envoy cannot start at all. Its memory allocator asks the
kernel for addresses around 23 TB, and Raspberry Pi OS builds its 64-bit kernel
with a 512 GB address space, so those addresses do not exist and envoy aborts:
| Text Only | |
|---|---|
This is not a RAM problem — it happens with 3 GB free — and no environment variable fixes it, because the address is baked into the envoy binary. Do not spend an evening on it as we did.
Mini only needs two things from that process, so you can replace it with about 250
lines of C. Create envoy_shim.c:
You need a program that does exactly two things:
- answers HTTP 200 on
127.0.0.1:9902/ready— Mini polls this and refuses to start without it; - accepts connections on
0.0.0.0:8080and relays them, byte for byte, to127.0.0.1:8081, where Mini's own server is listening.
That is the whole contract. A straightforward accept() / fork() TCP relay does
it. Two details are worth getting right, both of which bit us:
- Build it statically —
gcc -O2 -static -Wall -o envoy_shim envoy_shim.c— because the Mini container image is distroless and has no shared libraries for a dynamically-linked binary to find. - Make the relay non-blocking, with a separate buffer per direction. Our first
version used blocking writes, and it was crawlingly slow: this traffic is HTTP/2,
and a stalled write in one direction blocks the flow-control frames travelling
the other way. Poll each direction for reading only when its buffer has room and
for writing only when it has bytes pending, and propagate half-close with
shutdown(SHUT_WR).
Ours relays at ~480–519 MB/s, far above any link it will ever carry, so a simple
implementation is not the bottleneck. If you would rather not write this, ask
your EngFlow contact for envoy_shim.c — it is the one piece of this that is
ours rather than public.
Then the second half of the same problem. With envoy gone, nothing translates
protocols any more: Mini's server speaks only h2c (HTTP/2 without TLS), and
browsers refuse to speak h2c. So the cache works for Bazel but the UI will not load
in a browser. nghttpx (installed in step 3) fixes it by terminating ordinary
HTTP on port 8088 and forwarding as h2c:
So: Bazel talks to 8080, browsers to 8088. That split is why this guide's
.bazelrc has 8080 for the cache and 8088 for the results URL.
7. Start it¶
Paste this, and put your cluster UUID on the CLUSTER_UUID line. The
/home/pi paths must match your actual username:
Start it and watch:
| Bash | |
|---|---|
Wait for the EngFlow logo and a line naming the license validity window. Ctrl-C stops following the logs; the containers keep running. Then:
| Bash | |
|---|---|
HEALTHY means you are done. Note the --http2-prior-knowledge flag: it is
required. Port 8080 does not speak ordinary HTTP, so a plain curl reports
nothing at all and exit code 000 on a perfectly healthy box. That flag cost us real
debugging time.
Start it at boot, so the box works after a power cut with nobody logged in:
And add the license retry timer. Do not skip this one. Mini tries five times to get a license and then gives up permanently — the container stays up, serving errors, so nothing restarts it. If the box ever starts without a working network, it will sit there dead until someone restarts it by hand. We found this the slow way: connecting Wi-Fi hours later changed nothing.
With that timer, a box you plug into a network starts serving within about five minutes on its own. Measured on ours: 90 seconds, with nothing typed.
8. Point Bazel at it¶
Exactly as in step 2 of the quick start:
| Text Only | |
|---|---|
Build twice with bazel build --config=efcache //.... The first build fills the
cache; the second should be dramatically faster and report remote cache hit for
most actions. The UI is at
http://10.55.0.1:8088.
Everything else in this guide now applies to your box too — the UI, the
.local name, what to do when the cache fills up,
changing the password, the security
notes. Substitute your own username for myefbox and your own hostname for
myEFbox-001.
Things we got wrong, so you don't have to¶
Each of these cost us hours:
- A charge-only USB-C cable. By far the most common failure. The board powers up and its LED lights, so it looks fine, but no network interface ever appears. Use a cable you have moved data with.
curl http://…:8080/healthzreturning nothing. Not a broken box. Port 8080 is HTTP/2-only; you need--http2-prior-knowledge.- The UI not loading in a browser while Bazel works perfectly. That is the
missing h2c translation — step 6's
nghttpx. - Freeing space by deleting cache files. Do not.
find …/cas -deletefrees disk but the server tracks what it holds in a database, so it still refuses new writes and now has records pointing at files that are gone. We measured this: 300 MB deleted, filesystem 52% down to 36%, writes still refused. Restart instead — it reconciles the accounting and keeps every blob. See If the cache fills up. - Trying to give envoy the address space it wants. We built two 16 KB-page kernels to raise the address limit. Neither boots on a Pi 4 — no console, no network, dead. The shim is the answer; the kernel is not.
- Sizing the Java heap by RAM alone. Leave most of the RAM to the page cache rather than the heap. A cache server is fast because blobs are served from RAM instead of read back off the SD card, and that is the page cache's job, not the JVM's.
- Expecting the container memory limits to apply. Raspberry Pi OS ships with
cgroup memory accounting off, so Docker silently discards
memory:limits.JAVA_TOOL_OPTIONS -Xmxis the ceiling that actually takes effect. (We left the limits in the file above anyway; they start working if you ever enable accounting.)
What to expect from it¶
Measured on a Pi 4 (4 GB) with a MicroSD card, building a real 1804-action open-source Bazel project:
| Build time | |
|---|---|
| No cache | 85 s |
| Cached, over the USB-C cable | 3.8 s |
| Cached, over the Pi's Wi-Fi | 52 s |
Use the cable. It is about 13× faster than this board's Wi-Fi, which is the bottleneck for anything large. Wired Ethernet lands between the two.
An SSD over USB 3.0 instead of a MicroSD card is the usual advice, and it will help writes. But on our measurements the card was not the limit for cached builds — the USB 2.0 gadget link (~31–37 MB/s) was, and the CPU peaked at only ~54%. Start with a good card; add an SSD if you find yourself waiting on cold builds.
About the Pi 5¶
A Pi 5 works fine as a cache — it boots, provisions, licenses and serves, and we verified all of that on real hardware. But we could not get the single-USB-C-cable link working on it. On a Pi 5 connected to a MacBook Pro with the same cable and port that work on a Pi 4, the interface never appears and has carried zero packets. Raspberry Pi's own documentation says the Pi 5 supports device mode on that port, so this may be a cable or host issue rather than a hard limitation — but we could not resolve it, and it is why we build these boxes on Pi 4s.
If you use a Pi 5, plan on Ethernet or Wi-Fi and use the box's network address
everywhere this guide says 10.55.0.1. Everything else applies unchanged. The Pi 5
also draws more power: check vcgencmd get_throttled returns 0x0 on a
laptop-powered board, and if it does not, power it from a proper supply.
What is not in this guide¶
The boxes we hand out are made from a prebuilt card image that also does a few things this guide leaves out, because they matter for producing a hundred identical units and not for making one:
- provisioning entirely offline, with Docker and both container images baked into the image, so no unit needs Wi-Fi while it is being made;
- sizing the heap to the board automatically at every start, so one image works on 2 / 4 / 8 GB boards and corrects itself if the card is moved;
dnsmasqon the USB link,avahifor the.localname, UTC clock, and a per-unit hostname;- a verification tool that runs ~50 checks on a finished unit.
None of that changes how the box performs. If you want it, that tooling is what your EngFlow contact hands to whoever is building units in quantity.
Credits¶
The box these are given away in is a remix of https://makerworld.com/en/models/2311113, used under the Creative Commons Attribution 4.0 licence (https://creativecommons.org/licenses/by/4.0/).
© 2020–2026 EngFlow