Skip to content

Sysbox sandbox

Experimental

This feature is experimental. It may be incomplete, unstable, or removed in the future.

Info

This feature is disabled by default; please reach out to us if you would like to use it.

Sysbox is an alternative docker runtime that features better isolation than the default runc runtime and the ability to run system level software out-of-the-box (eg: systemd, docker itself, etc.).

Docker-IN-Docker (DIND)

One of the most useful applications of sysbox is running nested docker without forwarding the worker's docker socket. Not only does this setup avoid leaking docker containers after action execution, but it can also make it possible for users to setup dockerd to listen to additional addresses or sockets and also access ports forwarded on the docker host.

Example setup

In order to use DIND with sysbox on your Engflow cluster, you first need to setup a container image and push it to an OCI image registry accissble to the cluster.

Dockerfile
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
FROM ubuntu:22.04

RUN apt-get update -yq
RUN apt-get install -yq ca-certificates curl gnupg
RUN install -m 0755 -d /etc/apt/keyrings
RUN curl --insecure -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
RUN chmod a+r /etc/apt/keyrings/docker.gpg
RUN echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null

ENV ENGFLOW_RBE_USER_ID=108
ENV ENGFLOW_RBE_USER_NAME=engflow
ENV ENGFLOW_RBE_GROUP_ID=114
ENV ENGFLOW_RBE_GROUP_NAME=engflow

RUN groupadd --non-unique -g "${ENGFLOW_RBE_GROUP_ID}" docker
RUN groupadd --non-unique -g "${ENGFLOW_RBE_GROUP_ID}" "${ENGFLOW_RBE_GROUP_NAME}"

RUN useradd \
    -rm \
    -s /bin/bash \
    -g docker \
    -u "${ENGFLOW_RBE_USER_ID}" \
    "${ENGFLOW_RBE_USER_NAME}"

RUN apt-get update -yq

RUN apt-get install -yq sudo
RUN echo "${ENGFLOW_RBE_USER_NAME} ALL=(ALL) NOPASSWD:ALL" > "/etc/sudoers.d/${ENGFLOW_RBE_USER_NAME}"
RUN chmod 0440 "/etc/sudoers.d/${ENGFLOW_RBE_USER_NAME}"

RUN apt-get install -yq docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

COPY entrypoint.sh /entrypoint.sh

USER "${ENGFLOW_RBE_USER_ID}":"${ENGFLOW_RBE_GROUP_ID}"
WORKDIR "/home/${ENGFLOW_RBE_USER_NAME}"

ENTRYPOINT [ "/entrypoint.sh" ]
entrypoint.sh
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
#!/usr/bin/env bash
sudo dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock &>>/dev/null &

while (! docker stats --no-stream &>>/dev/null ); do
  # Docker takes a few seconds to initialize
  echo "Waiting for Docker to launch..."
  sleep 1
done

eval "$@"

You can test the image locally by installing sysbox-runc and running the following command:

docker run --rm -it -u 108:114 --runtime=sysbox-runc <YOUR_SYSBOX_DIND_DOCKER_IMAGE> docker ps

If this shows the output of a docker ps command with no containers running, then it works. After pushing the image to an OCI registry accessible to the cluster, you can now it as part of your build by adding the following platform options

BUILD.bazel
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
a_target(
    name = "fizz",
    srcs = ["buzz"],
    exec_properties = {
        "container-image": "docker://YOUR_SYSBOX_DIND_DOCKER_IMAGE",
        "dockerRuntime": "sysbox-runc",
        "Pool": "sysbox", # this can change depending on what pool has sysbox enabled
        "dockerNetwork": "standard",
        # DO NOT ENABLE!!! "dockerSiblingContainers": "True",
        # it will cause the worker to mount /var/run/docker.sock
        # but the goal is to have the container itself run dockerd
        # and this will break it
        # DO NOT ENABLE!!! "dockerPrivileged": "True",
        # sysbox containers will refuse to start
    },
)