Skip to content

Sysbox sandbox

Experimental

This feature is experimental. It may be incomplete, unstable, or removed in the future.

Info

This feature is disabled by default; please reach out to us if you would like to use it.

Sysbox is an alternative docker runtime that features better isolation than the default runc runtime, and the ability to run system level software such as systemd and docker itself, out-of-the-box.

Docker-in-Docker (DinD)

One of the most useful applications of sysbox is running nested docker without forwarding the worker's docker socket. Not only does this setup avoid leaking docker containers after action execution, but it can also make it possible for users to set up dockerd to listen to additional addresses or sockets, and also access ports forwarded on the docker host.

Using Docker-in-Docker on your EngFlow cluster

  1. Set up a container image with the following Dockerfile and entrypoint.sh script:

    Dockerfile
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
    FROM ubuntu:22.04

RUN apt-get update -yq
RUN apt-get install -yq ca-certificates curl gnupg
RUN install -m 0755 -d /etc/apt/keyrings
RUN curl --insecure -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
RUN chmod a+r /etc/apt/keyrings/docker.gpg
RUN echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  tee /etc/apt/sources.list.d/docker.list > /dev/null

ENV ENGFLOW_RBE_USER_ID=108
ENV ENGFLOW_RBE_USER_NAME=engflow
ENV ENGFLOW_RBE_GROUP_ID=114
ENV ENGFLOW_RBE_GROUP_NAME=engflow

RUN groupadd --non-unique -g "${ENGFLOW_RBE_GROUP_ID}" docker
RUN groupadd --non-unique -g "${ENGFLOW_RBE_GROUP_ID}" "${ENGFLOW_RBE_GROUP_NAME}"

RUN useradd \
    -rm \
    -s /bin/bash \
    -g docker \
    -u "${ENGFLOW_RBE_USER_ID}" \
    "${ENGFLOW_RBE_USER_NAME}"

RUN apt-get update -yq

RUN apt-get install -yq sudo
RUN echo "${ENGFLOW_RBE_USER_NAME} ALL=(ALL) NOPASSWD:ALL" > "/etc/sudoers.d/${ENGFLOW_RBE_USER_NAME}"
RUN chmod 0440 "/etc/sudoers.d/${ENGFLOW_RBE_USER_NAME}"

RUN apt-get install -yq docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

COPY --chmod=0755 entrypoint.sh /entrypoint.sh

USER "${ENGFLOW_RBE_USER_ID}":"${ENGFLOW_RBE_GROUP_ID}"
WORKDIR "/home/${ENGFLOW_RBE_USER_NAME}"

ENTRYPOINT [ "/entrypoint.sh" ]
    entrypoint.sh
    1
2
3
4
5
6
7
8
9
    #!/usr/bin/env bash
sudo dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock &>>/dev/null &

while (! docker stats --no-stream &>>/dev/null ); do
  # Docker takes a few seconds to initialize
  sleep 1
done

eval "$@"

  2. Test the image locally by installing sysbox-runc and running the following command:

    Bash
    1
    docker run --rm -it -u 108:114 --runtime=sysbox-runc <YOUR_SYSBOX_DIND_DOCKER_IMAGE> docker ps`

    Make sure it shows the output of a docker ps command with no containers running,

  3. Push the image to an OCI registry accessible to the cluster.

  4. Use DinD in your Remote Execution build by adding the following platform options

    BUILD.bazel
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
    a_target(
    name = "fizz",
    srcs = ["buzz"],
    exec_properties = {
        "container-image": "docker://YOUR_SYSBOX_DIND_DOCKER_IMAGE",
        "dockerRuntime": "sysbox-runc",
        "Pool": "sysbox", # this can change depending on what pool has sysbox enabled
        "dockerNetwork": "standard",
        # DO NOT ENABLE!!! "dockerSiblingContainers": "True",
        # it will cause the worker to mount /var/run/docker.sock
        # but the goal is to have the container itself run dockerd
        # and this will break it
        # DO NOT ENABLE!!! "dockerPrivileged": "True",
        # sysbox containers will refuse to start
    },
)