Sysbox sandbox
Experimental
This feature is experimental. It may be incomplete, unstable, or removed in the future.
Info
This feature is disabled by default; please reach out to us if you would like to use it.
Sysbox is an alternative docker runtime that features better isolation than the default runc
runtime and the ability to run system level software out-of-the-box (eg: systemd, docker itself, etc.).
Docker-IN-Docker (DIND)
One of the most useful applications of sysbox is running nested docker without forwarding the worker's docker socket.
Not only does this setup avoid leaking docker containers after action execution, but it can also make it possible for
users to setup dockerd to listen to additional addresses or sockets and also access ports forwarded on the docker host.
Example setup
In order to use DIND with sysbox on your Engflow cluster, you first need to setup a container image and push it to an OCI
image registry accissble to the cluster.
Dockerfile |
---|
| FROM ubuntu:22.04
RUN apt-get update -yq
RUN apt-get install -yq ca-certificates curl gnupg
RUN install -m 0755 -d /etc/apt/keyrings
RUN curl --insecure -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
RUN chmod a+r /etc/apt/keyrings/docker.gpg
RUN echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null
ENV ENGFLOW_RBE_USER_ID=108
ENV ENGFLOW_RBE_USER_NAME=engflow
ENV ENGFLOW_RBE_GROUP_ID=114
ENV ENGFLOW_RBE_GROUP_NAME=engflow
RUN groupadd --non-unique -g "${ENGFLOW_RBE_GROUP_ID}" docker
RUN groupadd --non-unique -g "${ENGFLOW_RBE_GROUP_ID}" "${ENGFLOW_RBE_GROUP_NAME}"
RUN useradd \
-rm \
-s /bin/bash \
-g docker \
-u "${ENGFLOW_RBE_USER_ID}" \
"${ENGFLOW_RBE_USER_NAME}"
RUN apt-get update -yq
RUN apt-get install -yq sudo
RUN echo "${ENGFLOW_RBE_USER_NAME} ALL=(ALL) NOPASSWD:ALL" > "/etc/sudoers.d/${ENGFLOW_RBE_USER_NAME}"
RUN chmod 0440 "/etc/sudoers.d/${ENGFLOW_RBE_USER_NAME}"
RUN apt-get install -yq docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
COPY entrypoint.sh /entrypoint.sh
USER "${ENGFLOW_RBE_USER_ID}":"${ENGFLOW_RBE_GROUP_ID}"
WORKDIR "/home/${ENGFLOW_RBE_USER_NAME}"
ENTRYPOINT [ "/entrypoint.sh" ]
|
entrypoint.sh |
---|
| #!/usr/bin/env bash
sudo dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock &>>/dev/null &
while (! docker stats --no-stream &>>/dev/null ); do
# Docker takes a few seconds to initialize
echo "Waiting for Docker to launch..."
sleep 1
done
eval "$@"
|
You can test the image locally by installing sysbox-runc and running the following command:
docker run --rm -it -u 108:114 --runtime=sysbox-runc <YOUR_SYSBOX_DIND_DOCKER_IMAGE> docker ps
If this shows the output of a docker ps
command with no containers running, then it works. After pushing the image to an
OCI registry accessible to the cluster, you can now it as part of your build by adding the following platform options
BUILD.bazel |
---|
| a_target(
name = "fizz",
srcs = ["buzz"],
exec_properties = {
"container-image": "docker://YOUR_SYSBOX_DIND_DOCKER_IMAGE",
"dockerRuntime": "sysbox-runc",
"Pool": "sysbox", # this can change depending on what pool has sysbox enabled
"dockerNetwork": "standard",
# DO NOT ENABLE!!! "dockerSiblingContainers": "True",
# it will cause the worker to mount /var/run/docker.sock
# but the goal is to have the container itself run dockerd
# and this will break it
# DO NOT ENABLE!!! "dockerPrivileged": "True",
# sysbox containers will refuse to start
},
)
|