Skip to content

Sysbox sandbox

Experimental

This feature is experimental. It may be incomplete, unstable, or removed in the future.

Info

This feature is disabled by default; please reach out to us if you would like to use it.

Sysbox is an alternative docker runtime that features better isolation than the default runc runtime, and the ability to run system level software such as systemd and docker itself, out-of-the-box.

Docker-in-Docker (DinD)

One of the most useful applications of sysbox is running nested docker without forwarding the worker's docker socket. Not only does this setup avoid leaking docker containers after action execution, but it can also make it possible for users to set up dockerd to listen to additional addresses or sockets, and also access ports forwarded on the docker host.

Using Docker-in-Docker on your EngFlow cluster

  1. Set up a container image with the following Dockerfile and entrypoint.sh script:

    Dockerfile
    FROM ubuntu:22.04
    
    RUN apt-get update -yq
    RUN apt-get install -yq ca-certificates curl gnupg
    RUN install -m 0755 -d /etc/apt/keyrings
    RUN curl --insecure -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    RUN chmod a+r /etc/apt/keyrings/docker.gpg
    RUN echo \
      "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
      "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
      tee /etc/apt/sources.list.d/docker.list > /dev/null
    
    ENV ENGFLOW_RBE_USER_ID=108
    ENV ENGFLOW_RBE_USER_NAME=engflow
    ENV ENGFLOW_RBE_GROUP_ID=114
    ENV ENGFLOW_RBE_GROUP_NAME=engflow
    
    RUN groupadd --non-unique -g "${ENGFLOW_RBE_GROUP_ID}" docker
    RUN groupadd --non-unique -g "${ENGFLOW_RBE_GROUP_ID}" "${ENGFLOW_RBE_GROUP_NAME}"
    
    RUN useradd \
        -rm \
        -s /bin/bash \
        -g docker \
        -u "${ENGFLOW_RBE_USER_ID}" \
        "${ENGFLOW_RBE_USER_NAME}"
    
    RUN apt-get update -yq
    
    RUN apt-get install -yq sudo
    RUN echo "${ENGFLOW_RBE_USER_NAME} ALL=(ALL) NOPASSWD:ALL" > "/etc/sudoers.d/${ENGFLOW_RBE_USER_NAME}"
    RUN chmod 0440 "/etc/sudoers.d/${ENGFLOW_RBE_USER_NAME}"
    
    RUN apt-get install -yq docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    
    COPY --chmod=0755 entrypoint.sh /entrypoint.sh
    
    USER "${ENGFLOW_RBE_USER_ID}":"${ENGFLOW_RBE_GROUP_ID}"
    WORKDIR "/home/${ENGFLOW_RBE_USER_NAME}"
    
    ENTRYPOINT [ "/entrypoint.sh" ]
    
    entrypoint.sh
    #!/usr/bin/env bash
    
    check_docker_is_alive () {
      while (! docker stats --no-stream &>>/dev/null ); do
        # Docker takes a few seconds to initialize.
        sleep 1
      done
    }
    export -f check_docker_is_alive
    
    sudo dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock &>>/dev/null &
     # Time out after 1m to avoid waiting on docker forever.
    timeout 60s bash -c check_docker_is_alive 
    eval "$@"
    
  2. Test the image locally by installing sysbox-runc and running the following command:

    Bash
    docker run --rm -it -u 108:114 --runtime=sysbox-runc <YOUR_SYSBOX_DIND_DOCKER_IMAGE> docker ps`
    

    Make sure it shows the output of a docker ps command with no containers running,

  3. Push the image to an OCI registry accessible to the cluster.

  4. Use DinD in your Remote Execution build by adding the following platform options

    BUILD.bazel
    a_target(
        name = "fizz",
        srcs = ["buzz"],
        exec_properties = {
            "container-image": "docker://YOUR_SYSBOX_DIND_DOCKER_IMAGE",
            "dockerRuntime": "sysbox-runc",
            "Pool": "sysbox", # this can change depending on what pool has sysbox enabled
            "dockerNetwork": "standard",
            # DO NOT ENABLE!!! "dockerSiblingContainers": "True",
            # it will cause the worker to mount /var/run/docker.sock
            # but the goal is to have the container itself run dockerd
            # and this will break it
            # DO NOT ENABLE!!! "dockerPrivileged": "True",
            # sysbox containers will refuse to start
        },
    )
    

Try out the example

See instructions in https://github.com/EngFlow/example/tree/main/docker/sysbox and try out your DinD container using the sample code.