Single-machine setup

To try Engflow Remote Exection locally, you can either run minikube / minishift, or run the service without Kubernetes.

Starting a Remote Execution Cluster

Using Kubernetes on minikube / minishift

You have to follow the same general process as with setting up a real Kubernetes cluster, with a few differences.

1. Generate configuration files.

   When the script asks for:

   • The Docker image tags, enter engflow-worker and engflow-scheduler.

   • The worker and scheduler instance count, use the default values.

2. Edit the generated scheduler.yaml and worker.yaml files.

   • Add imagePullPolicy: Never next to image: under .spec.template.spec.containers. This will prevent Pods from pulling the image from a registry: we will build and keep the images locally.

   • Remove the .spec.template.spec.nodeSelector entry. Since all Pods will be deployed to the same and only Node, we don’t need Node selection.

3. Build the Docker images and create the Kubernetes objects,

   These instructions are for minikube, but you can do the same for minishift.

   1. Apply minikube's context:

      eval $(minikube docker-env)
      

   2. In the same terminal, build the Docker images:

      docker build -t engflow-worker -f <OUT>/worker.Dockerfile <OUT>
      docker build -t engflow-scheduler -f <OUT>/scheduler.Dockerfile <OUT>
      

      You don’t need to tag or push the images.

   3. Run the kubectl commands printed by gen-k8s-config.py.

   4. Check if Pods are running:

      kubectl get pod -n <NAMESPACE>
      

4. Find the IP address and port of the Load Balancer scheduler-grpc-service.

   minikube service scheduler-grpc-service --url -n engflow-re
   

   Note the port for later when you edit the ~/.bazelrc file.

5. Set up DNS.

   You may temporarily add an entry to /etc/hosts with this IP and demo.engflow.com.

Without Kubernetes

You can try the service by running its instances locally, without Kubernetes.

1. Generate configuration files.

   Use the default values for all Docker- and Kubernetes-related questions.

2. Build the Docker images and start the service instances.

   1. Build the Docker images:

      docker build -t engflow-worker -f <OUT>/worker.Dockerfile <OUT>
      docker build -t engflow-scheduler -f <OUT>/scheduler.Dockerfile <OUT>
      

      No need to tag or push them.

   2. Pick four unused local ports (e.g. 30000, 30001, 30002, 30003), and start two Worker and one Scheduler instances.

      docker run -p 30000:10081 engflow-worker --discovery=multicast
      docker run -p 30001:10081 engflow-worker --discovery=multicast
      docker run -p 30002:10081 -p 30003:8080 engflow-scheduler --discovery=multicast
      

      The Scheduler will listen on port 30003. We will use this in ~/.bazelrc in the next step.

3. Set up DNS.

   Since everything runs locally, demo.engflow.com is 127.0.0.1. You may temporarily add an entry to /etc/hosts.

Bazel Configuration

In both cases, use the same WORKSPACE and .bazelrc files as with an actual Kubernetes cluster, but use the port from above when you add the --remote_executor flag to the ~/.bazelrc file.

Usage

Same as on an actual cluster.