Generate Config Files

./setup/k8s/gen-k8s-config.py asks about the following information before writing any output files.

Output directory ($ENGFLOW_OUT_DIR)

Path where the script will write all outputs. Can be relative or absolute. If missing, it will be created.

The script prints all output files it generates, and also prints how to use them. All paths in these messages reference the output directory.

License file ($ENGFLOW_LICENSE)

Path to the .license file containing a valid license.

The script copies the file into the output directory. The generated *.Dockerfile files copy the license file into the container image.

EngFlow service .deb file path ($ENGFLOW_DEB_FILE)

Path to the .deb file containing the EngFlow services. It is under the ./setup directory.

The script copies the file into the output directory. The generated *.Dockerfile files copy the .deb file into the container image.

Kubernetes namespace ($ENGFLOW_K8S_NAMESPACE)

The Kubernetes Namespace where all Kubernetes objects will be created.

The generated yaml files and config files all reference it.

Docker image tag for Pods ($ENGFLOW_IMAGE_TAG)

A Docker image tag, e.g. gcr.io/my/image:latest.

Worker and scheduler pods use the same image.

The generated service.yaml file references this in the Pod specifications. The Pods must be able to pull these images. You may need to add an imagePullPolicy entry in the yaml files, under .spec.template.spec.containers.

Node label selector for Pods ($ENGFLOW_WORKER_NODE_LABEL and $ENGFLOW_SCHEDULER_NODE_LABEL)

Kubernetes Node Labels, that Deployments can deploy Pods to. See Kubernetes objects for more info.

The generated worker.yaml and scheduler.yaml files reference these under .spec.template.spec.nodeSelector.

Number of Pod instances ($ENGFLOW_WORKER_INSTANCES and $ENGFLOW_SCHEDULER_INSTANCES)

Number of Pod instances that Kubernetes should deploy. Each Pod hosts one Worker or Scheduler instance, and one Node may host multiple Pods – deployment is up to Kubernetes.

The generated worker.yaml and scheduler.yaml files reference these numbers, under .spec.replicas.

Choose these limits according to the available resources on Nodes. If your worker pool has 4 Nodes that are big enough to host 2 worker instances, you can set the worker instances to 8.

Remember that the EngFlow Remote Execution service will use only as many CPU cores as your license allows.