Kubernetes single Pod driver

Preevy can provision environments on a Kubernetes cluster using the bundled Kubernetes single Pod driver.

Why deploy on Kubernetes?

If you're a a Kubernetes user, cost and time to provision ephemeral preview environments can be reduced considerably.

While preview environments live for the duration of a Pull Request, they will typically only see little bursts of actual usage. Kubernetes is a great way to oversubsrcibe compute resources. Environments can be configured to require little CPU and memory while idle and waiting for a review.

Deployments are faster on Kubernetes comparing to your regular cloud provider VMs.

How?

The driver creates a Kubernetes Pod running a Docker server for each environment (herby the name "single Pod"). Preevy then connects to the Docker server to build and run your services, just like it does on a regular VM.

Your services are still exposed using the Preevy Tunnel Service - there's no need to configure a specific ingress.

Requirements

• A Kubernetes cluster should be accessible from the CLI. A kubeconfig file may be specified - see flags below.
• When using RBAC authorization, the default context needs permissions for running exec and port-forward for specific pods in the configured namespace.
• The kubectl tool needs to be installed and available in the PATH.
• By default, the driver runs a Pod with privileged: true security context. In some cases, this requirement may be lifted by customizing the deployment template, see below.

Supported options

option	flag	default	env var	description
namespace	--kube-pod-namespace	default		Kubernetes namespace to provision resources in
kubeconfig	--kube-pod-kubeconfig	$HOME/.kube	KUBECONFIG	path to a kubeconfig file
pod-template	--kube-pod-template	default template		path to a nunjacks template used to provision Kubernetes resources per environment. See below for details
server-side-apply	--[no-]kube-pod-server-side-apply	true		if true, provision resources using server-side apply, else using client-side apply (CREATE/PATCH). Applies to preevy up only

Overriding options

Similar to other drivers, options are saved in the Preevy profile to be used as default values for all operations.

Options can be overridden for a specific compose file by adding them to the x-preevy section:

services:
  ...
x-preevy:
  driver: kube-pod
  drivers:
    kube-pod:
      namespace: other-namespace

Options can also be overridden using a CLI flag per command execution:

preevy up --kube-pod-namespace=other-namespace

Customizing the provisioned Kubernetes resources

It's possible to customize the Kubernetes resources provisioned by the driver per environment. Use cases include, but are not limited to:

• Customize the Docker Server image or the Docker Server configuration
• Add labels/annotations to the provisioned resources, e.g, for selecting specific Kuebernetes nodes
• Provisioning additional resources per environment, e.g, a database server

The resources are specified as Kubernetes object specs in a single YAML file rendered from a nunjucks template. The template file may contain multiple definitions separated by lines containing --- (three dashes).

Start by copying the default template. To use the custom template, enter the path to the custom template file at the preevy init or preevy profile create command, or specify the --kube-pod-template flag for the preevy up and preevy down commands. The template file at the specified path needs to be accessible at runtime to the CLI¹.

Requirements for the provisioned resources

All resources need to be deployed in a single namepsace, specified as a template argument (see below).

While multiple Kubernetes Deployment objects may be defined, exactly one Deployment must have the label app.kubernetes.io/component: docker-host:

• The status of the Deployment is used to determine whether the Preevy environment is ready.
• The first container of the Deployment spec is used for copying files, and so it must have the tar and find commands available.

A Docker server must be listening on port 2375 of the Deployment's Pod. As Preevy uses the port-forward API to connect to the Docker server, it does not need to be exposed as a service. For the same reason, TLS is not supported and needs to be disabled for this port.

The Docker server must also be listening on the unix socket path /var/run/docker.sock - this is used by the Preevy agent service running alongside your services.

The lifecycle of all resources is tied to a Preevy environment - they will be created and deleted with the environment.

Template arguments

The following arguments are specified when rendering the template:

• namespace: the Kuberentes namespace saved in the Preevy profile or specified in the --kube-pod-namespace flag. All resources must be defined in this namespace.
• id: A generated ID for this environment, 53 characters or less, comprised of the Preevy environment ID and a random suffix. id can be used as part for of a label value, with up to 10 additional characters as to not exceed the 63 character limit for labels

Configuring rootless unprivileged Docker-in-Docker

By default, the Kubernetes Docker-in-Docker driver creates a Deployment which runs the docker:dind image. Traditionally, running Docker inside a container requires the privileged: true security context, which may be a security concern.

Sysbox is an OSS project (acquired by Docker) which allows running unprivileged containers in a Kubernetes cluster. It can be installed on most of the popular Kubernetes distros including managed cloud platforms like Amazon EKS, Google GKE and Azure AKA. Once installed, a custom template can be used to provision Pods without the privileged security context.

---

1. Embedding the template in the profile, or specifying its path in the x-preevy section of the Docker Compose file is in the roadmap, but not implemented yet.↩