Kubernetes single Pod driver
Preevy can provision environments on a Kubernetes cluster using the bundled Kubernetes single Pod driver.
Why deploy on Kubernetes?
If you're a Kubernetes user, the cost and time to provision ephemeral preview environments can be reduced considerably.
While preview environments live for the duration of a Pull Request, they will typically only see little bursts of actual usage. Kubernetes is a great way to oversubscribe compute resources. Environments can be configured to require little CPU and memory while idle and waiting for a review.
Deployments are faster on Kubernetes compared to your regular cloud provider VMs.
The driver creates a Kubernetes Pod running a Docker server for each environment (herby the name "single Pod"). Preevy then connects to the Docker server to build and run your services, just like it does on a regular VM.
Your services are still exposed using the Preevy Tunnel Service - there's no need to configure a specific ingress.
- A Kubernetes cluster should be accessible from the CLI. A
kubeconfigfile may be specified - see flags below.
- When using RBAC authorization, the default context needs permissions for running exec and port-forward for specific pods in the configured namespace.
kubectltool needs to be installed and available in the PATH.
- By default, the driver runs a Pod with
privileged: truesecurity context. In some cases, this requirement may be lifted by customizing the deployment template, see below.
- A StorageClass must be defined in the cluster to enable dynamic volume provisioning. This is usually pre-defined in your Kubernetes cluster.
|Kubernetes namespace to provision resources in
|path to a
|path to a nunjacks template used to provision Kubernetes resources per environment. See below for details
|if true, provision resources using server-side apply, else using client-side apply (CREATE/PATCH). Applies to
preevy up only
|The Kubernetes StorageClass used in the StatefulSet template to provision the data volume for the Docker server in the Preevy environment Pod
|Size in GiB of the volume allocated for the Docker server in the Preevy environment Pod. Adjust this acccording to the storage requirements of your environment.
Similar to other drivers, options are saved in the Preevy profile to be used as default values for all operations.
Options can be overridden for a specific compose file by adding them to the
Options can also be overridden using a CLI flag per command execution:
preevy up --kube-pod-namespace=other-namespace --kube-pod-storage-size=12.5
Customizing the provisioned Kubernetes resources
It's possible to customize the Kubernetes resources provisioned by the driver per environment. Use cases include, but are not limited to:
- Customize the Docker Server image or the Docker Server configuration
- Add labels/annotations to the provisioned resources, e.g, for selecting specific Kubernetes nodes
- Provisioning additional resources per environment, e.g, a database server
The resources are specified as Kubernetes object specs in a single YAML file rendered from a nunjucks template. The template file may contain multiple definitions separated by lines containing
--- (three dashes).
Start by copying the default template. To use the custom template, enter the path to the custom template file at the
preevy init or
preevy profile create command, or specify the
--kube-pod-template flag for the
preevy up and
preevy down commands. The template file at the specified path needs to be accessible at runtime to the CLI1.
Requirements for the provisioned resources
All resources need to be deployed in a single namespace, specified as a template argument (see below).
While multiple Kubernetes StatefulSet objects may be defined, exactly one StatefulSet must have the label
- The status of the StatefulSet is used to determine whether the Preevy environment is ready.
- The first container of the StatefulSet spec is used for copying files, so it must have the
A Docker server must be listening on port 2375 of the StatefulSet's Pod. As Preevy uses the port-forward API to connect to the Docker server, it does not need to be exposed as a service. For the same reason, TLS is not supported and needs to be disabled for this port.
The Docker server must also be listening on the unix socket path
/var/run/docker.sock - this is used by the Preevy agent service running alongside your services.
The lifecycle of all resources is tied to a Preevy environment - they will be created and deleted with the environment.
The following arguments are specified when rendering the template:
namespace: the Kubernetes namespace saved in the Preevy profile or specified in the
--kube-pod-namespaceflag. All resources must be defined in this namespace.
id: A generated ID for this environment, 53 characters or less, comprised of the Preevy environment ID and a random suffix.
idcan be used as part of a label value, with up to 10 additional characters so as to not exceed the 63-character limit for labels
storageClass: The Kubernetes StorageClass used to dynamically provision a volume for the StatefulSet. Saved in the Preevy profile or specified in the
--kube-pod-storage-classflag. Leaving this undefined will cause the default storage class to be used.
storageSize: The size of the allocated volume in GiB. Saved in the Preevy profile or specified in the
Configuring rootless unprivileged Docker-in-Docker
By default, the Kubernetes Docker-in-Docker driver creates a StatefulSet which runs the
docker:dind image. Traditionally, running Docker inside a container requires the
privileged: true security context, which may be a security concern.
Sysbox is an OSS project (acquired by Docker) that allows running unprivileged containers in a Kubernetes cluster. It can be installed on most of the popular Kubernetes distros including managed cloud platforms like Amazon EKS, Google GKE, and Azure AKA. Once installed, a custom template can be used to provision Pods without the
privileged security context.
- Embedding the template in the profile, or specifying its path in the
x-preevysection of the Docker Compose file is in the roadmap, but not implemented yet.↩