Kubernetes single Pod driver
Preevy can provision environments on a Kubernetes cluster using the bundled Kubernetes single Pod driver.
Why deploy on Kubernetes?
If you're a a Kubernetes user, cost and time to provision ephemeral preview environments can be reduced considerably.
While preview environments live for the duration of a Pull Request, they will typically only see little bursts of actual usage. Kubernetes is a great way to oversubsrcibe compute resources. Environments can be configured to require little CPU and memory while idle and waiting for a review.
Deployments are faster on Kubernetes comparing to your regular cloud provider VMs.
The driver creates a Kubernetes Pod running a Docker server for each environment (herby the name "single Pod"). Preevy then connects to the Docker server to build and run your services, just like it does on a regular VM.
Your services are still exposed using the Preevy Tunnel Service - there's no need to configure a specific ingress.
- A Kubernetes cluster should be accessible from the CLI. A
kubeconfigfile may be specified - see flags below.
- When using RBAC authorization, the default context needs permissions for running exec and port-forward for specific pods in the configured namespace.
kubectltool needs to be installed and available in the PATH.
- By default, the driver runs a Pod with
privileged: truesecurity context. In some cases, this requirement may be lifted by customizing the deployment template, see below.
|Kubernetes namespace to provision resources in|
|path to a |
|default template||path to a nunjacks template used to provision Kubernetes resources per environment. See below for details|
|true||if true, provision resources using server-side apply, else using client-side apply (CREATE/PATCH). Applies to |
Similar to other drivers, options are saved in the Preevy profile to be used as default values for all operations.
Options can be overridden for a specific compose file by adding them to the
Options can also be overridden using a CLI flag per command execution:
preevy up --kube-pod-namespace=other-namespace
Customizing the provisioned Kubernetes resources
It's possible to customize the Kubernetes resources provisioned by the driver per environment. Use cases include, but are not limited to:
- Customize the Docker Server image or the Docker Server configuration
- Add labels/annotations to the provisioned resources, e.g, for selecting specific Kuebernetes nodes
- Provisioning additional resources per environment, e.g, a database server
The resources are specified as Kubernetes object specs in a single YAML file rendered from a nunjucks template. The template file may contain multiple definitions separated by lines containing
--- (three dashes).
Start by copying the default template. To use the custom template, enter the path to the custom template file at the
preevy init or
preevy profile create command, or specify the
--kube-pod-template flag for the
preevy up and
preevy down commands. The template file at the specified path needs to be accessible at runtime to the CLI1.
Requirements for the provisioned resources
All resources need to be deployed in a single namepsace, specified as a template argument (see below).
While multiple Kubernetes Deployment objects may be defined, exactly one Deployment must have the label
- The status of the Deployment is used to determine whether the Preevy environment is ready.
- The first container of the Deployment spec is used for copying files, and so it must have the
A Docker server must be listening on port 2375 of the Deployment's Pod. As Preevy uses the port-forward API to connect to the Docker server, it does not need to be exposed as a service. For the same reason, TLS is not supported and needs to be disabled for this port.
The Docker server must also be listening on the unix socket path
/var/run/docker.sock - this is used by the Preevy agent service running alongside your services.
The lifecycle of all resources is tied to a Preevy environment - they will be created and deleted with the environment.
The following arguments are specified when rendering the template:
namespace: the Kuberentes namespace saved in the Preevy profile or specified in the
--kube-pod-namespaceflag. All resources must be defined in this namespace.
id: A generated ID for this environment, 53 characters or less, comprised of the Preevy environment ID and a random suffix.
idcan be used as part for of a label value, with up to 10 additional characters as to not exceed the 63 character limit for labels
Configuring rootless unprivileged Docker-in-Docker
By default, the Kubernetes Docker-in-Docker driver creates a Deployment which runs the
docker:dind image. Traditionally, running Docker inside a container requires the
privileged: true security context, which may be a security concern.
Sysbox is an OSS project (acquired by Docker) which allows running unprivileged containers in a Kubernetes cluster. It can be installed on most of the popular Kubernetes distros including managed cloud platforms like Amazon EKS, Google GKE and Azure AKA. Once installed, a custom template can be used to provision Pods without the
privileged security context.
- Embedding the template in the profile, or specifying its path in the
x-preevysection of the Docker Compose file is in the roadmap, but not implemented yet.↩