Running your workloads on Kubernetes(k8s) is becoming a de facto standard for many companies building software. The usual steps involve building software, running tests, building a container, hopefully doing some security scanning in the process, pushing to the registry, and deploying.
In general, there are two different risks with k8s images deployed:
- Vulnerabilities in Operating System packages and the software packages
- Tampered or deliberately deployed images.
Many of the workloads running on your cluster didn’t pass through your security tooling in your CI/CD pipelines. Tools directly installed using helm with publicly available container images. These usually have a high degree of vulnerabilities that are hidden away from you.
Also, software that is not deployed all the time can amass vulnerabilities over time. While code doesn’t change, people keep finding vulnerabilities in existing software.
There are four stages of kubernetes supply chain security:
- CI/CD: scan your images and dependencies
- Container registry scanning
- Workload scanning while running and deployed in the cluster
- Admission control only allows images based on an allow list and valid signatures.
We have a introduction to K8S security if you want to know more.
CI/CD image scanning
Tools like Trivy can detect vulnerabilities in the container image. Next to the operating system issues, it can also detect problems with installed libraries of your software and secrets might have accidentally been added.
This will catch issues at the time of the build and prevent potentially shipping new vulnerabilities.
Container registry scanning
Registries like the ones offered by Google and AWS have the ability to scan existing images. This can make sense, but it usually is a dump yard that many people never look into. If set up properly with alerting this can be useful. However, in my experience these are not on the radar of many teams.
Workload Scanning
Scanning and alerting on your currently running images helps you understand your current problems. It also catches container images that did not pass through your regular CI/CD pipelines or your registry.
Tools like Trivy operator and Clair can support this. They are regularly scanning your running containers and will emit alerts for you. This adds extra load to your system but allows for the most accurate representation of vulnerabilities in your clusters.
Admission Control
By controlling what can run on your cluster, you enhance an extra level of security.
Having an allowed list of images will reduce attack vectors by quite a bit. If, on top if this, you verify the signatures of the images you will get even more peace of mind. Docker Content Trust and Cosign by sigstore are currently the best supported.
Need help with securing your Kubernetes deployments? Lets chat: andy@occamslabs.com
Andreas likes security, enabling teams to ship secure products and coffee.