Kubernetes Secrets Injection with Fortanix DSM

Security Issues with Kubernetes Secrets

In today’s IT world, Kubernetes (K8s) has become indispensable for many companies due to its huge advantages, such as automated scaling and failover.

This makes it much easier to manage large scale applications, which is important as the number of those applications is growing quickly. But what about security?

These deployments need rely on secrets, like passwords, cryptographic keys, and API keys, to authenticate connections to databases, proxies, application servers, and so forth.

Proper secrets management has become a challenging task. How can these secrets be managed in a secure fashion? This blog post covers the protection of secrets against theft.

Ideally, secrets should be stored in a hardware security module (HSM) that is compliant to FIPS 140-2 Level 3, as software-based key stores have proven to have security limitations.

Secrets governance is also important. We see a rising number of DevOps teams with high labor turnover. Businesses demand quick go-to-market solutions.

This leads to a proliferation of secrets, which are often not properly managed and are sometimes even stored in cleartext on an employee’s laptop. This creates a huge business risk as well as possible compliance violations.

Because of the move to zero trust approaches, the IT market is demanding security mechanisms to prevent of unauthorized access, including role-based access control (RBAC), multi-control, two-factor-authentication, utilization of an existing SSO, and adherence to other principles of least privileges. These security needs are not provided by Kubernetes alone.

The Fortanix Kubernetes Secrets Injection solution answers all these security related questions. It utilizes the Data Security Manager (DSM), which comprises of a FIPS 140-2 Level 3 compliant HSM featuring an inbuilt, web-based Key Management System (KMS).

This offers powerful yet straightforward solutions for enterprises and DevOps teams alike to centralize the storage and management of security artifacts, which are built on these concepts for the Kubernetes control plane:

• KMS Provider
• Admission Controller

The architecture of both is explained in the two following sections.

KMS Provider

Typically, Kubernetes stores all the secrets in its etcd database. But the secrets are not encrypted, they’re only Base64 encoded. Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd.

Under the KMS Provider concept, however, Kubernetes allows the encryption of the secrets in etcd through a plugin. This solution was put into effect in Kubernetes 1.10 and is implemented by the Fortanix KMS Plugin.

Using this method, all secrets are encrypted with Data Encryption Keys (DEK) through the Kubernetes API Server. These DEKs in turn are encrypted by a master key, which resides only in DSM and never leaves it.

When the Kubernetes API Server wants to inject a secret into a Pod, it reaches out to the DSM through the Fortanix KMS Plugin to decrypt the DEKs with the master key. With this DEK, it is able to decrypt the secret.

Figure 1 shows the architecture of the Fortanix KMS Plugin within the Kubernetes KMS Provider concept.

Figure 1: Fortanix KMS Plugin Architecture

This encryption scheme is called an envelope scheme. Fortanix uses the Advanced Encryption Standard (AES) and the AES-GCM algorithm, which is considered to be very secure.

At bootup, the API Server automatically generates the DEKs and encrypts them using the master key.

The master key is generated and managed by the operators within the key lifecycle management inside DSM. The DSM features a user-friendly GUI which facilitates the management of the master key.

The DSM also offers state of the art protection for the master key. For example, a master key rotation can be protected with multi-control through an asynchronous approval workflow. Operator accounts can be protected against misuse with two-factor authentication.

Admission Controller

As described, secrets can be encrypted within the etcd database by using the KMS Provider feature together with the Fortanix KMS plugin. But this still has a few downsides:

Different implementations of the KMS Provider are required for different environments, like Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS) or on-premises deployments.

Secrets are not managed centrally.

Secrets are not protected with RBAC or principle of least privileges.

Secrets reside in clear in the RAM of the Kubernetes API Server.

Luckily, Kubernetes has another element which addresses all these issues, the Admission Controller. The Admission Controller is a 3rd party extension that intercepts pod CREATE and UPDATE requests to the Kubernetes API server, before the pod is created or altered.

The interception occurs after the request to the Kubernetes API Server is authenticated and authorized, however.

Fortanix has implemented a solution based on the Kubernetes Admission Controller. The solution consists of a Mutating Admission Webhook, which, on interception, deploys a sidecar next to the pod.

This sidecar is fed with the credentials to reach out to Fortanix DSM to retrieve the secrets. After receiving them, they are made available to the pod.

The secrets can then be stored in the on the file system or as an environment variable inside the pod. This allows applications to remain unaltered and their dependency on secrets seamlessly switches to Fortanix DSM from legacy repositories.

In addition, this solution is completely K8s platform provider agnostic. So, there is no vendor lock-in.

Figure 2 depicts the architecture of the Fortanix Admission Controller solution.

Figure 2: Fortanix Admission Controller Architecture

The Fortanix Admission Controller supports basic API key authentication, which is simple to implement and allows you to get things working quickly.

But the solution also offers a more sophisticated authentication through JSON Web Tokens (JWT). Using JWT, the credentials to DSM are not stored inside of the application, but the security relies on the established Public Key Infrastructure (PKI) of the control plane.

This provides an additional layer of security and solves the Secret Zero problem.

All secrets are managed by DSM, which provides a unified view of all your security objects, whether they are passwords, PINs, cryptographic keys, certificates, or opaques.

Access to these can be restricted with an elaborated RBAC system, where even custom roles are supported. These custom roles can be created from a myriad of privileges resulting in a very fine-grained access control.

Multi-control is achieved by a very user-friendly quorum policy.

Conclusion

The difficulty of secure secrets injection into a Kubernetes pod is solved by two different Fortanix solutions. The Admission Controller approach, in particular, ticks all the boxes for usability, security, compliance, and flexibility.

Related Read:

Fortanix Confidential Containers on Microsoft Azure Kubernetes Service (AKS)

Microsoft AKS Secrets Injection with Fortanix DSM

For more information, please contact Fortanix.