Kubernetes Secrets were, in the artchitect's view, not secure enough, and storing sensitive information in both Vault and Kubernetes created unnecessary duplication. On paper, the idea looked clean and consistent. In practice, it ignited a debate about how Kubernetes actually operates.

The request dropping Kubernetes secrets left the engineer unsure whether the zero-Secrets vision was even achievable or whether tools such as External Secrets Operator, Sealed Secrets, or the Secrets Store CSI driver were more realistic options

The team operating the platform had already removed Secrets for internal applications. The problems appeared once they reached components they did not control. Operators, controllers, and many Helm charts rely on Kubernetes Secrets as part of their core design. This left the engineer unsure whether the zero-Secrets vision was even achievable or whether tools such as External Secrets Operator, Sealed Secrets, or the Secrets Store CSI driver were more realistic options.

This question reflects a familiar discussions seen across large Kubernetes deployments. On one side is the desire to establish a unified security model using a Secrets Manager as Vault to be the sole authority. On the other side is the practical reality that most of the Kubernetes ecosystem is built around Secrets. When I first began working with Kubernetes platform, the handling of sensitive data was always one of the challenges. I tested many open-source tool available, and while each one offered its own style of abstraction or workflow, they all target the same problem: Kubernetes workloads expect Secrets, and many tools embed this assumption deep into their logic.

Take cert-manager as one example: it stores generated TLS certificates as Secrets. PostgreSQL operators such as CloudNativePG store database credentials the same way. Continuous delivery tools like Argo CD keep repository access details in Secrets. Service mesh components depend on Secrets for mTLS. Removing these objects means rewriting, forking, or wrapping a large portion of the ecosystem. Any attempt to erase Secrets completely forces the organization into complex maintenance mode, where every chart upgrade or operator release becomes a significant engineering task. Some teams  could consider building custom operators to bridge this gap, but that level of investment rarely aligns with the value gained.

The argument that Kubernetes Secrets lack security also requires more nuance. Secrets can be protected with encryption at rest. In the major cloud distributions, enabling this requires only minimal configuration, and once it is active, Secrets inside etcd are secured with strong encryption. Without this feature, Secrets are only Base64-encoded, which offers no protection. With it active, the risk model shifts away from the storage layer and toward access control.

“The difference between a Secret volume and Vault-injected data becomes less meaningful when the surrounding controls are broad."

And that brings us to the factor that matters just as much as encryption: RBAC. Misconfigured permissions create a dangerous exposure. Anyone who can read Secrets usually can also read pods, where credentials appear in plain form once an application is running. If Vault Agent is writing secrets to the filesystem inside a pod, the contents are just as accessible to anyone with the ability to exec into that container. The difference between a Secret volume and Vault-injected data becomes less meaningful when the surrounding controls are broad.

Vault and most of the secrets manager provides significant advantages: centralized audit logs, secret rotation, dynamic credentials with limited lifetimes, and fine-grained access policies. But none of those advantages change the fact that secrets eventually appear inside a pod, and once they do, the protection depends on RBAC, workload isolation, and network boundaries.

“If RBAC is weak, the method of secret delivery becomes secondary.”

A more grounded approach is to acknowledge that some Secrets will always exist in the cluster, especially for platform components. Vault can remain the central source of truth, and operators like External Secrets Operator can sync Vault data into the Secret objects required by the ecosystem. This avoids the maintenance overhead of rewriting or patching third-party tools while still ensuring that Vault controls the authoritative version of every credential. Combined with encryption at rest, careful RBAC, and strict administrative access patterns, this model provides strong security without working against Kubernetes itself.

One production environment I worked with implemented short-lived administrative permissions. Engineers held no permanent cluster-admin rights. Instead, they requested temporary elevated access only when needed, and the system automatically revoked it after a short period. This design reduced the chance of broad Secret exposure and forced teams to handle sensitive operations with more discipline. There are many ways to achieve a similar result, but the principle is consistent: even the most advanced secret management strategy weakens if the surrounding permissions are loose.

When considering a Vault-only design, the better question is not whether Kubernetes Secrets should disappear, but whether the risks they introduce can be managed effectively with encryption, RBAC, and workload isolation. The zero-Secrets vision may appear straightforward, yet it often overlooks the operational and architectural layers that make Kubernetes successful. A balanced model — Vault as the source of truth, ESO as the bridge, and Kubernetes Secrets as the caching mechanism required by controllers — tends to deliver stronger long-term stability.

Related Posts

You can find some posts and tutorials under the Secrets category.

You may find this article interesting as well: https://devoriales.com/post/150/kubernetes-rbac-and-admission-controllers

This post is part of the Debate Series on devoriales.com, where I share my perspective on topics that often spark strong opinions in the tech community. My view is not the final answer, and it shouldn’t be treated as such. The goal is to exchange ideas, question assumptions, and learn from each other. If you see something differently or have experience that adds another angle, I welcome your thoughts. These discussions shape how we grow, and my own opinions may shift as the conversations continue.