Credential Vending in Apache Polaris: Securing Data Access Without Sharing Keys

By Prithvi S – Staff Software Engineer at Cloudera

Introduction

In modern data architectures, managing who can access what data is a constant challenge. Traditional approaches rely on long‑lived access keys or service accounts that are difficult to rotate and can become a security liability. Apache Polaris tackles this problem head‑on with a built‑in credential vending mechanism. Instead of distributing static keys, Polaris mints short‑lived, scoped credentials on demand, giving each request exactly the permissions it needs and expiring them after a few minutes.

This post walks through the design, implementation, and benefits of credential vending in Polaris. It also shows how the feature integrates with the rest of the system, discusses best practices, and provides a practical example of using the API.

Why Credential Vending?

Data engineers and scientists often need to read or write to cloud storage (S3, GCS, Azure) as part of their pipelines. Giving them permanent access keys creates several problems:

• Key leakage – a single compromised key can expose an entire bucket.
• Rotation overhead – keys must be rotated regularly, which is operationally heavy.
• Principle of least privilege – static keys usually have broad permissions, violating least‑privilege best practices.

Credential vending solves these issues by generating short‑lived, scoped tokens that are tied to a specific operation (read‑only, read‑write) and a narrow resource path. Tokens expire after a configurable period (default ~15 minutes) and can be revoked instantly if needed.

Architecture Overview

Below is a high‑level diagram of the credential vending flow (illustrated with a professional image from Unsplash – placeholder):

1. Client Request – An engine (Spark, Flink, Trino) sends an HTTP request to Polaris to perform an action on a table.
2. Auth Check – Polaris authorizes the request using its two‑tier RBAC model.
3. Storage Lookup – The system determines which cloud storage backend backs the catalog (S3, GCS, Azure).
4. Credential Minting – Polaris calls the cloud provider’s token service (AWS STS, GCS token API, Azure AD) to create a temporary token with the exact permissions required.
5. Response – The temporary credential is returned to the client, which uses it for the subsequent data operation.

Deep Dive: How Polaris Mints Credentials

1. Authorization Layer

Polaris first evaluates the request against its RBAC model. The model consists of:

• Principal Roles – assigned to users, service accounts, or automated agents.
• Catalog Roles – define privileges on catalog objects (e.g., TABLE_READ_DATA, TABLE_WRITE_DATA).
• PolarisAuthorizer – resolves the effective privileges for the request.

Only if the request has the required privilege does Polaris proceed to credential vending.

2. Storage Integration

Polaris supports three major cloud storage providers via the PolarisStorageIntegration interface. Each implementation knows how to:

• Translate a credential scope (e.g., s3://my-bucket/path/) into a provider‑specific request.
• Call the provider’s temporary credential service.
• Apply any additional constraints (IP allow‑list, expiration window).

AWS Example

AssumeRoleRequest req = AssumeRoleRequest.builder()
    .roleArn(storageConfig.getAwsRoleArn())
    .durationSeconds(900) // 15 minutes
    .policy(scopedPolicy) // restrict to specific bucket/prefix
    .build();
Credentials creds = stsClient.assumeRole(req).credentials();

GCS Example

GoogleCredentials scoped = GoogleCredentials.createFromSecret(
    storageConfig.getServiceAccountJson())
    .createScoped(List.of("https://www.googleapis.com/auth/devstorage.read_write"))
    .createDelegated(storageConfig.getServiceAccountEmail());
AccessToken token = scoped.refreshAccessToken();

3. Token Construction and Caching

After receiving the provider token, Polaris wraps it in a PolarisCredential object that includes:

• Provider name (aws, gcs, azure)
• Expiration timestamp
• Scoped resource path
• Original request ID for tracing

Polaris also caches tokens for a short window to reduce provider calls when identical scopes are requested repeatedly.

Benefits in Real‑World Deployments

Practical Example: Reading a Table from Spark

import org.apache.polaris.client.PolarisClient
import org.apache.spark.sql.SparkSession

val polaris = PolarisClient.builder()
  .endpoint("https://polaris.mycompany.com")
  .authToken("Bearer <user‑jwt>")
  .build()

val cred = polaris.getTemporaryCredential(
  catalog = "analytics",
  namespace = "sales",
  table = "transactions",
  privilege = "TABLE_READ_DATA"
)

// Spark can now read directly using the temporary S3 credentials
val df = SparkSession.builder()
  .appName("PolarisDemo")
  .getOrCreate()

df.read
  .format("iceberg")
  .option("fs.s3a.access.key", cred.accessKey)
  .option("fs.s3a.secret.key", cred.secretKey)
  .option("fs.s3a.session.token", cred.sessionToken)
  .load("s3://my‑bucket/analytics/sales/transactions")

df.show()

The Spark job never sees a permanent AWS key; it receives a scoped token that expires after 15 minutes.

Best Practices for Using Credential Vending

1. Limit Scope Aggressively – Include the bucket and prefix that the request truly needs.
2. Set Short Expiration – Default of 5‑15 minutes is usually sufficient for a data pipeline step.
3. Cache Wisely – Enable short‑term caching to reduce provider latency, but ensure cache invalidation on role changes.
4. Monitor Token Usage – Polaris logs each token issuance; integrate with your observability stack to detect anomalies.
5. Rotate Underlying IAM Roles – Even though tokens are short‑lived, the underlying IAM role should be rotated periodically.

Image Gallery

• Credential Vending Diagram – Visualizes the flow from request to temporary token (placeholder image URL).
• Polaris Dashboard Screenshot – Shows the token issuance metrics in the admin UI (placeholder image URL).

Conclusion

Apache Polaris’ credential vending mechanism provides a modern, secure alternative to static access keys. By issuing short‑lived, scoped tokens on demand, Polaris reduces the attack surface, simplifies compliance, and aligns with the principle of least privilege. As data pipelines continue to scale and integrate with multiple cloud providers, such dynamic credential management becomes a cornerstone of a robust data governance strategy.

If you want to try it yourself, check out the Polaris GitHub repository and the official documentation. Feel free to reach out with questions or share your own experiences – secure data access is a community effort.

Author Bio: I'm Prithvi S, Staff Software Engineer at Cloudera and Open‑source Enthusiast. Follow my work on GitHub: https://github.com/iprithv