Handling Vector Database API Key Rotation Without Breaking Deployed Applications

API key rotation is one of those tasks that feels simple in principle and becomes complicated in practice. The goal is straightforward: replace an old credential with a new one. The challenge is doing it without causing a gap in service — the window between when the old key stops working and when the new key is in place.

For vector databases running in production, that window can mean failed queries, broken ingestion pipelines, and errors surfacing to end users. Understanding how to close that window is the difference between rotation being a routine maintenance task and an incident.

Why Rotation Matters and When to Do It

API keys are long-lived credentials. Unlike session tokens that expire after minutes or hours, API keys often remain valid for months or years unless explicitly rotated. That longevity is convenient but creates risk: a key that has existed long enough has probably appeared in more places than you intended — log files, deployment artifacts, shell histories, chat messages.

Rotation is the mechanism for resetting that exposure. The general rule is to rotate in three situations:

On a schedule. Regular rotation limits the useful lifetime of any compromised credential. Even if a key has leaked somewhere you do not know about, rotation ensures it stops working before it can cause significant damage. Quarterly or annual rotation is common depending on the sensitivity of the data.

On personnel changes. When a developer who had access to a key leaves the team or organization, their copy of that key should stop working. Rotating the key achieves this even if you do not know exactly where they stored it.

After a suspected or confirmed exposure. If a key appears in a public repository, a log, a support ticket, or any other unintended location, rotate it immediately without waiting.

The Fundamental Pre-Condition: No Hard-Coded Keys

Key rotation only works cleanly if the application reads its credentials from the environment rather than having them embedded in source code or configuration files.

When a key is hard-coded, rotation requires a code change — edit the file, test the change, deploy a new version. That process takes time, introduces risk, and typically creates a gap where the old key has been rotated but the new code is not yet deployed.

When a key is stored as an environment variable, rotation requires updating one value in the environment and restarting or reloading the application. The code itself does not change. The deployment artifact does not change. The gap between old key and new key is limited to the restart time of the application.

This is why environment variable storage is not just a best practice for security — it is a practical requirement for making rotation viable.

How Key Rotation Works in Managed Services

Most managed vector database platforms provide two ways to rotate a key: through the console UI and through an API.

Console rotation is the manual path. You navigate to the key management section, click rotate on the relevant key, copy the new value, and update the environment variable in your deployment. The old key is invalidated immediately on rotation. This approach works well for occasional rotation or for keys that are used by a small number of services.

API rotation is the programmatic path. You call an endpoint or SDK method that invalidates the old key and returns the new one. This is useful when rotation needs to be part of an automated workflow — a CI/CD pipeline, a scheduled job, or a security automation tool. The new key can be written directly to a secret management system and propagated to services without any manual step.

The right choice depends on your operational model. Teams that manage infrastructure as code and use secret management tooling will want the API. Teams doing occasional manual maintenance will find the console sufficient.

The Kubernetes Restart Problem

One failure mode that catches teams running vector databases in Kubernetes is the disconnect between updating a secret and restarting the pods that use it.

When you update a Kubernetes secret — for example, by changing the API key value in a Helm values file and running helm upgrade — the secret value in the cluster is updated, but running pods continue to use the environment variables that were injected at startup. They do not automatically pick up the new secret value. The application is still using the old key, which has now been rotated and invalidated, and the result is authentication failures.

The fix is to explicitly restart the affected pods after updating the secret, which forces them to start fresh with the new environment values. This is a known quirk of how Kubernetes handles secret propagation, and it applies to any credential managed through environment variables in a Kubernetes deployment, not just vector database keys.

Being aware of this prevents a confusing scenario where the key has been correctly rotated and the secret correctly updated, but the application is still failing because the pods are running with stale environment state.

Zero-Downtime Rotation for Production

The cleanest rotation for production environments is one where the old key continues to work while the new key is being distributed to all services, with a brief overlap period before the old key is invalidated. This eliminates the gap entirely.

Not all platforms support this natively — some only support immediate invalidation of the old key on rotation. When immediate invalidation is the only option, the safest approach is to rotate during low-traffic periods and have rollback steps prepared in case the new key does not propagate in time.

Platforms that support per-user or per-service keys (rather than a single shared key) make rotation significantly less risky, because you are rotating one service’s credential rather than a shared credential that all services depend on simultaneously.

Weaviate: Safe Key Rotation in Practice

Here is how each of these patterns maps to Weaviate.

Programmatic Rotation via the User Management API

For clusters running Weaviate v1.30 or later, the database user management API provides zero-downtime rotation — no cluster restart required:

new_api_key = client.users.db.rotate_key(user_id="custom-user")
print(new_api_key)

This immediately invalidates the old key for that user and generates a new one. Because the key is read from an environment variable in your application, you update the variable and redeploy or restart the application — Weaviate itself is not restarted and continues serving requests throughout.

Console Rotation for Weaviate Cloud

For Weaviate Cloud deployments, rotation is also available directly in the console:

1. Open the console and select your cluster
2. Go to Cluster details → API Keys
3. Click Rotate next to the key you want to cycle
4. Confirm — the old key is invalidated immediately
5. Copy the new key before closing the dialog — it will not be shown again
6. Update the environment variable in your deployed application

Environment Variable Setup

The pre-condition for safe rotation in either case:

export WEAVIATE_API_KEY="your-new-rotated-key"

import os
import weaviate
from weaviate.classes.init import Auth

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.environ["WEAVIATE_URL"],
    auth_credentials=Auth.api_key(os.environ["WEAVIATE_API_KEY"]),
)

The Kubernetes Restart Step

If your application runs in Kubernetes and reads the key from a Helm-managed environment variable, updating the values file and running helm upgrade updates the secret but does not restart the pods. You need to trigger the rollout explicitly:

kubectl rollout restart statefulset/weaviate -n weaviate

Run this after the secret is updated and before verifying that the new key is working.

Verifying the Rotation

After completing the rotation and updating the environment, confirm the application connects correctly:

print(client.is_ready())  # Should return True

If it returns False or raises an authentication error, the most likely causes are that the environment variable was not updated in all deployment targets, or that pods were not restarted after a Kubernetes secret update.

Summary of Safe Rotation Steps