API Gateway Secrets Management: Verify Credential Revocation

API gateway secrets management is incomplete until you can prove that a revoked credential no longer works. Storing a key in a vault reduces exposure; rotating it provides a replacement. Neither action, by itself, proves that the old key has stopped authenticating at the gateway, an upstream provider, or another route.

For a suspected leak, revoke at the credential's issuer, update dependent workloads, and test the old and replacement credentials at their respective boundaries. Removing a leaked string from Git history is not revocation.

This guide provides an inventory, a bounded Apache APISIX 3.18.0 caller-key drill, and an acceptance record. The drill uses the Admin API and standalone key-auth authentication, without anonymous fallback or multi-auth. It is not a claim about AISIX's caller-key model, OAuth token revocation, or automatic propagation times.

The gateway operations follow the version-pinned APISIX 3.18.0 Admin API documentation and key-auth guide. The Python probe was executed against a local HTTP fixture. We did not execute the gateway drill on an APISIX deployment; its expected statuses come from those documented contracts. No cluster propagation measurement is reported.

Separate the Credentials Before Rotating Them

A gateway deployment normally has several credential issuers:

For each credential, record the owner, issuer, consumer applications, storage/reference locations, last rotation, revocation action, and verifier endpoints. Track credential identifiers or internal secret-version references, not raw values.

Recent Kong Token Vault coverage highlights credential custody for agents. The operational question applies more broadly: which component can still use the old credential after custody or configuration changes? The drill below answers that narrower question for APISIX caller authentication; it does not reproduce Kong's feature.

Keep Storage, Rotation, and Revocation Distinct

Use a secret manager or approved injection mechanism to keep credentials out of repositories, prompts, images, and routine logs. The APISIX key-auth documentation describes supported secret references; configure and test the backend appropriate to your deployment. Do not assume that updating a secret manager invalidates copies already accepted by another service.

For a planned rotation, a short, controlled overlap can keep legitimate clients working:

1. Create a replacement credential.
2. Verify it at the intended endpoints.
3. Move clients to it.
4. Revoke the old credential.
5. Prove old-key rejection and replacement-key continuity.

For a known compromise, the overlap may be unacceptable. Revoke the exposed credential promptly and use the incident process for service recovery. Do not keep an attacker-valid key alive merely to achieve a smooth rollout.

The OWASP secrets lifecycle guidance is useful for the surrounding ownership and recovery process. A gateway is one enforcement boundary; it does not secure a compromised package-publishing system or revoke a provider credential on the provider's behalf.

Prepare an Isolated APISIX Drill

Use an existing non-production APISIX 3.18.0 instance with:

• Admin API access through a private, trusted management path;
• the key-auth plugin enabled and no anonymous fallback on the test route;
• curl 7.76 or later, Python 3, and one approved administrative key;
• a harmless HTTP upstream reachable from the APISIX data plane;
• a list of the actual data-plane endpoints you intend to verify.

These examples create only revocation-drill test resources. Check that the names are unused. Do not overwrite an existing consumer or route.

Set APISIX_ADMIN to the origin plus /apisix/admin, with no trailing slash. Set APISIX_ADMIN_KEY through your approved secret injection method. On a default local test deployment, the Admin API base is http://127.0.0.1:9180/apisix/admin.

A tiny upstream lets you verify both connectivity and header removal. Save as mock_upstream.py and run python3 mock_upstream.py on the same host/network namespace as a local APISIX instance:

import json
from http.server import BaseHTTPRequestHandler, HTTPServer

class Probe(BaseHTTPRequestHandler):
    def do_GET(self):
        body = json.dumps({
            "probe_version": 1,
            "credential_seen": self.headers.get("apikey") is not None
        }).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *_):
        pass

HTTPServer(("127.0.0.1", 19880), Probe).serve_forever()

For containerized APISIX, loopback means the container itself. Put the fixture in the same private test network and substitute its reachable service name for 127.0.0.1:19880 below. Do not expose the fixture on a public interface.

Generate two disposable keys and their Admin API documents. The restrictive directory and file modes keep these fixtures out of routine shared files:

umask 077
mkdir revocation-drill
cd revocation-drill
python3 - <<'PY'
import json
import secrets
from pathlib import Path

Path("consumer.json").write_text(json.dumps({
    "username": "revocation-drill"
}))
for label in ("old", "new"):
    key = secrets.token_urlsafe(32)
    Path(f"{label}.key").write_text(key)
    Path(f"{label}.json").write_text(json.dumps({
        "plugins": {"key-auth": {"key": key}}
    }))
PY

Do not print or commit these generated files. They are test credentials, not credential-storage recommendations for production.

Create the consumer and two distinct credentials using the Admin API’s Consumer and Credential operations:

admin_api() {
  printf 'header = "X-API-KEY: %s"\n' "${APISIX_ADMIN_KEY:?}" |
    curl --fail-with-body -sS --config - "$@"
}

admin_api -X PUT "${APISIX_ADMIN}/consumers" \
  -H "Content-Type: application/json" \
  --data-binary @consumer.json > consumer-result.json

for label in old new; do
  admin_api -X PUT \
    "${APISIX_ADMIN}/consumers/revocation-drill/credentials/drill-${label}" \
    -H "Content-Type: application/json" \
    --data-binary @"${label}.json" > "${label}-result.json"
done

Admin responses can contain configuration secrets, so keep the result files under the same restrictive permissions and sanitize them before sharing.

Create the route using the documented Route resource and key-auth configuration. This JSON assumes the mock upstream is in APISIX's own host/network namespace:

cat > route.json <<'JSON'
{
  "uri": "/credential-drill",
  "plugins": {
    "key-auth": {"hide_credentials": true}
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": {"127.0.0.1:19880": 1}
  }
}
JSON

admin_api -X PUT \
  "${APISIX_ADMIN}/routes/revocation-drill" \
  -H "Content-Type: application/json" \
  --data-binary @route.json > route-result.json

Here hide_credentials removes the credential selected for authentication from the upstream request. It is not a promise to redact gateway logs or every duplicate location carrying the same secret. Send the key in the apikey header, not in a URL query string.

Test Before and After Revoking the Old Credential

Save this probe as revoke_probe.py inside the same directory:

import json
import os
import sys
import time
from pathlib import Path
from urllib.error import HTTPError
from urllib.request import HTTPRedirectHandler, Request, build_opener

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, *_):
        return None

opener = build_opener(NoRedirect())

phase = sys.argv[1]
if phase not in ("before", "after"):
    raise SystemExit("Use before or after")
targets = os.environ["APISIX_TARGETS"].split(",")
keys = {name: Path(f"{name}.key").read_text().strip()
        for name in ("old", "new")}
expected = {"old": 200 if phase == "before" else 401,
            "new": 200, "missing": 401}
failed = False
for round_no in range(1, 4):
    for target in targets:
        for label, wanted in expected.items():
            headers = {"apikey": keys[label]} if label in keys else {}
            request = Request(target.strip(), headers=headers)
            try:
                with opener.open(request, timeout=5) as response:
                    status = response.status
                    body = response.read()
            except HTTPError as error:
                status = error.code
                body = error.read()
            except OSError:
                print(round_no, target.strip(), label, "TRANSPORT_ERROR")
                failed = True
                continue
            ok = status == wanted
            if status == 200:
                try:
                    payload = json.loads(body)
                    ok = ok and payload.get("probe_version") == 1
                    ok = ok and payload.get("credential_seen") is False
                except (ValueError, AttributeError):
                    ok = False
            print(round_no, target.strip(), label, status,
                  "PASS" if ok else "FAIL")
            failed |= not ok
    time.sleep(0.5)
raise SystemExit(1 if failed else 0)

Use one test-route URL per intended data-plane endpoint. For the local fixture:

export APISIX_TARGETS="http://127.0.0.1:9080/credential-drill"
python3 revoke_probe.py before

For each target and each round, the old and new keys should return 200, with the fixture marker and credential_seen: false; a request without a key should return 401. Correct baseline failures before proceeding.

Now delete only the old credential:

admin_api -X DELETE \
  "${APISIX_ADMIN}/consumers/revocation-drill/credentials/drill-old" > revoke-result.json

python3 revoke_probe.py after

The expected post-revocation observations are old key 401, replacement key 200, and missing key 401. These are acceptance expectations, not measured propagation times for your cluster.

Save the administrative acknowledgement time and the time of each probe. If a node still accepts the old key, retain that failure, investigate synchronization and route selection, and rerun. Do not turn “failed once, passed after retry” into an undocumented immediate-revocation guarantee.

Redirects are not followed, so the probe does not forward a key to a redirected endpoint. A redirect is a failed observation to investigate. The probe checks three short rounds. Extend the observation interval according to your deployment's configuration delivery and failure modes. An undiscovered replica or bypass route is outside that sample.

Diagnose a Failed Revocation Check

The pinned key-auth implementation shows why the route assumptions matter: missing or invalid credentials return 401 on the ordinary path, while an explicitly configured anonymous consumer can take a different path.

Do not generalize these statuses to an OAuth server, JWT policy, AISIX caller key, or a route using multi-auth. Give each mechanism its own acceptance test.

Complete the Incident Record and Clean Up

A useful revocation record contains no raw secret:

Review gateway, upstream, CI, APM, and application logs for credential exposure under your approved incident-handling process. hide_credentials does not retroactively clean stored logs. Preserve forensic evidence according to that process rather than casually deleting it.

After a successful drill, the old credential is already gone. Remove the remaining disposable resources:

admin_api -X DELETE "${APISIX_ADMIN}/routes/revocation-drill" > /dev/null
admin_api -X DELETE \
  "${APISIX_ADMIN}/consumers/revocation-drill/credentials/drill-new" > /dev/null
admin_api -X DELETE "${APISIX_ADMIN}/consumers/revocation-drill" > /dev/null

If you stopped earlier, remove either test credential that still exists before deleting the consumer. Stop the mock server. Remove the generated key and Admin-response files from the test directory after any required sanitized evidence is retained. Never reuse the old key as a rollback.

Start with one non-production route and record actual observations. Expand the verifier inventory before applying the procedure to a live incident. For a broader test inventory, use the API gateway security-scanning guide; for a specific leak, the issuer's revocation procedure and your incident owner remain the authority.