A newly created access token may be rejected, fail intermittently, or take up to ~60 seconds to become usable when sent to a federated JPD other than the one that issued it. Federation is working — the token does sync — but the lag surprises CI/CD scripts and automation that create-and-use a token in the same step.
Symptoms
-
Access token created on JPD A (source), used on JPD B (target, federated to A) within seconds
-
Target rejects with 401 and TOKEN_VERIFICATION_FAILED: revoked for the first 40–60 seconds, then starts working
-
Retrying the same operation a minute later succeeds — making it look like an intermittent issue
-
Manual inspection confirms the token did arrive on the target — the rejection is coming from a stale cache, not a missing record
Why this happens
The visible delay is two independent delays stacked on top of each other.
1. Federation buffer on the source (default 30s)
Federation outbound events are batched on the source. A token-create event sits in the buffer until either the buffer hits its size cap or the buffer-wait-millis timer expires (default 30000ms). Since token creates typically arrive one at a time, the timer is usually what fires — the token isn't sent to the target until ~30s after creation.
2. Negative verification cache on the target (default 60s)
When a request hits the target before federation has delivered the token, the target Artifactory asks Access service to look the token up. Access returns NOT_FOUND, and the result is cached as REVOKED for 60 seconds on that Artifactory node. Subsequent requests to the same node with the same token are served from cache — no re-query to Access, even after federation has written the token to the database.
The federation write does not invalidate the verification cache. The cache has to expire naturally.
The cache is per-node. Behind a load balancer, two consecutive requests can land on differently-cached nodes — one with a poisoned cache, one without — which is why the failure pattern often reads as intermittent rather than a clean ~60-second wait.
Why the visible delay is ~60s, not ~90s
The two delays run in parallel, not in series. Federation needs up to ~30s to deliver the token to the target's database; the cache needs 60s to expire — both clocks ticking concurrently. The visible delay is whichever one finishes last. With defaults that's the cache, which is why lowering the cache TTL has the largest immediate impact — until the TTL drops below the federation delay, at which point federation takes over as the bottleneck.
Why Circle of Trust doesn't help here
A common assumption: "We use CoT and non-revocable tokens, so they should just work across JPDs without federation." That's correct — but only for JWT tokens. CoT validates a JWT's signature locally against the root certificate; no database lookup, no federation dependency.
Reference tokens are different. They're identifier strings (recognizable by the cmVmdGtu... prefix — base64 for reftkn) that must be resolved to the underlying record through a database lookup. There is no signature to validate locally, so CoT cannot apply. If the token isn't in the target's database yet, the lookup fails and the negative cache trap fires.
The following are all reference tokens and therefore subject to the delay:
-
Tokens generated via the UI's User Profile → Identity Token page
-
Tokens generated by a repo’s Set Me Up
-
Any token created via include_reference_token=true
The official docs call this out: reference tokens and UI-generated tokens are not supported in a Circle of Trust.
What to do
Listed in order of effectiveness — the client-side wait is typically the most reliable fix.
1. Add a wait in the client between token creation and first use
If the client creates a token and immediately uses it on a different JPD, the first request poisons the cache before federation has delivered the token. Sleeping N seconds between creation and first use (where N is the source's buffer-wait-millis + ~5s for pipeline overhead) lets the token arrive at the target before the lookup happens — the verification succeeds and a positive result is cached. The negative-cache trap never fires.
This is the most reliable fix and works regardless of cluster-side tuning.
2. Lower buffer-wait-millis on the source
Reduce the federation batching delay from the 30s default. Set in the source Access federation outbound config (Access Yaml):
federation:
outbound:
buffer-wait-millis: 15000
Trade-off: more frequent (smaller) HTTP POSTs from source to each federated target. Negligible for most environments.
3. Lower the verification cache TTL on the target
Reduce the negative-cache TTL from the 60s default. Set in $JFROG_HOME/artifactory/var/etc/artifactory/artifactory.system.properties on every target node, then restart Artifactory:
artifactory.access.client.token.verify.result.cache.expiry=30
Trade-off: more frequent gRPC calls from Artifactory to Access on cache misses. These are cheap primary-key lookups — minimal overhead. No additional DB writes are introduced; the target still receives the same volume of tokens, just in smaller batches.
Related