Skip to content
English
  • There are no suggestions because the search field is empty.

Improving Kargo Warehouse Discovery Performance at Scale

How to speed up slow Warehouse freight discovery when you have many Warehouses subscribed to a shared image registry

Description

As the number of Warehouses in a Kargo instance grows, freight discovery can slow down significantly, in some cases taking 15+ minutes instead of seconds. This is usually caused by a combination of three factors:

  1. Per-registry rate limiting. Kargo applies a per-host rate limit , i.e. 20 requests/second) that is shared across every Warehouse pointed at the same registry host per shard.
  2. Subscription strategy cost. imageSelectionStrategy: NewestBuild is more expensive to evaluate per refresh than SemVer, since it has to retrieve manifests for more candidate tags.
  3. Limited reconcile parallelism. A single Kargo shard processes Warehouse reconciles with a capped concurrency, so a large Warehouse count backs up behind that cap.

Steps to improve discovery time:

  1. Shard your Warehouses across multiple agents. Set .spec.shard on Warehouse manifests (the same mechanism used for Stage sharding) and run multiple self-hosted Kargo agents to process shards in parallel, rather than relying on one agent to reconcile every Warehouse.
  2. Tune concurrency per shard. Increase MAX_CONCURRENT_WAREHOUSE_RECONCILES in the Kargo agent settings. This is most effective when combined with sharding. Raising it on a single shard alone may not help if a per-registry rate limit is the actual bottleneck.
  3. Reduce the cost of each refresh.
    • Prefer imageSelectionStrategy: SemVer over NewestBuild where the registry supports it.
    • If NewestBuild is required, scope down the tags considered per refresh using allowTagsRegexes / ignoreTagsRegexes and discoveryLimit.
  4. Enable cacheByTag (available starting in Kargo v1.9) if your registry's tags are immutable. This caches per-tag metadata so Kargo doesn't re-fetch manifests for tags it has already seen. 

    ⚠️ Only enable this if tags in your registry are truly immutable — if a tag can be overwritten after being pushed, cacheByTag will serve stale results.

Real-world result: one customer running ~145 Warehouses against a single self-hosted registry host reduced tag-discovery latency from a median of ~18 minutes to ~5 minutes by splitting into 4 shards and setting MAX_CONCURRENT_WAREHOUSE_RECONCILES=20 on each.