Skip to content
English
  • There are no suggestions because the search field is empty.

Clusters Showing “❓” (Unknown State) After Installing Akuity Agent via API-Generated Manifests

Resolving Cluster Health Discrepancies Between API-Fetched and UI-Generated Akuity Agent Manifests

When installing the Akuity Agent through CI/CD pipelines using manifests retrieved from the Akuity API, users may observe that clusters remain in an unknown state (❓) on the Akuity Cloud Console instead of a healthy state (💚). This behavior leads Argo CD to believe there are no clusters available for application deployments.

Symptoms:

  • Clusters appear with a ❓ icon in the Akuity Cloud Console.

  • Argo CD reports no available clusters for deployments.

  • Installing the same agent manifest manually via the Cloud Console UI → Clusters → Install results in a healthy (💚) cluster.

  • Comparing API-fetched and UI-fetched manifests reveals missing resources such as: statefulset.apps/argocd-redis-ha-server

  • The agent logs may show connection-related messages such as:

    websocket: bad handshake

    client: Give up

Root Cause:

This issue arises due to a mismatch in the generated manifest between the API endpoint and the Cloud Console UI.

The API endpoint:

 
https://akuity.cloud/api/v1/orgs/{org_id}/argocd/instances/{instance_id}/clusters/{cluster_id}/manifests

returns the same manifest as the UI under normal conditions, but in some cases:

  • Certain feature flags (e.g., repo-server delegation, redis-ha) are applied only when the organization or instance context is set accordingly.

  • The UI “Install” workflow may dynamically include feature-specific resources (like the Redis HA StatefulSet) that the API version omits if not requested under the same feature context.

  • Missing these resources may prevent the agent from completing its bootstrap process and establishing a tunnel to the control plane, resulting in an unknown cluster status.

Expected Behavior:

Both the API and UI should return identical manifests.
By default:

  • The statefulset.apps/argocd-redis-ha-server resource should be present but scaled to 0 unless repo-server delegation is explicitly enabled.

Resolution Steps:

  1. Verify API and UI Manifests Match 

    curl -s -H "Authorization: Bearer $TOKEN" \
    "https://akuity.cloud/api/v1/orgs/$ORG_ID/argocd/instances/$INSTANCE_ID/clusters/$CLUSTER_ID/manifests" \
    -o api_manifest.yaml

     

  2. Then, in the Akuity Cloud Console:

    • Go to Clusters → [Cluster Name] → Install

    • Copy the manifest and save as ui_manifest.yaml

    • Compare: 

      diff -u api_manifest.yaml ui_manifest.yaml

  3. If Differences Exist:

    • Check if repo-server delegation or related features are enabled for the instance.

    • Confirm whether any feature flags have been added recently that alter generated manifests.

    • Reapply the manifest retrieved from the UI as a temporary fix.

  4. Validate Agent Connection
    Ensure the agent pod logs show a successful tunnel connection:

     
    {"msg": "Starting tunnel"}
    {"msg": "Connected successfully"}

    If it fails with websocket: bad handshake, verify:

    • The IP Allow List is correctly configured to allow control plane connections.

    • The agent can reach the Akuity control plane endpoint (no firewall/DNS restrictions).

  5. Confirm Cluster Health
    Once the agent connects successfully, the cluster should display a 💚 in the console and appear in Argo CD as a valid deployment target.

References: