Applications Intermittently Showing Unknown Status with Invalid Username/Password Errors in Akuity Cloud

In some environments, Argo CD applications managed through Akuity Cloud may intermittently enter an Unknown status. When inspecting affected applications, users may see errors similar to:
ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 1:
rpc error: code = Unknown desc = failed to list refs:
WRONGPASS invalid username-password pair or user is disabled.

Although this error appears to indicate a Git credential problem, the root cause is often Akuity Agent instability or connectivity issues, rather than invalid repository credentials.

Symptoms

You may observe one or more of the following:

• A subset of applications intermittently show Status: Unknown in the Akuity Cloud Argo CD UI.

• Inspecting an affected application shows a credential-related ComparisonError.

• The issue is isolated to applications deployed to a specific cluster (for example, stage-us-east1-orange).

• The akuity-agent pod in the affected cluster shows:

  • A high restart count

  • Repeated tunnel connection errors in logs, such as:

  • {"level":"INFO","caller":"proxy/tunnel.go:82","msg":"agent tunnel failed, retrying"}
client: Connecting to wss://<agentsvr>.cdsvcs.akuity.cloud:443
client: Connection error: server: Server cannot listen on R:1040=>8001
client: Give up

Root Cause

This behavior is typically caused by an unstable or unhealthy Akuity Agent in the affected cluster.

When the agent:

• Fails to maintain a stable tunnel to the Akuity control plane, or

• Is repeatedly restarting,

Argo CD may be temporarily unable to:

• Reach the repository server

• Fetch Git references

• Generate manifests

As a result, Argo CD surfaces a misleading Git credential error, even though credentials are unchanged and valid.

Because the issue is cluster-specific, only applications deployed to that cluster are affected.

Why This Appears as a Credential Error

The Argo CD application controller relies on the Akuity Agent as a secure transport to:

• Access Git repositories

• Render manifests

• Compare desired vs live state

When the agent connection fails mid-operation, Argo CD cannot complete repository operations and reports a generic authentication-style failure (WRONGPASS), even though the underlying issue is connectivity, not credentials.

Resolution

Recommended Fix: Reinstall the Akuity Agent

Manually re-applying the Akuity Agent to the affected cluster typically resolves the issue.

Steps (via Akuity Cloud UI):

1. Navigate to:
   Akuity Cloud → Argo CD → Your Instance → Clusters

2. Locate the affected cluster.

3. Click Install in the Manifest column.

4. Copy the generated install command.

5. Run the command against the affected Kubernetes cluster.

Once the agent is reinstalled and healthy:

• The cluster should return to a 💚 Healthy state.

• Applications should recover from Unknown to their correct health status.

• Credential errors should no longer appear.

After reinstalling the agent, verify:

• akuity-agent pods are Running with low or zero restarts.

• Agent logs no longer show repeated tunnel failures.

• Cluster status in Akuity Cloud shows 💚 Healthy.

• Applications previously in Unknown state return to Healthy / Synced.