Agent remains stuck in Outdated policy state for over 10 minutes on updating spaces used by the policy.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.18.0 SNAPSHOT
BUILD: 81372
COMMIT: 2b68edac50a036040cbdaf0f14030d39d39da677

Preconditions:

1. 8.18.0-SNAPSHOT Kibana cloud environment should be available.
2. A few agents should be installed.
3. A few spaces should be used by the agent policy.

Steps to reproduce:

1. Navigate to Fleet>Agents tab.
2. Update spaces for any agent policy used by Agents.
3. Observe Agent remains stuck in Outdated policy state for over 10 minutes.
4. Observe Agent policy remain stuck in updating under agent activity.
5. Collect logs for the agent.[Before this logs were not collected]
6. Observe error: Rate limit exceeded.
7. Wait for sometime and observe agent gets recovered from outdated state and logs are collected.

NOTE:

• Issue is not observed for agent policy without Agent installed.
  

Expected Result:
Agent should get updated on updating spaces used by the policy.

Screen Recording:

Agents.-.Fleet.-.Elastic.-.Google.Chrome.2024-12-31.14-47-29.mp4

Feature:
https://github.com/elastic/ingest-dev/issues/1664

Logs:
elastic-agent-diagnostics-2024-12-31T09-24-56Z-00.zip

JSON:
EC2AMAZ-SN9KQPI-agent-details.zip

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[amolnater-qasource]

@muskangulati-qasource Please review.

[muskangulati-qasource]

Secondary review is Done for this ticket!

[jlind23]

FYI @nchaulet adding this one to our current sprint
cc @kpollich

[nchaulet]

It seems it's an issue when updating previous agent action it trigger fleet server to distribute them again. Introduced by elastic/kibana#203683

I am not sure what will be the best way to solve that one,
@kpollich @nimarezainia will it be an acceptable behaviour to not update previous agent actions?

[nchaulet]

How to reproduce the issue

with space awareness enabled, enroll an agent and request a diagnostics, than assign that agent to multiple space.