OEM 24ai Agent Unreachable / Metric Collection Error: 12 Real Fixes

Published Sep 2026 · 9 min read

The red "Agent Unreachable" icon is the ticket every Enterprise Manager administrator sees most, and it is also the one most often fixed the wrong way — by restarting things until the icon changes colour. Almost every case is one of a handful of causes: the agent cannot reach the OMS, the OMS no longer trusts the agent, the agent's local state is out of step with the repository, or a single monitoring credential has gone stale.

Below are twelve real situations, in the order they usually come up, each with the exact check and the exact fix. Commands were run against OEM 24ai (Cloud Control 24.1) agents on Oracle Linux 9; the same emctl commands work on 13.5 agents. Every command runs from the agent's $AGENT_HOME/bin unless it says otherwise.

Before anything else: the 60-second triage

Run these three commands before you touch the console. Between them they answer whether the agent is running, whether it can talk to the OMS, and whether it has a backlog it cannot upload.

# 1. is the agent process alive, and when did it last talk to the OMS?
$AGENT_HOME/bin/emctl status agent
#   look for: "Last successful heartbeat to OMS", "Last attempted upload",
#             "Number of XML files pending upload"

# 2. can this agent reach the OMS right now?
$AGENT_HOME/bin/emctl pingOMS

# 3. force an upload and read the error, if any
$AGENT_HOME/bin/emctl upload agent

1. emctl status agent says "Agent is Running" but the console shows Agent Unreachable — what do I check first?

These are two different facts. "Running" means the Java process is alive on the host; "Unreachable" means the OMS has not received a heartbeat from it recently. Read the timestamps in the emctl status agent output: if the last successful heartbeat is minutes old but the last attempted upload is failing, the agent is up and the problem is between the agent and the OMS — go to fix 2. If the heartbeat itself is stale, the agent is alive but stuck — go to fix 7. Restarting the agent before reading those two lines throws away the evidence.

2. "Agent Unreachable (Communication broken)" or "OMS is unreachable" — how do I fix it?

This is a network, port or trust problem, in that order. The agent uploads to the OMS on the secure upload port (4903 by default) and the OMS calls back to the agent on 3872. Check both directions before touching certificates.

# from the AGENT host: can I reach the OMS upload port?
nc -zv oms.example.com 4903

# from the OMS host: can the OMS reach the agent?
nc -zv agenthost.example.com 3872

# agent-side verdict
$AGENT_HOME/bin/emctl pingOMS
# "EMD pingOMS completed successfully"  -> network is fine, look at fix 3 or 4
# "EMD pingOMS error: ... REJECTED"     -> trust problem: re-secure the agent

# re-register the agent with the OMS (needs the agent registration password)
$AGENT_HOME/bin/emctl secure agent
$AGENT_HOME/bin/emctl start agent
$AGENT_HOME/bin/emctl upload agent

Also confirm both hosts agree on the time. A clock drifting by more than a few minutes makes uploads fail with certificate errors that look like a trust problem but are really an NTP problem.

3. The agent shows Blocked — "agent is out-of-sync with repository" — after a host restore. How do I unblock it?

The OMS blocks an agent when the agent's local state is older than what the repository holds — classic after restoring the agent host from a backup, cloning it, or restoring the agent directory. The fix is not to clear the agent; it is to resynchronise it from the repository side.

# from the OMS host, as the EM CLI user
emcli login -username=sysman
emcli resyncAgent -agent="agenthost.example.com:3872"

# then on the agent host
$AGENT_HOME/bin/emctl start agent
$AGENT_HOME/bin/emctl upload agent

The console route is Setup → Manage Cloud Control → Agents, select the blocked agent, Unblock. Use it when EM CLI is not set up; it does the same thing.

4. Target shows Metric Collection Error but the database is up — how do I find the failing metric?

A metric collection error means the agent is fine but one collection against the target failed, and nine times out of ten the cause is the monitoring account: DBSNMP locked, expired, or its password changed without updating the target's monitoring credentials. Ask the agent which collection is failing rather than guessing.

# what does the agent think of this target?
$AGENT_HOME/bin/emctl status agent target orcl,oracle_database

# re-run the collection that raises the error and read the message
$AGENT_HOME/bin/emctl control agent runCollection orcl:oracle_database Response

# on the target database
SELECT username, account_status, expiry_date FROM dba_users WHERE username = 'DBSNMP';
ALTER USER dbsnmp ACCOUNT UNLOCK;
ALTER USER dbsnmp IDENTIFIED BY <password>;

Then update the monitoring credentials in the console (Setup → Security → Monitoring Credentials) so the agent and the database agree again. If the account is fine, the collection output names the real cause — usually a listener change or a moved ORACLE_HOME.

5. New targets are stuck in Pending — why, and how do I force them through?

Pending is normal for a few minutes after discovery; it means the target is registered but the agent has not yet uploaded its first data. If it stays Pending, the agent has an upload backlog. Force it and read the count.

$AGENT_HOME/bin/emctl upload agent
$AGENT_HOME/bin/emctl status agent | grep -i "pending upload"
$AGENT_HOME/bin/emctl status agent scheduler | head -20

A pending count that keeps climbing while uploads "succeed" means the OMS is accepting but not processing — check the OMS side with emctl status oms -details and the loader status in the console before blaming the agent.

6. "EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet" — what does it mean?

The agent has never completed its first handshake with the OMS since it started, so it refuses to upload anything. It happens when the agent came up while the OMS was down or unreachable, or right after a re-secure. Confirm the OMS is up, then re-secure and upload.

# on the OMS host
$OMS_HOME/bin/emctl status oms -details

# on the agent host, once the OMS is up
$AGENT_HOME/bin/emctl secure agent
$AGENT_HOME/bin/emctl upload agent

7. When is emctl clearstate agent safe, and what does it delete?

clearstate removes the agent's local state files — the collected metric data and upload backlog it has not yet sent. It does not touch target definitions or the repository. It is the right tool when the upload backlog is corrupt or so large the agent chokes on it; it is the wrong tool for a trust or network problem, because you lose the unuploaded metric history for that window and gain nothing. Always stop the agent first.

$AGENT_HOME/bin/emctl stop agent
$AGENT_HOME/bin/emctl clearstate agent
$AGENT_HOME/bin/emctl start agent
$AGENT_HOME/bin/emctl upload agent

8. How do I blackout the agent during OS patching so I don't get a page at 2am?

Blackouts suppress alerts and stop the availability record showing a false outage. A node-level blackout from the agent covers every target on that host, which is exactly what you want for OS patching.

# start, do the work, stop
$AGENT_HOME/bin/emctl start blackout os_patch_sept -nodeLevel
$AGENT_HOME/bin/emctl status blackout
$AGENT_HOME/bin/emctl stop blackout os_patch_sept

# the same from the OMS, for many hosts at once
emcli create_blackout -name="os_patch_sept" -add_targets="agenthost.example.com:host" -reason="OS patching" -schedule="duration:2:00"

9. The agent will not start — where do I look?

Two log files answer almost every failed start. Read them before removing any lock file.

tail -100 $AGENT_INST/sysman/log/gcagent.log
tail -100 $AGENT_INST/sysman/log/emagent.trc

# the usual suspects
df -h $AGENT_INST          # full filesystem
hostname -f; nslookup $(hostname -f)   # name resolution changed
ps -ef | grep -i emagent   # a half-dead previous process

Only remove a stale lock file when the log says the agent is already running and ps proves it is not.

10. The agent is healthy but nobody ever receives an alert — is that the agent's fault?

No. Alerting is an OMS function. If targets show incidents in the console but no email arrives, check Setup → Notifications → Mail Servers (send a test), then the incident rule set that should match those incidents, then the recipient's notification schedule. An agent that uploads cleanly has done its job; silence after that point is configuration on the OMS.

11. After patching the OMS the console says the agent version is old — do I have to touch every host?

Agents are upgraded from the console, not host by host: Setup → Manage Cloud Control → Upgrade Agents. Select the agents, let the OMS push the new software, and it restarts them for you. Keep agents within the versions the OMS supports; an agent too far behind is one of the quieter causes of "unreachable" after an upgrade weekend.

12. The same agent goes unreachable at the same time every night — what is that?

Look at what else runs at that time. Backups saturating the network, a firewall that drops idle sessions, a nightly reboot, or an NTP step that jumps the clock all produce a short unreachable window that clears itself. Match the timestamps in emagent.trc with cron and the firewall logs. Set a scheduled blackout for the genuine maintenance and fix the network for the rest — do not raise the unreachable threshold to hide it.

FAQ

What is the difference between Agent Unreachable and Agent Blocked?

Unreachable means the OMS is not hearing from the agent — a running, network or trust problem on the agent side. Blocked means the OMS is deliberately refusing the agent because its state is out of sync with the repository, which needs emcli resyncAgent or the console Unblock, not a restart.

Does emctl clearstate agent remove targets?

No. It only clears the agent's local collected data and upload backlog. Targets and their history in the repository are untouched. You lose the metric data the agent had not yet uploaded.

Which ports must be open between an OEM 24ai agent and the OMS?

The agent uploads to the OMS on the secure upload port, 4903 by default, and the OMS connects back to the agent on the agent port, 3872 by default. Both directions must be open; emctl pingOMS tests the first and the console's agent page tests the second.

Why does a Metric Collection Error appear when the database is clearly up?

Because the agent could not log in to collect, not because the database is down. Nearly always the DBSNMP monitoring account is locked or expired, or its password changed without updating the target's monitoring credentials in Enterprise Manager.

Where this leads

An agent that is unreachable is annoying; a monitoring estate nobody trusts is dangerous, because the alerts that matter get ignored with the noise. The habit that fixes both is the triage at the top of this page — status, ping, upload — before any restart.

If you would rather learn this on a working estate than from a runbook, our Oracle Enterprise Manager training covers agent deployment, incident rules and this exact troubleshooting on a live OMS, and it sits alongside the rest of our live Oracle DBA training programs. The two companion labs — installing OEM 24ai on Oracle Linux 9 and adding targets and building your first dashboard — are the natural next reads. For the full command reference, Oracle's EMCTL commands for the Management Agent is the page to keep open.