Wazuh 5.0 events and alerts appear about 20 seconds late

grep -v '^#' /var/wazuh-manager/etc/wazuh-manager-internal-options.conf

strings /var/wazuh-manager/bin/wazuh-manager-analysisd \
  | grep -E 'indexer_(flush_interval|bulk_max_bytes)|INDEXER_(FLUSH_INTERVAL|BULK_MAX_BYTES)'

analysisd.indexer_bulk_max_bytes must be between {} and {} (got {})
analysisd.indexer_flush_interval must be between 1 and {} (got {})
analysisd.indexer_flush_interval
analysisd.indexer_bulk_max_bytes
WAZUH_INDEXER_BULK_MAX_BYTES
WAZUH_INDEXER_FLUSH_INTERVAL

# /var/log/lab-auth.log must already exist and be in the agent's <localfile> list
# (log_format syslog) before you write to it.
IP=203.0.113.200            # a test IP that is not in your data; use a new one for each try
IDX=http://localhost:9200   # your indexer. Ours had the security plugin off: plain HTTP, no login.
CURL="curl -s"              # normal install: CURL="curl -sk -u admin:YOUR_PASSWORD" and IDX=https://...
t0=$(date +%s.%N)
echo "$(date '+%b %e %H:%M:%S') $(hostname) sshd[4900]: Failed password for invalid user root from $IP port 55000 ssh2" >> /var/log/lab-auth.log
until $CURL -X POST "$IDX/wazuh-events-v5-*/_count" -H 'Content-Type: application/json' \
      -d "{\"query\":{\"term\":{\"source.ip\":\"$IP\"}}}" | grep -q '"count":[1-9]'; do sleep 0.3; done
awk -v a="$t0" -v b="$(date +%s.%N)" 'BEGIN{printf "searchable after %.1f s\n", b-a}'

Why it happens

In 5.0.0-rc1, analysisd sends events to the indexer through its indexer connector. It does not send each event as it arrives. It collects them and sends them in a bulk request when a timer fires. The timer is analysisd.indexer_flush_interval, in seconds. It also sends early when the batch reaches the size limit analysisd.indexer_bulk_max_bytes; in our tests that happened at 500 events per second, not at 200 or below (see Under load). Until the bulk request is sent, the event cannot be found in the index.

So the delay is not constant. An event that arrives just before a flush becomes searchable soon after it (our shortest wait was 2.2 seconds); an event that arrives just after one waits a full interval. With the default, new events appear about every 20 seconds, in batches.

The range 1 to 3600 is also what analysisd says when the value is out of range. We set the interval to 0 and restarted. analysisd refused to start and logged:

wazuh-manager-analysisd: ERROR: An error occurred while initializing the modules: Could not initialize Indexer Connector: analysisd.indexer_flush_interval must be between 1 and 3600 (got 0).

In the same restart, the status command reported wazuh-manager-remoted, wazuh-manager-modulesd and wazuh-manager-clusterd as not running as well. A bad value does not fall back to the default; it stops the manager from processing events. Check the status after every change.

How to change it

Add one line to the internal options file and restart the manager. As root, on the manager. Remove any earlier line for the same option first, so the file holds only one:

F=/var/wazuh-manager/etc/wazuh-manager-internal-options.conf
sed -i '/^analysisd\.indexer_flush_interval/d' $F
echo 'analysisd.indexer_flush_interval=5' >> $F
/var/wazuh-manager/bin/wazuh-manager-control restart
/var/wazuh-manager/bin/wazuh-manager-control status

After a good restart, status lists wazuh-manager-analysisd is running..., and the manager log has these two lines:

grep -E "Indexer Connector initialized|Raw Event Indexer initialized" /var/wazuh-manager/logs/wazuh-manager.log | tail -2

The file's own header says it “will not be overwritten during upgrades”. To go back to the default, delete the line and restart; in our test the cycle returned to about 20 seconds. wazuh-manager-control restart is the command we used inside the container. We did not test other ways of restarting the service.

What we measured

Wazuh 5.0.0-rc1 (commit aff7a0b), our test containers, 15 October 2026. Two paths into the manager:

• Socket: one log line per event, pushed straight into the manager's engine through its local ingest socket (engine-ingest-http.sock).
• Agent: a Wazuh 4.14.7 agent reading a log file, connected to the 5.0.0-rc1 manager over 1514/tcp.

Each run sent 12 events, one at a time. For each event we timed from sending it until a _count query on wazuh-events-v5-* first found it, polling every 0.3 seconds. After each event became searchable, the script waited 3.7 × (n mod 4) seconds before sending the next. Times are in seconds.

With 5 seconds, the gaps between events were not all 5 seconds, because our script waited different times between events. The cycle shows in where the events landed: in every 5-second run, each event became searchable at nearly the same point of the 5-second cycle, all within about 1.4 seconds of each other.

The agent path behaved like the socket path. At this load (one log line at a time, an idle agent) the 4.14.7 agent added no delay we could see on top of the flush cycle.

The shortest waits are not a typical value. They depend on where each event landed relative to the timer, and our sending rhythm locked onto the cycle. Read the table for the cycle and the longest wait, not for an average.

Under load (added 15 October 2026, after the video)

Wazuh 5.0.0-rc1, our test containers, 15 October 2026: one manager and one indexer node (OpenSearch 3.6.0, 3 GB heap, security plugin off). These tests answer three questions the first runs left open: how often the events index refreshes, what a 5-second flush costs when events arrive steadily, and whether analysisd.indexer_bulk_max_bytes sends a bulk before the timer.

How we loaded it. A generator sent sshd “Failed password” lines straight into the manager's local ingest socket, one batch every 0.1 seconds for 115 seconds, at 50, 200 or 500 events per second (it reached 50.0, 200.2 and 500.4). It did not go through agents. While it ran, a second script sent 16 marked events 1.9 seconds apart, without waiting for each to appear, and searched for them every 0.25 seconds. Over a 75-second window in each run we counted the bulk requests the indexer received, the bytes the manager container sent on its network interface, the CPU time of analysisd, and docker stats for the containers. Each load rate at each flush setting ran twice; the runs without load and the bulk-size runs ran once. Because the probe rhythm differs from the table above, compare longest waits and cycles with it, not shortest waits.

A. How often the events index refreshes

Our test event landed in the data stream wazuh-events-v5-system-activity. Its backing index has refresh_interval set to 2s, explicitly, from the index template wazuh-events-v5-system-activity-template; it is not left to the OpenSearch default. All eight wazuh-events-v5-* and all eight wazuh-findings-v5-* backing indices had 2s; wazuh-events-raw-v5, wazuh-active-responses and the three wazuh-metrics-* streams had 5s. So after a bulk arrives, an event can take up to about 2 more seconds to become searchable. Because the value is set explicitly, OpenSearch's documented search-idle pause (index.search.idle.after, 30s by default here) should not hold back refreshes on a quiet index. We did not test that.

B. What a 5-second flush costs under load

Two runs per row, shown as first / second. Bulk requests per second are what the indexer received over the window. Bytes per bulk are what the manager container sent on its network interface, divided by the number of bulks; we only recorded them from the sixth run on.

* The 75-second window cut across the 20-second cycle in this run, so it undercounts; all 23,020 events were indexed in the end. At 20 seconds the window holds only 4 or 5 bulks, so those bulk rates can be off by one bulk. Without load (one run each), the default sent 2 bulk requests in the window and the longest wait was 20.9 s; the 5-second setting sent 7, each 5,828 bytes, and the longest wait was 6.3 s.

• Nothing was lost at 5 or 20 seconds. In all 12 load runs, every event sent was indexed and every send got HTTP 200. 500 events per second is the highest rate we tried; we did not look for the limit.
• More, smaller requests, same traffic. At 5 seconds the indexer got about 0.2 bulk requests per second instead of 0.05 to 0.07, each smaller: about a quarter of the size at 50 and 200 events/s, and about 37% at 500, where the default bulks were already cut by the size limit (see the last points). The bytes the manager sent per second stayed the same: about 240 KB/s at 200 events/s and 556 to 607 KB/s at 500 events/s, at both settings.
• analysisd CPU followed the event rate, not the flush interval: 5.1 to 6.4% of one core at 50 events/s, 11.4 to 14.7% at 200, 22.0 to 26.1% at 500, at both settings.
• Indexer CPU: too noisy to compare. Sampled about every 2 seconds, the per-run average was 9 to 57% and single samples reached 742% (100% is one core), at both settings. We see no consistent difference between 5 and 20 seconds and, with two runs per setting, draw no conclusion. Indexer heap readings stayed between 11% and 67% of 3 GB.
• Load did not stretch the 5-second cycle: the longest wait was 6.3 to 7.0 seconds at every rate.
• At 500 events/s the default cycle got shorter. The longest wait fell to 14.1 and 14.5 seconds, and probes became searchable in batches about 14 seconds apart. Each bulk held 7,098 events in both runs, and the manager sent 8,410,769 bytes per bulk: just above 8 MiB (8,388,608 bytes), the default analysisd.indexer_bulk_max_bytes. At 500 events per second, 7,098 events arrive in about 14.2 seconds. So the batch hit the size limit and went out before the 20-second timer. Part C tests that directly.
• No empty bulks. With no load, the 5-second setting sent one bulk per cycle in which a probe was waiting (7 in the window), not one every 5 seconds.

Indexer write blocks. During the session the indexer's Security Analytics plugin logged 12 times “Events ingestion backpressure: Applying write block on 'wazuh-events-v5-*' (correlation backlog 10000/10000)”, and lifted each block 0.4 to 1.2 seconds later (11 blocks; the 12th fell between runs, so we have only its start). Around several of them it also logged that it was dropping correlation and enrichment for findings. Blocks came at both flush settings and at 50, 200 and 500 events/s, always 16 to 21 seconds after an even-numbered minute, not in step with the flush timer; we did not establish what triggers them. Our load was one event type that matches a detection rule, which may push the correlation queue harder than a mixed real stream. At 5 and 20 seconds, the manager logged no indexer warnings and every event was indexed.

C. Does analysisd.indexer_bulk_max_bytes send early? Yes (one run per rate)

We set analysisd.indexer_bulk_max_bytes=65536, the minimum, and left the flush interval at the default 20 seconds. analysisd started normally.

• A size limit does send a bulk before the timer. Bulks were cut at about 64 KiB, and the indexer got about 68 times more bulk requests at 200 events/s and about 136 times more at 500 than with the defaults. The longest wait fell to 2.2 to 2.3 seconds, close to the 2-second refresh from part A. analysisd CPU did not change.
• The 500 events/s run lost 112 events. At 05:52:17 UTC the manager logged twice “Index write(s) blocked by the cluster (cluster_block_exception) - 56 document(s) in this batch (112 total) not indexed due to this expected condition. Backing off future sends.” (the first line reads “56 total”). That matches an indexer write block from 05:52:16.981 to 05:52:17.414. The final count was 112 short. With about 9 bulks a second, a block of under half a second catches some of them, and the log says those documents were not indexed. With 5- and 20-second flushes, no bulk in our 12 load runs landed in a block; that may only be chance.
• This is one run per rate, not repeated. On what we saw, we would not lower analysisd.indexer_bulk_max_bytes on 5.0.0-rc1 to cut the delay.

Limits of the load tests. The runs without load and both bulk-size runs ran once; each load rate at 5 and 20 seconds ran twice. One event type, one simulated agent, sent through the local ingest socket rather than through agents and remoted; a real mix has different document sizes and a different share of events that match rules. We stopped at 500 events per second and did not try a 1-second flush. Bytes per bulk come from the manager container's whole network interface, so they include a little other traffic. CPU from docker stats is sampled about every 2 seconds and is noisy, and the indexer ran its own periodic work. All containers ran on one machine.

Trade-offs

• A smaller interval means events become searchable sooner, and analysisd sends bulk requests to the indexer more often. Under steady load, that means more, smaller requests. In our load tests (one manager, one indexer node, up to 500 events per second), going from 20 to 5 seconds gave about 0.2 bulk requests per second instead of 0.05 to 0.07, with the same total bytes, the same analysisd CPU and every event indexed. Indexer CPU was too noisy to show a difference either way. We did not measure memory or throughput beyond that, and did not test a cluster.
• A smaller analysisd.indexer_bulk_max_bytes also sends sooner: at the minimum, 65,536 bytes, our longest wait fell to 2.2 to 2.3 seconds. It also multiplied bulk requests by about 68 to 136, and in our one run at 500 events per second, 112 events were not indexed during an indexer write block. One run per rate.
• A larger interval means fewer bulk requests and a longer wait. At 40 seconds our longest wait was 39.9 seconds.
• Anything that reads the index on a schedule, such as dashboards and alerting monitors, sees an event only after its flush. We did not measure how this affects monitors.
• Keep the value inside 1 to 3600. Outside it, analysisd does not start.