OpenSearch Alerting sends one notification for three matching lines. Which monitor sends one per line?

This note comes from a question on the OpenSearch forum: alert on any log line whose message contains one of ten words (error, warning, alert, failed, unavailable, lost, wrong, unable, invalid, failure), except lines containing six exclusion strings and lines from the sshd-session service, with one notification for every matching line. An earlier reply in that thread, from this author, suggested a per query monitor because “you get one alert per run instead of one per matching line.” For this goal that was the wrong pick, and the table below shows why. We measured all three monitor types instead of guessing.

IDX=http://localhost:9200   # your cluster. Ours had the security plugin off: plain HTTP, no login.
CURL="curl -s"              # with security on: CURL="curl -sk -u admin:YOUR_PASSWORD" and IDX=https://...

$CURL "$IDX/_plugins/_alerting/monitors/MONITOR_ID" | python3 -m json.tool \
  | grep -E '"monitor_type"|"interval"|"unit"|per_alert|per_execution|actionable_alerts|throttle_enabled'

$CURL "$IDX/_plugins/_alerting/monitors/alerts?size=200" | python3 -c '
import json,sys
for a in json.load(sys.stdin).get("alerts",[]):
    print(a["monitor_name"], a["id"], a["state"], a["start_time"], a.get("related_doc_ids"))'

What we measured

OpenSearch 3.6.0 (the indexer bundled with Wazuh 5.0.0-rc1, Alerting and Notifications plugins as shipped there), one node, security plugin off, our test containers, 28 October 2026. Times are UTC. A test index with message as text (plus a message.keyword subfield), service and host as keyword, and @timestamp as date. Three monitors, all created through POST /_plugins/_alerting/monitors, all on a 1-minute schedule, each with one webhook action to the same listener, "throttle_enabled": false.

Run 1: three matching lines, three monitor types

Three documents containing “error”, written at 04:04:07.7, 04:04:12.9 and 04:04:18.0, plus one control document with none of the words. All three monitors ran at 04:04:52. We counted what the webhook listener received, then kept watching until 04:07:46.

• No repeats. From 04:04:52 to 04:07:46, about 3 minutes and at least 2 more runs, no monitor sent anything again. The control document produced nothing.
• Per bucket gave 3 only because the three lines had different text. It groups by the exact message, so two lines with identical text should share one bucket. We did not measure that case, so for “one notification per line” per document is the type that matches the goal by design.

Run 2: the full condition, six documents

One per document monitor with the whole condition as its doc-level query (below, in how to set it up), the same per_alert / ["NEW"] action. Six documents, 5 seconds apart, from 04:10:27 to 04:10:53:

The run at 04:11:12 sent exactly 3 notifications, and the 3 alerts' related_doc_ids were exactly documents 1, 2 and 3. Nothing more was sent through 04:14:20.

Same run on stock OpenSearch 3.8.0. We repeated run 2 on the opensearchproject/opensearch:3.8.0 image (Alerting 3.8.0.0, security plugin off), same index mapping, same query and action, same six documents from 04:58:19 to 04:58:45. The run at 04:59:04 sent exactly 3 notifications, the 3 alerts' related_doc_ids were documents 1, 2 and 3, and nothing more was sent through 05:02:13. We did not repeat run 1 (the per query and per bucket comparison) on 3.8.0.

Why it happens

A per query monitor runs your search once per run and evaluates the trigger once, on the whole result. Three hits or three hundred, the trigger is either true or false for that run. In our test the run that saw all three lines sent one notification from an action with no execution policy. That is useful when you want one message per run, which is what the earlier forum reply optimised for, and the wrong shape when you want one message per line.

A per document monitor checks each new document against its queries and keeps an alert per matching document. With the action on per_alert and actionable_alerts: ["NEW"], each new alert sent one notification, and none was sent again in the 3 minutes we watched: 3 documents, 3 alerts, 3 notifications, 0 repeats.

One thing we did not expect to be true, so we checked it: the API accepted per_alert with ["NEW"] on a per query monitor too, and stored it. We did not run that monitor, so we cannot tell you what it does in practice, and we did not check what the Dashboards editor offers for a per query monitor.

How to set it up (tested)

The per document monitor from run 2, with the whole condition in one doc-level query. Names and the message template differ from our test run; the query and the action policy are the ones we ran. Replace the index, the channel id and the message template with yours:

$CURL -XPOST "$IDX/_plugins/_alerting/monitors" -H 'Content-Type: application/json' -d '{
  "type": "monitor",
  "monitor_type": "doc_level_monitor",
  "name": "critical-lines",
  "enabled": true,
  "schedule": { "period": { "interval": 1, "unit": "MINUTES" } },
  "inputs": [ { "doc_level_input": {
    "description": "",
    "indices": [ "YOUR-INDEX" ],
    "queries": [ {
      "id": "q-critical", "name": "q-critical", "tags": [],
      "query": "message:(error OR warning OR alert OR failed OR unavailable OR lost OR wrong OR unable OR invalid OR failure) AND NOT message:\"disabling inotify\" AND NOT message:\"user config - ignore\" AND NOT message:\"_put_session\" AND NOT message:\"PVE ticket\" AND NOT message:\"guest-ping\" AND NOT message:\"Cyberprotect\" AND NOT service:\"sshd-session\""
    } ]
  } } ],
  "triggers": [ { "document_level_trigger": {
    "name": "critical-line",
    "severity": "1",
    "condition": { "script": { "source": "query[name=q-critical]", "lang": "painless" } },
    "actions": [ {
      "name": "notify",
      "destination_id": "YOUR-CHANNEL-ID",
      "subject_template": { "source": "Critical log line", "lang": "mustache" },
      "message_template": { "source": "Critical log line, run ending {{ctx.periodEnd}}", "lang": "mustache" },
      "throttle_enabled": false,
      "action_execution_policy": { "action_execution_scope": { "per_alert": { "actionable_alerts": [ "NEW" ] } } }
    } ]
  } } ]
}'

• The trigger condition names the query: query[name=q-critical]. Doc-level queries use query string syntax, so the exclusion list is one string with AND NOT clauses.
• Check your service field. In our test index service was a keyword field, so service:"sshd-session" matched it exactly. If yours is text, test that the clause still excludes those lines, or point it at the keyword subfield.
• Only two of the six exclusions were tested with a real line: “disabling inotify” and the sshd-session service. For the other four strings we only know the query was accepted. Send one line for each before you rely on it.
• If your channel is a webhook to a private address, notifications can fail silently: the default host_deny_list refuses private ranges. We cleared it in the lab for this test and restored it afterwards. See webhooks to private addresses; allow only the address you need.