Alerts

Alert types

An alert watches a filter and fires when its condition holds. To create one, open Alerts > All alerts in the sidebar and choose Add alert, or choose Subscribe on a filter's page to start with that filter selected.

The form asks for:

• Name - unique within the account.
• Filter - the saved filter the alert watches. The alert uses the filter as it is now, so editing the filter changes what the alert watches. A filter cannot be deleted while an alert watches it.
• Notification channels - where incidents are delivered (see Channels).
• Fires when - the alert type. The four types are described below, and each one shows its own settings.

Choosing a type

Pick the type by the problem it should catch:

• The problem is that something happened: Matching events occur.
• The problem is that something stopped happening, with no timetable: Matching events stop.
• The problem is that something did not happen by the time it should have: An expected event is late.
• You want the events themselves delivered somewhere: Every matching event.

How alerts fire covers when alerts are checked, reminders and what a notification contains.

Matching events occur

Fires when the filter matches at least Threshold events within the trailing Window. A threshold of 1 fires on any match. A threshold of 50 in a 5-minute window fires on a burst and ignores a trickle.

Use this type for errors, failed jobs and anything else that is a problem when it appears.

The window can be anything from 1 minute to 14 days.

Matching events stop

Fires when the filter matches nothing for the whole Window plus the Grace period. Use it as a heartbeat: a worker that logs a line every minute, a backup that reports when it finishes, a source that should never go quiet.

The grace allows for ordinary lateness, so a job that finishes a few minutes late does not fire the alert. It can be None, 5, 15 or 30 minutes, 1 hour or 2 hours.

An expected event is late

Fires when nothing matches within the Grace period after an expected time. For example, a nightly backup expected at 00:20 with 30 minutes of grace must have logged its confirmation by 00:50.

Choose when the event is expected under Expected:

Expected	When
Every day	At the time in Expected at, every day.
Weekdays (Mon-Fri)	At the time in Expected at, Monday to Friday.
Every week	At the time in Expected at on the weekday chosen under Day.
Custom (cron expression)	At the times a five-field Cron expression gives (minute, hour, day of month, month, day of week). Use it for anything else, such as every six hours (0 */6 * * *) or the first of the month.

Every schedule is read in the account's timezone (the Default timezone under Account > Settings), so "02:00" means 02:00 in that timezone, including across daylight-saving changes. The alert's page shows the timezone and the next expected time.

Each deadline comes from the schedule, so a late run never moves the next day's deadline.

The grace starts at 30 minutes when you choose this type. With None, the alert fires as soon as the expected time passes with no match.

Edge cases:

• A new alert first checks the first expected time after it was created. It never fires for a run that was due before the alert existed.
• Each missed expected time is its own incident. A daily backup that misses two days running records two incidents, and each one sends a notification.

Every matching event

Sends every event the filter matches to the channels, in batches of at most one notification per minute. Use it to deliver the events themselves: a stream of specific events into a webhook, or every critical line by email.

This type has no open or resolved state and records no incidents. The alert's page shows the most recent delivery.

Details:

• Delivery starts when the alert is saved. Events from before then are never sent.
• A batch carries up to 100 events. When more matched, the notification says so and the rest are in the Log Explorer.
• Events that match while the alert is muted or snoozed, or has no channel, are skipped and not sent later.

Channels

A channel is a place alerts deliver to: email addresses, a Slack or Discord channel, a Pushover user or group, or a webhook URL. One channel can be used by any number of alerts. Channels are listed under Alerts > Channels, where Add channel creates one. You can also create one from an alert's form, with Create a notification channel in the Notification channels list.

To check that a channel delivers, open its page, choose Send test and confirm. The confirmation says where the test will go. A test notification goes to that channel alone, whether or not any alert uses it. It is titled "Test notification:" and the channel's name, and a webhook receives it with "event": "channel_test".

Deleting a channel removes it from every alert that used it. The alerts themselves and their incidents stay.

Email

One email address, or up to 20 separated by commas or new lines, so one channel such as "Ops team" can reach several people. Each incident arrives as one email, addressed to everyone on the channel, with the subject "Alert triggered:" and the alert's name. The email contains:

• the alert's filter and condition, and what happened compared with what the filter usually does
• up to 5 recent matching lines, or the last matching line for an alert about something that stopped or is late
• View incident, which opens the incident's page
• Open in Log Explorer, which opens the Log Explorer on the alert's filter
• a line naming the incident for a coding agent connected to the MCP server (see Handing an incident to an agent)
• Manage this alert, which opens the alert's page

Every user gets an email channel with their own address, named after them ("Jenny's email"), so an alert can notify a person without anyone setting up a channel first. A person who joins by invitation gets theirs when they accept. No channel is created when the account already has an email channel with that address.

From then on it is an ordinary channel: you can rename it, change its address or delete it. A later change to the user's email address does not change the channel.

Deactivating or deleting the user does not change the channel either: alerts keep sending to every address on it. The user's page under Account > Users lists the channels that send to their address or that they created, and the delete confirmation repeats the list, so the channels can be edited or deleted along with the user.

Slack

Posts each incident to a Slack channel through an incoming webhook. In Slack, create an app for the workspace (or open an existing one), turn on Incoming Webhooks, choose Add New Webhook to Workspace and pick the channel. Paste the URL it gives you, which starts with https://hooks.slack.com/services/, into Slack webhook URL.

Each message contains the alert's filter and condition, what happened, up to 5 recent matching lines, View incident and Open in Log Explorer buttons, and a link that opens the alert's page.

Discord

Posts each incident to a Discord channel through a webhook. In Discord, open the channel's settings, choose Integrations, then Webhooks and New Webhook, and use Copy Webhook URL. Paste it into Discord webhook URL.

The message has the same content as the Slack one. The webhook's own name and avatar, set in Discord, are shown as the sender. Mentions such as @everyone in a log line do not notify anyone.

Pushover

Sends each incident as a notification to the Pushover app on every device of a Pushover user or group. Paste the user key from your Pushover dashboard, or a group's key, into User or group key.

The notification contains the alert's name, what happened and the first matching lines, cut to Pushover's 1,024-character limit, and a link that opens the incident.

• Priority set to High makes the notification sound even during the recipient's quiet hours and shows it highlighted in red.
• Notifications come from SaneLog's own Pushover application, with its name and icon. To send them from your own Pushover application instead, paste its API token into Application token.

Webhook

An HTTPS URL that resolves to a public internet address. Plain HTTP URLs are not accepted. Each notification is sent as a JSON POST with the same content as the email, structured for a program to read. Use it to send incidents to an incident tool, a chat integration or your own endpoint.

The event field says what the post is, so check it before treating a post as a new incident:

event	Sent when
triggered	An incident opens, and again for each reminder while it stays open.
resolved	An incident resolves, when the alert sends resolutions.
forward	An Every matching event alert delivers a batch. The events are under events, and more is true when more matched than the batch holds.
test	Send test on the alert's page.
channel_test	Send test on the channel's page.

A triggered post looks like this:

{
  "event": "triggered",
  "alert": {
    "uuid": "42fccbae-bb13-485c-ab5b-cd50b727e069",
    "name": "Production errors",
    "type": "match",
    "condition": "at least 50 matches within 5 minutes",
    "url": "https://app.sanelog.com/alerts/42fccbae-bb13-485c-ab5b-cd50b727e069",
    "threshold": 50,
    "window_minutes": 5,
    "grace_minutes": 0,
    "expected_at": null,
    "due_at": null,
    "expected_at_label": null,
    "due_at_label": null,
    "timezone": null
  },
  "filter": {
    "uuid": "a1f7930e-ee52-40fa-a7bc-0b2e3020fe0c",
    "name": "Production errors"
  },
  "match_count": 812,
  "triggered_at": "2026-09-06T09:14:00+00:00",
  "url": "https://app.sanelog.com/log-explorer?source=filter%3Aa1f7930e-ee52-40fa-a7bc-0b2e3020fe0c",
  "samples": [
    {
      "id": "01a0c8bb-f376-7249-9bae-63f61d7e1752",
      "dt": "2026-09-06 09:13:58.412",
      "level": "error",
      "source": "web-1",
      "program": "laravel",
      "message": "SQLSTATE[HY000] [2002] Connection refused"
    }
  ],
  "history": {
    "narrative": "812 matches in the last 5 minutes. That is about 16 times its usual rate for 5 minutes.",
    "stats": { "kind": "match", "window_minutes": 5, "lookback_days": 7, "baseline_total": 3520, "baseline_per_hour": 21, "previous_window": 4 },
    "last_match": null
  },
  "incident": {
    "uuid": "7c1e5f0a-2b8d-4d6e-9f3a-1b2c3d4e5f60",
    "url": "https://app.sanelog.com/alerts/42fccbae-bb13-485c-ab5b-cd50b727e069/incidents/7c1e5f0a-2b8d-4d6e-9f3a-1b2c3d4e5f60"
  },
  "api": {
    "events_url": "https://app.sanelog.com/api/v1/events?filter=a1f7930e-ee52-40fa-a7bc-0b2e3020fe0c&since=5m",
    "docs_url": "https://app.sanelog.com/docs/api-reference",
    "mcp_url": "https://app.sanelog.com/mcp"
  }
}

• samples holds up to 5 recent matching lines, for a Matching events occur alert only.
• For Matching events stop and An expected event is late alerts, history.last_match holds the last matching line instead. A late-event alert also sets expected_at and due_at.
• A resolved post has the same alert, filter, incident and history, plus resolved_at.
• incident.uuid names the incident for the API's /incidents?incident= and the MCP server's incidents tool, and each line's id names the event for /context and /events.

Every post is signed, so your endpoint can check that it came from SaneLog. See Checking a webhook's signature.

Checking a webhook's signature

Each webhook channel has its own signing secret, shown under Signing secret on the channel's page behind the eye button. Store it with your endpoint's configuration, not in the webhook URL.

Every post carries an X-SaneLog-Signature header:

X-SaneLog-Signature: t=1788686040,v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd

t is the time the post was signed, in Unix seconds. v1 is a hex HMAC-SHA256 of the timestamp, a full stop and the raw request body, keyed with the signing secret. To check a post:

1. Read the raw body before any JSON parsing. A body that is parsed and encoded again will not match.
2. Compute the HMAC-SHA256 of t, . and the raw body with the signing secret.
3. Accept the post when your value equals any v1 in the header, compared in constant time.
4. Reject the post when t is more than 5 minutes from your own clock, so a captured post cannot be replayed later.

In PHP:

$body = file_get_contents('php://input');
$header = $_SERVER['HTTP_X_SANELOG_SIGNATURE'] ?? '';

$timestamp = null;
$signatures = [];
foreach (explode(',', $header) as $part) {
    [$key, $value] = array_pad(explode('=', $part, 2), 2, '');
    if ($key === 't') {
        $timestamp = (int) $value;
    } elseif ($key === 'v1') {
        $signatures[] = $value;
    }
}

$expected = hash_hmac('sha256', $timestamp . '.' . $body, getenv('SANELOG_SIGNING_SECRET'));
$valid = $timestamp !== null
    && abs(time() - $timestamp) <= 300
    && array_filter($signatures, fn ($signature) => hash_equals($expected, $signature)) !== [];

In Node.js:

const crypto = require('crypto');

function isValid(rawBody, header, secret) {
  const parts = header.split(',').map((part) => part.split('='));
  const timestamp = Number(parts.find(([key]) => key === 't')?.[1]);
  const signatures = parts.filter(([key]) => key === 'v1').map(([, value]) => value);
  if (!timestamp || Math.abs(Date.now() / 1000 - timestamp) > 300) return false;

  const expected = crypto.createHmac('sha256', secret).update(`${timestamp}.${rawBody}`).digest('hex');
  return signatures.some((signature) => signature.length === expected.length
    && crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected)));
}

Rotate signing secret on the channel's page issues a new secret. For a day after a rotation, each post carries two v1 values, one from the new secret and one from the previous secret, so a post still checks against the previous secret while you switch your endpoint to the new one.

Test posts from Send test are signed the same way, so they are a quick way to check your endpoint. Slack, Discord and Pushover channels are not signed, because those services receive the posts themselves.

Delivery and visibility

SaneLog waits up to 8 seconds for an answer and does not follow redirects. A Slack, Discord, Pushover or webhook delivery that gets no answer, or an answer of 429 or 5xx, is tried twice more, after 10 seconds and then after a minute. Any other answer is not retried. An endpoint that received a notification but answered too late can receive it twice, so a receiver that must act only once should check whether it has already handled an identical post.

When a notification still fails, the channel list marks the channel Last notification failed, and the channel's page says when and why. Send test checks the channel again.

Webhook URLs and Pushover keys are shown masked in the channel list, on the alerts that use the channel and on the channel's page. On the channel's page, the eye button beside the destination shows the full value to users who can edit channels. Read Only users can only see the masked value.

Channels on an alert

An alert can have several channels, and every one receives every notification.

An alert with no channel is still checked and still records every incident in full. This is useful while you tune a threshold before anyone is notified.

How alerts fire

Evaluation

Every alert is checked once a minute against the events stored at that moment. Each check counts the filter's matches and decides whether the condition holds. The Log Explorer does not need to be open.

Transitions, not repeats

A notification is sent when the condition starts to hold. That opens an incident, and the alert shows as Open. An alert that stays open across many checks sends one notification, not one per minute.

When the condition stops holding, the incident is recorded as resolved and the alert returns to OK.

To hear when an incident is over, turn on Notify the channels when an incident resolves. The channels then receive a second notification, titled "Alert resolved:" and the alert's name, saying how long the incident lasted and what ended it. For Matching events stop and An expected event is late alerts it shows the line that came back. A new alert of those two types starts with the switch on, and a Matching events occur alert starts with it off, because a busy filter can cross its threshold back and forth.

A resolution is only sent for an incident whose own notification was sent. An incident that opened while the alert was snoozed or muted, and never re-notified, resolves without a notification. A resolution that happens while the alert is snoozed or muted is not sent either. Pushover delivers a resolution at normal priority, whatever the channel's priority.

To get reminders while an incident stays open, set Re-notify while open to an interval (15 or 30 minutes, 1 hour, 6 hours or 1 day). The alert then sends the notification again at that interval until the incident resolves. Never, the default, sends one notification per incident.

Every matching event alerts work differently: they have no incidents and deliver every match in batches.

Testing, snoozing and muting

The alert's page has these controls:

• Send test sends a test notification to every channel on the alert, so you can check delivery before the alert fires for real.
• Snooze stops notifications for 1 hour or 24 hours, after which they start again by themselves. Clear snooze ends a snooze early.
• Mute stops notifications until you choose Unmute.

While an alert is snoozed or muted it is still checked and every incident is still recorded. An incident that opened during that time sends nothing when the snooze or mute ends, unless Re-notify while open is set. In that case its reminders resume, and its resolution is sent too.

Users with Read Only access can see alerts but cannot create, change, test, snooze, mute or delete them.

What a notification carries

• The alert's name and a sentence describing its condition ("at least 50 matches within 5 minutes").
• What happened, compared with what the filter usually does: "That is about 16 times its usual rate for 15 minutes", "usually every 23 hours", "Of the previous 14 expected occurrences in the last 30 days, all arrived within 6 minutes of the expected time". The comparison uses the filter's history at the moment the alert fired: the last 7 days for Matching events occur, and the last 30 days for the other two types, limited to your plan's retention. When the filter has no earlier matches in that time, the notification says so instead of giving a usual rate.
• The number of matching events and up to 5 recent matching lines. An alert about something that stopped or is late shows the last matching line instead.
• A link to the incident's page, and a link to the Log Explorer with the alert's filter applied.
• A line naming the incident for a coding agent connected to the MCP server - see Handing an incident to an agent.

Every incident is kept

Each incident is recorded in full when it opens, with everything the notification says, whether or not the alert has a channel. Each incident has its own page, which you can open from a notification, from the alert's history or from the dashboard.

• The dashboard lists every incident of the last 24 hours under Incidents - last 24 h, newest first, marked open or resolved, with one line on what happened. A backup that missed at 2am and ran at 5am is still listed in the morning, marked resolved.
• The dashboard's Needs attention card lists every alert that is open right now, unless it is muted or snoozed. An open Matching events stop alert there says when the filter last matched ("quiet since ..."). Two such alerts watching the same filter with different windows then show the same time, which tells you they are one outage.
• The alert's page shows its state, when it last triggered and was last checked, and its History of triggered and resolved entries. View incident on a triggered entry opens that incident.
• The incident's page describes what happened, the alert and its condition, when the incident was triggered and resolved, and the matching lines as the Log Explorer shows them. Open in Log Explorer opens the Log Explorer at the moment the incident opened. Once the incident is resolved, the page says how long it lasted. For a Matching events stop or An expected event is late alert it also shows the line that resolved it - the backup that finally ran, or the heartbeat that came back.

Handing an incident to an agent

Copy for agent on an incident's page copies the whole incident as text a coding agent can read: what happened, the lines the incident carried, the events around that moment across every source, and links. An alert email names the incident by its uuid for a coding agent connected to the MCP server, so a connected agent can be given the incident straight from an inbox.

Agent handoffs describes what a handoff holds. A script with an API token can read the list of incidents directly, and each incident there carries the same description.

Editing

Saving an alert never sends a notification or resolves an open incident by itself, whatever you changed. The next check, within a minute, applies the alert as saved:

• An open incident whose new condition no longer holds is recorded as resolved.
• An alert in the OK state whose new condition holds fires as usual.
• An open incident whose new condition still holds stays open, and no new notification is sent.

When the data store is unavailable

If the stored events cannot be read when an alert is checked, that check is skipped. It is not treated as "no matches". An outage on SaneLog's side therefore never fires a Matching events stop or An expected event is late alert, and never resolves an open incident.

A check is skipped in the same way when the alert's filter currently covers no sources, for example when the only source it covers has been deleted.