FAQ

Log Explorer

The live stream shows events out of order, or with timestamps a few seconds behind the clock

The live stream shows events in the order they arrive at SaneLog, not in timestamp order, unless it is set to hold them (below). Each event's timestamp is the time the sending system recorded it, so a line can carry a timestamp that is several seconds older than the line above it.

Both come from the same cause: events do not all take the same time to travel. Every server queues and sends its logs in its own batches, so when many servers log at once, such as cron jobs firing on the hour, one server's batch can arrive a few seconds after another's. The live stream adds each line the moment it lands, so every new line shows at the bottom where you are watching, and lines from different servers can be out of timestamp order for a few seconds.

Nothing is lost or misfiled. Every event is stored with its original timestamp, and whenever the stream is loaded from storage rather than watched live, the events appear in timestamp order. That covers scrolling up, Go to time, Show surrounding events, reconnecting after a pause, and every filter, alert and API query.

To watch the live stream in timestamp order, open Display settings (the gear on the Log Explorer bar) and set Live tail order to hold lines for 1, 2 or 5 seconds. Each new line then waits that long and is placed among the lines on screen by its timestamp. The live stream then runs that far behind. Two seconds puts nearly every line in order.

A server that runs ten seconds or more behind is not put in order by the hold. Its lines land where their timestamps put them, higher up the stream. Such a server is queuing or batching its logs, or its clock is slow. Its arrival lag is shown on its page and in the sources list, and the dashboard's Needs attention list names it for as long as the lag lasts. A server whose timestamps run ahead of the clock is named the same way.

Sending logs

A source named after an IP address appeared, waiting for approval, and nobody set it up

Every account's syslog port can be reached from the whole internet, and internet-wide scanning services (Censys, Shodan and the like) connect to every open port they find to see what answers. A scanner completes the secure connection, then sends web, WebSocket and MQTT requests to find out what kind of service is listening. Those requests have no syslog header, so a line from one is attributed to the address it came from, and that is the name the source appears under.

Nothing was compromised. The scanner learned only that a secure service answers on that port and that it is not a web server. It could not read any logs. A source waiting for approval is left out of the Log Explorer's All sources view, filters and alerts, and its lines expire after seven days.

SaneLog drops lines from the addresses the major scanning services publish, and a line with no syslog header from an address that is not approved does not create a source, so these rows are rare. When one does appear, Reject removes the source, deletes its stored lines (the option on the confirmation is ticked by default) and drops any further lines from that address. Approving senders describes both choices.

If the address is one of the account's own machines, that machine is not sending syslog format at all. Approve keeps its lines under the source named after the address while its configuration is fixed.

My servers run syslog-ng, not rsyslog

syslog-ng works. The Sources page shows ready-made configurations for rsyslog only, because rsyslog is the default on most Linux distributions, but SaneLog accepts the same syslog from syslog-ng on the same hostname and port.

Save the following as /etc/syslog-ng/conf.d/sanelog.conf, put the hostname and port from the Sources page in place of logs.example.com and 24567, and restart syslog-ng:

# Send the full hostname so machines sharing a short name stay separate
# sources. This is a global option, so local log files show the full
# hostname too.
options { use-fqdn(yes); };

destination d_sanelog {
    syslog("logs.example.com" port(24567)
        transport("tls")
        tls(ca-dir("/etc/ssl/certs") peer-verify(required-trusted))
        so-keepalive(yes)
        tcp-keepalive-time(60) tcp-keepalive-intvl(10) tcp-keepalive-probes(3)
        disk-buffer(disk-buf-size(104857600) reliable(no) dir("/var/lib/syslog-ng"))
    );
};

log { source(s_src); destination(d_sanelog); };

This sends every line over TLS, checked against the server's own list of trusted certificate authorities. While the connection is down, up to 100 MB of lines wait on disk and are sent when the connection returns. A dead connection is noticed within about 90 seconds.

s_src is the name of the system log source on Debian and Ubuntu. On RHEL, Fedora and SUSE the name is s_sys. Use the name your main syslog-ng.conf defines.

One difference from the rsyslog configuration: syslog-ng looks up the SaneLog hostname again only when it reconnects, while the rsyslog configuration reconnects every 5,000 messages. A change of address on the SaneLog side reaches a syslog-ng sender at its next reconnect, or when syslog-ng is restarted.

How do I send logs from Windows servers?

Windows has no built-in syslog sender, so a Windows server needs a log forwarder installed. NXLog Community Edition is free and widely used, and any forwarder that sends syslog over TLS works the same way.

Set the forwarder up with:

• the hostname and port from the Sources page
• TLS, trusting the Let's Encrypt root certificate (ISRG Root X1), which SaneLog's certificate is currently issued under
• RFC 5424 syslog, with octet-counted framing (NXLog calls this output type Syslog_TLS)
• the Windows event logs to send as its input

Each Windows server becomes a source the first time one of its lines arrives, named after its hostname, like any other syslog sender.

For an application running on Windows, sending over HTTPS with a source token is often simpler than installing a forwarder. See HTTP.

AWS

How do I send logs from AWS?

AWS logs reach SaneLog with a source's token, the same way an application or a script sends. Two kinds of AWS data can be sent:

• CloudWatch Logs (Lambda output, RDS and Aurora logs, ECS containers, CloudTrail trails) is forwarded by a small function deployed into the AWS account.
• EventBridge service events (an RDS failover, a GuardDuty finding, a scheduled maintenance notice) are delivered with no code, through an API destination.

The AWS section has the setup for both, step by step in the AWS console or with the AWS CLI, with the forwarding function's code ready to paste.

An Aurora Serverless v2 error log floods the source with InnoDB notes

An Aurora MySQL Serverless v2 cluster on the default parameter group writes an InnoDB note to its error log every time its capacity changes. On a busy cluster that is tens of thousands of lines a day, which can take the source over its flood limit and crowd out the warnings and errors that matter.

Setting log_error_verbosity to 2 keeps warnings and errors and leaves out the notes. The setting lives in a DB instance parameter group, not the cluster parameter group, and the default group cannot be edited, so the cluster needs a custom one:

1. Open RDS > Parameter groups > Create parameter group. Choose the type DB parameter group, the family that matches the cluster's engine version (aurora-mysql8.0 for Aurora MySQL 3), and a name such as <cluster>-instances.
2. Open the new group, choose Edit, search for log_error_verbosity, set it to 2 and save.
3. For every instance in the cluster, open the instance and choose Modify. Pick the new group under DB parameter group, choose Continue, choose Apply immediately and confirm.
4. Reboot each instance once. A newly attached parameter group takes effect at the next reboot, and the instance shows pending-reboot until then. Later changes to a value inside an attached group apply at once, with no reboot.

Any reader or replacement instance added to the cluster later needs the same parameter group, or the notes come back.

Plans and billing

What happens when the free period ends?

A new account holds the plan chosen at signup free for 14 days, with no card. If no card has been added by the end of the last free day, SaneLog stops storing new logs for the account. Everything already stored stays, and the account's users can still log in and search it.

The account is kept for 30 days from the day the free period ended. Choosing Set up billing on the dashboard or the Plan page and adding a card at any point in those 30 days starts storing logs again at once, with everything in place. An account with no subscription by the 30th day is closed permanently. The Plan page and the dashboard show the closing date.

After a plan ends has the details.

What happens when the account sends more than the plan's monthly allowance?

On a paid plan, nothing past the allowance is dropped and nothing extra is charged. The bill is always the plan's price, plus any tax that applies. The allowance is set well above what a typical account sends, and an account that keeps sending more than its allowance month after month is asked to move to a larger plan.

During the 14 free days the allowance is a hard limit. Logs past it are not stored until the next month begins or the account moves to a paid plan, and the Plan page, the dashboard and the Sources page say so while it is happening. Log allowance has the details.