Log Analysis

What Log Analysis shows

Open Log Analysis in the sidebar to see every line the account received over a window, grouped by the message patterns behind the lines and compared with the window of the same length before it. A week of hundreds of thousands of lines fits on one page.

Only lines from approved sources are counted. Lines from a source waiting for approval are left out until the source is approved.

The window ends at the start of the current minute, and the page can take up to another minute to include the newest lines.

Patterns and clusters

Each line's message is reduced to its pattern by replacing the parts that change from line to line with ?:

• text in quotes
• UUIDs
• IPv4 addresses
• runs of eight or more hexadecimal characters, such as hashes and ids
• any remaining numbers

So "Connection timeout to redis:6379 after 4632ms" and "Connection timeout to redis:6380 after 12ms" count as one pattern, "Connection timeout to redis:? after ?ms".

Patterns that start with the same words are grouped into one cluster. "connection failed for ? bad command" and "connection failed for ? bad sequence" form one cluster labelled "connection failed for ? bad ...". The cluster lists both patterns as variants, each with its own count. A cluster of one pattern is labelled by that pattern.

Patterns join a cluster only when they share at least their first three words, and those shared words make up at least half of the shorter pattern. "Connection failed" and "Connection established" share only one word, so they stay apart.

Reading the page

The page opens on the lines at Warning and above over the last 7 days, one row per program. Use the controls at the top to choose what the page covers:

• Selection - All sources, one source, or a filter (listed as Filter: followed by its name).
• Levels - Warning and above (the default), Error and above, or All levels.
• Window - 24 hours, or 7 days (the default).
• Group by - Program (the default) or Source.
• Order - Most lines (the default), or Most severe (the most lines at warning and above).

Every control is kept in the page's address, so you can share any view as a link.

The summary

The strip under the controls describes the whole window:

• Lines - every line counted, with the previous window's count and the change as a percentage. Shown only when Levels is All levels.
• Warning and above (or Error and above) - the lines at that level or a more severe one, with the previous window's count and the change as a percentage.
• Programs or Sources - how many had lines in the window.
• Patterns - the number of clusters.

The table

Below the summary, each program (or source) with lines in the window is one row showing its line count, its change against the previous window, a bar of its levels, a small chart of its lines across the window, and its top pattern.

Choose the arrow at the start of a row to show the program's five biggest clusters, or its five most severe with Most severe picked. The first row starts open. When a program has more clusters, a link under them opens the program's own page.

Each cluster row shows:

• Pattern - the cluster's label, with the number of variants when there is more than one.
• Lines - the lines in the window.
• Change - the change against the previous window (see below).
• Levels - a bar of the cluster's levels: error and worse in red, warning in amber, notice in blue, info and debug in grey. Point at the bar to see the counts.
• Trend - a small chart of the cluster's lines across the window, drawn in red when the cluster has any lines at error or above.
• Sources - the first two sources the cluster appears on and how many more. Point at the list to see them all. Grouped by source, the column is left out.

Only patterns and programs with lines in the window are listed. A pattern that had lines in the previous window and none in this one is left out, since there is nothing to look at. To be told when a line that should keep arriving goes quiet, such as a backup that reports when it finishes, set up a Matching events stop alert.

Change

The Change column reads:

• new when the cluster had no lines in the previous window
• rising with a percentage when it grew by at least 10 lines and at least doubled, or as a multiple such as "763×" once it grew tenfold
• falling with a percentage when it shrank by at least 10 lines and at least halved
• the percentage alone for any smaller change

Zooming in

Click a program's name to open its page, which lists every cluster in the program, not only the biggest. Click a cluster's label to open the cluster's page, which shows:

• its change, lines, lines in the previous window and lines at warning and above
• the sources it appears on, and when it was first and last seen
• its levels, with a count for each
• a chart of its lines over the window
• the Newest line in the cluster
• every pattern in the cluster, with its own lines, change, levels and last seen time

The trail in the top bar leads back to the program and to the overview.

Changing Window, Group by, Levels or Selection returns to the overview, since the programs and clusters of one view are not those of another.

A program can have more lines than its listed clusters add up to. Patterns too rare to list are counted in a note under the table, such as "120 more lines fall under patterns too rare to list", so a program's count always covers all of its lines.

Opening the lines

Choose Open in Log Explorer to see the lines behind a count. It is the button on a program's (or source's) page and on a cluster's page, and the icon at the end of each program, cluster and pattern row. The Log Explorer opens with the same Selection and Levels as the page, plus:

• for a program, that program picked in the program picker
• for a source, that source picked in the source picker
• for a cluster, an advanced search expression naming the cluster's patterns
• for a single pattern, an expression with pattern is and that pattern

When the page is grouped by program, a cluster's expression also names the program the cluster was counted in (grouped by source, the source is picked instead):

program is imap and pattern is any of "connection failed for ? bad command", "connection failed for ? bad sequence"

A pattern condition can also be saved in a filter. pattern is none of in a filter keeps a noisy pattern out of that filter. Query expressions describes the pattern field.

Reports and agents

The log analysis report is this page for the previous day or week, sent by email: the lines and lines at warning and above, the clusters that are new, the biggest risers and fallers, then each program with its biggest clusters.

A coding agent can read the same tree through the summary call of the API reference, or the same tool of a connected agent.

The page, the report and the summary call count lines the same way, so for the same window and selection their numbers agree.