Redaction

What redaction does

Redaction finds personal data and credentials in log lines as they arrive. It can count them, and it can replace them before the line is stored.

Applications often write this data into their logs without meaning to: an email address in a "user logged in" line, a card number in a failed-payment dump, a bearer token in a request log, a private key in a stack trace.

Redaction recognises formats, not meaning. A detector recognises an email address, a card number that passes its checksum, or a token with a vendor prefix. Names, street addresses and free-text identifiers are not detected. The Redaction page describes what each detector finds.

Redaction applies to the whole account. An Account Admin sets it under Account > Redaction.

The three modes

Choose a mode under Mode on the Redaction page, then choose Save.

• Off - no detector runs, and log lines are stored as they arrive.
• Detect only - every match is counted, on the Redaction page and on each source's page. Log lines are stored as they arrive. Every new account starts in this mode.
• Redact - every match is replaced before the line is stored, shown in the Log Explorer or read through the API. The original text cannot be recovered. This mode is included in the Business and Enterprise plans, and on other plans it cannot be chosen.

Changing the mode only affects lines that arrive afterwards. Logs already stored keep their text until the account's retention period removes them.

The detectors

Each detector is a tickbox on the Redaction page, in two groups: Personal data and Secrets and credentials. While the mode is Off, the tickboxes cannot be changed.

In Redact mode, a match is replaced by the token in the table. The key names the detector in a query expression, and the last column shows which detectors a new account starts with.

Some detectors check more than the shape of a value, so that ordinary numbers and names are left alone:

• Card numbers must carry a known issuer prefix and pass the Luhn check. A number stored in an attribute whose name ends in id or ids is never treated as a card number.
• IBAN, tax file and Medicare numbers must pass their check digit.
• Phone numbers must start with a plus sign. Local formats without one are not detected.
• Email addresses are not matched in message ids (Message-ID: <...>) or in systemd unit names such as user@1000.service.
• Bearer tokens replaces the token after the word Bearer, and the word stays.
• Vendor API keys recognises keys by their prefix, from Stripe, Anthropic, OpenAI, Hugging Face, GitHub, GitLab, Slack, Google, SendGrid, npm, PyPI, DigitalOcean, Vault, Grafana, Sentry, Shopify, Square and others.
• Values of secret-shaped keys replaces the value after a key such as password, secret, token or api_key, both in the line (password=...) and in an attribute with that name. The key stays.

In Redact mode, two detectors offer an option that keeps part of the value in the line:

• Keep the last four digits under Payment card numbers replaces a card number with [card *1234].
• Keep the domain under Email addresses replaces an address with [email]@example.com.

What a redacted line looks like

A match is replaced in place by its token, so the rest of the line stays as it was:

Payment failed for jo@example.com with card 4111 1111 1111 1111 from +61 412 345 678

becomes

Payment failed for [email] with card [card] from [phone]

You can search for a token in the Log Explorer. Searching for [card] lists every line that contained a card number, which is often the quickest way to find the code that logs one.

Counting

Every match is counted per source and per detector, in both Detect only and Redact mode.

• The Redaction page shows the calendar month's matches by detector, with Seen (matches in lines stored as they arrived) and Replaced (matches replaced before the line was stored). Below that, Sources with matches lists every source with a match this month.
• Each source's page shows that source's matches for the month by detector, under Redaction matches (this month). The card is hidden while the mode is Off.

The counts show which application logs personal data. Changing that application so it no longer logs the data is the only complete fix.

Seeing the lines

Every line records which detectors matched it, so you can go from a count to the lines behind it. Show lines beside a detector's count opens the Log Explorer on the lines that detector matched. Show lines beside a source opens the lines from that source that any detector matched.

You can write the same condition yourself in the Log Explorer's advanced search or in a filter:

• redaction is present - lines any detector matched.
• redaction is email - lines the email detector matched.
• redaction is any of email, card - lines either detector matched.
• redaction is none of secret_value - lines the secret-value detector did not match.

Detectors are named by the keys in the detectors table. To be alerted when personal data appears from any source, build a filter with redaction is present and add an alert to it. The condition also covers detectors added later.

In Redact mode, a matched line shows the token where the value was. In Detect only mode, it shows the value as it arrived, which helps a developer find the code that logged it.

A line only records the detectors that ran when it arrived. Lines stored while the mode was Off, or before a detector was ticked, are not found by these conditions.

What is not detected

• Names, street addresses and free-text identifiers. In a log line, these cannot be told apart from usernames, hostnames and other identifiers.
• IP addresses. An address in a log line is often needed to trace a problem, and sources, sender rules and the location shown beside an address all depend on it.
• The event's own fields. The sending address, the program and the level are never changed. Only the message and the attribute values are checked.