# Redaction

> Finding personal data and credentials in log lines as they arrive, counting them, and replacing them before the lines are stored.

## What redaction does

Redaction finds personal data and credentials in log lines as they arrive. It can count them, and it can replace them before the line is stored.

Applications often write this data into their logs without meaning to: an email address in a "user logged in" line, a card number in a failed-payment dump, a bearer token in a request log, a private key in a stack trace.

Redaction recognises formats, not meaning. A detector recognises an email address, a card number that passes its checksum, or a token with a vendor prefix. Names, street addresses and free-text identifiers are not detected. The **Redaction** page describes what each detector finds.

Redaction applies to the whole account. An Account Admin sets it under **Account > Redaction**.

## The three modes

Choose a mode under **Mode** on the **Redaction** page, then choose **Save**.

- **Off** - no detector runs, and log lines are stored as they arrive.
- **Detect only** - every match is counted, on the **Redaction** page and on each source's page. Log lines are stored as they arrive. Every new account starts in this mode.
- **Redact** - every match is replaced before the line is stored, shown in the Log Explorer or read through the API. The original text cannot be recovered. This mode is included in the Business and Enterprise plans, and on other plans it cannot be chosen.

Changing the mode only affects lines that arrive afterwards. Logs already stored keep their text until the account's retention period removes them.

## The detectors

Each detector is a tickbox on the **Redaction** page, in two groups: **Personal data** and **Secrets and credentials**. While the mode is **Off**, the tickboxes cannot be changed.

In **Redact** mode, a match is replaced by the token in the table. The key names the detector in a [query expression](https://app.sanelog.com/docs/redaction.md#seeing-the-lines), and the last column shows which detectors a new account starts with.

| Detector | Replaced by | Key | On for a new account |
|---|---|---|---|
| **Email addresses** | `[email]` | `email` | Yes |
| **Payment card numbers** | `[card]` | `card` | Yes |
| **IBAN account numbers** | `[iban]` | `iban` | Yes |
| **International phone numbers** | `[phone]` | `phone` | Yes |
| **Australian tax file numbers** | `[tfn]` | `au_tfn` | Yes |
| **Australian Medicare numbers** | `[medicare]` | `au_medicare` | Yes |
| **US social security numbers** | `[ssn]` | `us_ssn` | No |
| **UK national insurance numbers** | `[nino]` | `uk_nino` | No |
| **Private key blocks** | `[private-key]` | `private_key` | Yes |
| **Bearer tokens** | `[token]` | `bearer` | Yes |
| **JSON web tokens** | `[jwt]` | `jwt` | Yes |
| **AWS access key ids** | `[aws-key]` | `aws_key` | Yes |
| **Vendor API keys** | `[api-key]` | `vendor_key` | Yes |
| **Values of secret-shaped keys** | `[secret]` | `secret_value` | Yes |

Some detectors check more than the shape of a value, so that ordinary numbers and names are left alone:

- Card numbers must carry a known issuer prefix and pass the Luhn check. A number stored in an attribute whose name ends in `id` or `ids` is never treated as a card number.
- IBAN, tax file and Medicare numbers must pass their check digit.
- Phone numbers must start with a plus sign. Local formats without one are not detected.
- Email addresses are not matched in message ids (`Message-ID: <...>`) or in systemd unit names such as `user@1000.service`.
- **Bearer tokens** replaces the token after the word Bearer, and the word stays.
- **Vendor API keys** recognises keys by their prefix, from Stripe, Anthropic, OpenAI, Hugging Face, GitHub, GitLab, Slack, Google, SendGrid, npm, PyPI, DigitalOcean, Vault, Grafana, Sentry, Shopify, Square and others.
- **Values of secret-shaped keys** replaces the value after a key such as `password`, `secret`, `token` or `api_key`, both in the line (`password=...`) and in an attribute with that name. The key stays.

In **Redact** mode, two detectors offer an option that keeps part of the value in the line:

- **Keep the last four digits** under **Payment card numbers** replaces a card number with `[card *1234]`.
- **Keep the domain** under **Email addresses** replaces an address with `[email]@example.com`.

## What a redacted line looks like

A match is replaced in place by its token, so the rest of the line stays as it was:

```
Payment failed for jo@example.com with card 4111 1111 1111 1111 from +61 412 345 678
```

becomes

```
Payment failed for [email] with card [card] from [phone]
```

You can search for a token in the Log Explorer. Searching for `[card]` lists every line that contained a card number, which is often the quickest way to find the code that logs one.

## Counting

Every match is counted per source and per detector, in both **Detect only** and **Redact** mode.

- The **Redaction** page shows the calendar month's matches by detector, with **Seen** (matches in lines stored as they arrived) and **Replaced** (matches replaced before the line was stored). Below that, **Sources with matches** lists every source with a match this month.
- Each source's page shows that source's matches for the month by detector, under **Redaction matches (this month)**. The card is hidden while the mode is **Off**.

The counts show which application logs personal data. Changing that application so it no longer logs the data is the only complete fix.

## Seeing the lines

Every line records which detectors matched it, so you can go from a count to the lines behind it. **Show lines** beside a detector's count opens the Log Explorer on the lines that detector matched. **Show lines** beside a source opens the lines from that source that any detector matched.

You can write the same condition yourself in the Log Explorer's advanced search or in a filter:

- `redaction is present` - lines any detector matched.
- `redaction is email` - lines the email detector matched.
- `redaction is any of email, card` - lines either detector matched.
- `redaction is none of secret_value` - lines the secret-value detector did not match.

Detectors are named by the keys in the [detectors table](https://app.sanelog.com/docs/redaction.md#the-detectors). To be alerted when personal data appears from any source, build a filter with `redaction is present` and add an alert to it. The condition also covers detectors added later.

In **Redact** mode, a matched line shows the token where the value was. In **Detect only** mode, it shows the value as it arrived, which helps a developer find the code that logged it.

A line only records the detectors that ran when it arrived. Lines stored while the mode was **Off**, or before a detector was ticked, are not found by these conditions.

## What is not detected

- **Names, street addresses and free-text identifiers.** In a log line, these cannot be told apart from usernames, hostnames and other identifiers.
- **IP addresses.** An address in a log line is often needed to trace a problem, and sources, sender rules and the location shown beside an address all depend on it.
- **The event's own fields.** The sending address, the program and the level are never changed. Only the message and the attribute values are checked.