AWS

Overview

SaneLog can take two kinds of data from an AWS account. Each has its own setup, and you can use either one or both.

• CloudWatch Logs holds what your workloads write: Lambda function output, RDS and Aurora error and slow query logs, ECS and Fargate container output, API Gateway execution logs, VPC flow logs, and CloudTrail trails delivered to a log group. You deploy a small forwarding function into your account. It subscribes to the log groups you choose and posts their lines to SaneLog.
• EventBridge carries what AWS reports about the account. Examples are an RDS failover or maintenance notice, a GuardDuty finding, an AWS Health event, a CloudWatch alarm changing state, a backup job result, a certificate about to expire and a console sign-in. These events never appear in CloudWatch Logs. You deliver them with an EventBridge API destination pointed at SaneLog, with no code to deploy.

Set up CloudWatch Logs for application, container and database logs. Set up EventBridge for what AWS reports about the account. The two setups are independent, and each has its own section below.

Both setups send with a source's token. Use a separate source for each, so log lines and service events stay apart in the Log Explorer and each can have its own config set (which holds the flood limit), filters and alerts.

If your workloads run in several regions, repeat the CloudWatch Logs steps in each region that has log groups, and the EventBridge steps in each region whose events you want.

Every AWS step below is given twice, once for the AWS console and once for the AWS CLI. Either path works on its own. The CLI commands assume a configured profile with permission to manage IAM roles, Lambda, CloudWatch Logs and EventBridge. They use <placeholders> for the values you fill in, such as your account id, region, token and resource names.

Before you start

1. Create two sources in SaneLog under Sources > All sources > Add source: one for the log lines and one for the service events. SaneLog issues each source's token when you create it, and the token is shown in the Source token card on the source's page whenever you need it. Naming the sources after the AWS account (for example payments-aws-logs and payments-aws-events) keeps them apart from your servers in the sources list.
2. Pick the region your workloads run in. The CloudWatch Logs setup lives entirely in that region. The EventBridge setup lives there too, and is repeated in us-east-1 for the services AWS runs globally.
3. Note the two SaneLog addresses. The forwarder posts to https://app.sanelog.com/ingest, and EventBridge posts to https://app.sanelog.com/ingest/aws-events.

CloudWatch Logs

How the forwarder works

CloudWatch Logs invokes the forwarder through subscription filters, within seconds of a line being written. Each invocation carries one compressed batch of log lines. The function turns each line into a SaneLog event and posts the events to SaneLog with the log source's token.

Each event carries:

Event field	Value
Message	The log line, with trailing whitespace removed. Blank lines are skipped.
Level	Read from the line. A line that is a JSON object uses its level, severity or log_level key. Any other line uses the first severity word in its first 200 characters (such as WARN, ERROR or FATAL). A line with neither arrives as info.
Program	The last part of the log group name, so /aws/lambda/checkout becomes checkout.
Time	The time CloudWatch Logs recorded for the line.
Attributes	log_group and log_stream, so you can search by the full origin.

The function stays within SaneLog's request limits by itself. It sends at most 500 events per request and splits any request that would be too large to accept.

When a request fails, what happens next depends on the reason:

• SaneLog is busy or unavailable (a throttle, a server error or a network failure). The function tries the request again up to three more times within the same invocation, waiting a little longer each time. If it still fails, the function fails the invocation and Lambda runs it again with the same batch. Lambda retries a failed invocation twice by default, so a batch that still fails after those retries is lost.
• The source's flood limit is reached. SaneLog has already dropped the batch's events and counts them under Dropped events (last 24 hours) on the source's page. The function writes the response to its own log group and does not retry.
• SaneLog rejects the batch (a wrong or revoked token, a paused source, or an account that has ended its free period or used its monthly allowance). The function writes SaneLog's response to its own log group and drops the batch, because a retry would fail the same way.

To keep batches that fail every retry, add an on-failure destination (an SQS queue) under the function's Configuration > Asynchronous invocation. Lambda then sends each lost batch there.

One function can serve any number of log groups. Deploy a second copy with a different token only when some log groups should arrive in a different source.

The function code

Save this code as index.mjs. It has no dependencies and runs on Node.js 22 or any later Lambda runtime. You use the same file whether you create the function in the console or with the CLI.

// SaneLog CloudWatch Logs forwarder.
//
// Deployed as an AWS Lambda (nodejs22.x, no dependencies) that CloudWatch
// Logs invokes via subscription filters. Each invocation carries one gzipped
// batch of log events; the handler decodes it, maps the events onto SaneLog's
// ingest envelope, and forwards them to POST /ingest as gzipped NDJSON.
// Setup and behaviour: https://app.sanelog.com/docs/aws
//
// Environment:
//   SANELOG_INGEST_URL    e.g. https://app.sanelog.com/ingest
//   SANELOG_SOURCE_TOKEN  the source's slt_... bearer token

import { createHash } from 'node:crypto';
import { gunzipSync, gzipSync } from 'node:zlib';

// SaneLog ingest limits: 500 events per batch, 1 MiB request
// body, 5 MiB decoded. The uncompressed budget stays far below the decoded
// cap so a batch never straddles it; post() re-splits if the compressed body
// still lands over the wire cap.
const MAX_BATCH_EVENTS = 500;
const MAX_UNCOMPRESSED_BYTES = 2 * 1024 * 1024;
const MAX_COMPRESSED_BYTES = 1000 * 1000;

// Transient failures (a throttle, a 5xx, a network error) are retried inside
// the invocation first, so a whole-batch redelivery by Lambda - which would
// post every chunk again - is the exception. Waits double from the base (up
// to the cap), a Retry-After from SaneLog is honoured as given, and a retry is
// only attempted while the invocation has time left for it.
const MAX_ATTEMPTS = 4;
const BASE_DELAY_MS = 1000;
const MAX_DELAY_MS = 10000;
const TIME_MARGIN_MS = 5000;

// Chunks this container has delivered, by content hash. When Lambda
// redelivers a batch after a later chunk failed, the retry skips the chunks
// that already landed - best effort, since a retry can run on a different
// container.
const DELIVERED_TTL_MS = 30 * 60 * 1000;
const delivered = new Map();

const LEVELS = new Set(['debug', 'info', 'notice', 'warning', 'error', 'critical', 'alert', 'emergency']);

const LEVEL_ALIASES = {
  trace: 'debug',
  warn: 'warning',
  err: 'error',
  crit: 'critical',
  fatal: 'critical',
  emerg: 'emergency',
  panic: 'emergency',
};

export async function handler(event, context) {
  if (!event?.awslogs?.data) {
    return;
  }

  const batch = JSON.parse(gunzipSync(Buffer.from(event.awslogs.data, 'base64')).toString('utf8'));

  // CONTROL_MESSAGE is the subscription handshake probe, not log data.
  if (batch.messageType !== 'DATA_MESSAGE') {
    return;
  }

  const program = programFor(batch.logGroup);

  const lines = [];

  for (const logEvent of batch.logEvents ?? []) {
    const message = String(logEvent.message ?? '').trimEnd();

    if (message === '') {
      continue;
    }

    lines.push(JSON.stringify({
      message,
      level: detectLevel(message),
      program,
      dt: new Date(logEvent.timestamp).toISOString(),
      attributes: { log_group: batch.logGroup, log_stream: batch.logStream },
    }));
  }

  const timeLeft = () => context?.getRemainingTimeInMillis?.() ?? Infinity;

  for (const chunk of chunked(lines)) {
    await post(chunk, timeLeft);
  }
}

// The last path segment of the log group ("/aws/lambda/checkout" -> "checkout")
// keeps `program` low-cardinality and human-sized; the full group and stream
// ride along as attributes.
function programFor(logGroup) {
  const segments = String(logGroup ?? '').split('/').filter(Boolean);

  return segments[segments.length - 1] ?? 'cloudwatch';
}

// Structured JSON lines carry their own level; for plain text, the first
// recognisable severity token wins. Anything else lands as info (which is
// also what the server coerces unknown levels to).
function detectLevel(message) {
  if (message.startsWith('{')) {
    try {
      const parsed = JSON.parse(message);
      const level = normaliseLevel(parsed.level ?? parsed.severity ?? parsed.log_level);

      if (level) {
        return level;
      }
    } catch {
      // Not JSON after all - fall through to the token scan.
    }
  }

  const match = message.slice(0, 200).match(
    /\b(TRACE|DEBUG|INFO|NOTICE|WARN(?:ING)?|ERR(?:OR)?|CRIT(?:ICAL)?|FATAL|ALERT|EMERG(?:ENCY)?|PANIC)\b/i,
  );

  return (match && normaliseLevel(match[1])) || 'info';
}

function normaliseLevel(value) {
  if (typeof value !== 'string') {
    return null;
  }

  const lower = value.toLowerCase();

  if (LEVELS.has(lower)) {
    return lower;
  }

  return LEVEL_ALIASES[lower] ?? null;
}

function* chunked(lines) {
  let current = [];
  let bytes = 0;

  for (const line of lines) {
    const size = Buffer.byteLength(line) + 1;

    if (current.length > 0 && (current.length >= MAX_BATCH_EVENTS || bytes + size > MAX_UNCOMPRESSED_BYTES)) {
      yield current;
      current = [];
      bytes = 0;
    }

    current.push(line);
    bytes += size;
  }

  if (current.length > 0) {
    yield current;
  }
}

async function post(lines, timeLeft) {
  const body = gzipSync(Buffer.from(lines.join('\n'), 'utf8'));

  if (body.length > MAX_COMPRESSED_BYTES && lines.length > 1) {
    const half = Math.ceil(lines.length / 2);
    await post(lines.slice(0, half), timeLeft);
    await post(lines.slice(half), timeLeft);

    return;
  }

  const key = createHash('sha256').update(body).digest('hex');
  forgetExpired();

  if (delivered.has(key)) {
    return;
  }

  for (let attempt = 1; ; attempt++) {
    const outcome = await send(body);

    if (outcome.done) {
      delivered.set(key, Date.now());

      return;
    }

    const wait = outcome.retryAfterMs ?? Math.min(MAX_DELAY_MS, BASE_DELAY_MS * 2 ** (attempt - 1));

    // Throwing makes Lambda run the invocation again with the same batch;
    // chunks already delivered from this container are skipped then.
    if (attempt >= MAX_ATTEMPTS || timeLeft() < wait + TIME_MARGIN_MS) {
      throw new Error(`${outcome.reason}; batch will be retried`);
    }

    await new Promise((resolve) => setTimeout(resolve, wait));
  }
}

// One POST. done: true when the chunk needs no further attempt - accepted,
// or refused in a way a retry would repeat (400/401/402/403/413, and a 429
// for the flood limit, whose events SaneLog has already dropped and
// counted). Those refusals are logged to the function's log group.
async function send(body) {
  let response;
  let text;

  try {
    response = await fetch(process.env.SANELOG_INGEST_URL, {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${process.env.SANELOG_SOURCE_TOKEN}`,
        'Content-Type': 'application/x-ndjson',
        'Content-Encoding': 'gzip',
      },
      body,
    });
    text = await response.text();
  } catch (error) {
    return { done: false, reason: `SaneLog ingest could not be reached (${error.message})` };
  }

  if (response.ok) {
    return { done: true };
  }

  if (response.status === 429 && reasonOf(text) === 'flood_limit') {
    console.warn(`SaneLog dropped a batch over the source's flood limit: ${text}`);

    return { done: true };
  }

  if (response.status === 429 || response.status >= 500) {
    const retryAfter = Number(response.headers.get('retry-after'));

    return {
      done: false,
      reason: `SaneLog ingest returned ${response.status}`,
      retryAfterMs: Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1000 : undefined,
    };
  }

  console.error(`SaneLog ingest rejected a batch: ${response.status} ${text}`);

  return { done: true };
}

function reasonOf(text) {
  try {
    return JSON.parse(text).reason ?? null;
  } catch {
    return null;
  }
}

function forgetExpired() {
  const cutoff = Date.now() - DELIVERED_TTL_MS;

  for (const [key, at] of delivered) {
    if (at < cutoff) {
      delivered.delete(key);
    }
  }
}

Step 1: The execution role

The function needs a role that lets it write its own logs and nothing else. It reaches SaneLog over HTTPS, which needs no AWS permission.

Console. Skip this step. Creating the function in step 2 creates this role for you.

CLI. Create the role with a trust policy for Lambda, and attach the managed basic execution policy:

cat > trust-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "lambda.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role \
  --role-name sanelog-forwarder-role \
  --assume-role-policy-document file://trust-policy.json

aws iam attach-role-policy \
  --role-name sanelog-forwarder-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

The create-role output includes the role's ARN, which is the --role value in step 2.

A new role takes a few seconds to become usable. If creating the function fails with "role cannot be assumed", wait ten seconds and run the command again.

Step 2: Create the function

Console. In your workload region, open Lambda > Functions > Create function and choose Author from scratch. Then:

1. Set the function name to sanelog-cloudwatch-forwarder. Keep the runtime the form offers (Node.js 24.x at the time of writing). Any Node.js runtime from 22 up can run the code.
2. Leave everything else as it is. The Permissions note says Lambda will create an execution role that can write the function's logs, which is the role from step 1. Leave the Custom settings toggles and the Additional settings unchanged.
3. Choose Create function. The code editor opens on a file named index.mjs. Replace its contents with the code above and choose Deploy.
4. On Configuration > General configuration, set the timeout to 30 seconds. A batch can hold hundreds of lines, and the default of 3 seconds is not enough to compress and post them.
5. On Configuration > Environment variables, add SANELOG_INGEST_URL with the value https://app.sanelog.com/ingest, and SANELOG_SOURCE_TOKEN with the log source's token.

CLI. Zip the file, then create the function with the role from step 1 and the same settings:

zip function.zip index.mjs

aws lambda create-function \
  --function-name sanelog-cloudwatch-forwarder \
  --runtime nodejs22.x \
  --handler index.handler \
  --zip-file fileb://function.zip \
  --role arn:aws:iam::<account-id>:role/sanelog-forwarder-role \
  --timeout 30 \
  --environment "Variables={SANELOG_INGEST_URL=https://app.sanelog.com/ingest,SANELOG_SOURCE_TOKEN=<token>}"

On Windows without zip, this command creates the same archive:

powershell -Command "Compress-Archive -Path index.mjs -DestinationPath function.zip -Force"

Step 3: Let CloudWatch Logs invoke it

CloudWatch Logs can only invoke a function that grants it permission.

Console. Skip this step. Creating a subscription filter in the console adds the permission for that log group.

CLI. Grant the permission once for every log group in the account and region:

aws lambda add-permission \
  --function-name sanelog-cloudwatch-forwarder \
  --statement-id cloudwatch-logs \
  --principal logs.amazonaws.com \
  --action lambda:InvokeFunction \
  --source-arn "arn:aws:logs:<region>:<account-id>:log-group:*" \
  --source-account <account-id>

Step 4: Subscribe log groups

A subscription filter sends one log group's lines to the function. An empty filter pattern sends every line. You can subscribe log groups one at a time, or cover the whole region at once.

Never subscribe the function's own log group, /aws/lambda/sanelog-cloudwatch-forwarder. The function writes every rejected batch to that log group, so forwarding it would turn each rejection into a new batch, which would be rejected in turn.

Selected log groups, console. Repeat these steps for each log group:

1. Open CloudWatch > Log groups and open the log group.
2. On the Subscription filters tab, choose Create > Create Lambda subscription filter.
3. Under Choose destination, pick sanelog-cloudwatch-forwarder as the Lambda function.
4. Under Configure log format and filters, leave the log format on Other. Leave the Subscription filter pattern empty so every line is forwarded, and set the Subscription filter name to sanelog.
5. Skip the test section and choose Start streaming.

Selected log groups, CLI. Run this once for each log group:

aws logs put-subscription-filter \
  --log-group-name /aws/rds/cluster/<cluster>/error \
  --filter-name sanelog \
  --filter-pattern "" \
  --destination-arn arn:aws:lambda:<region>:<account-id>:function:sanelog-cloudwatch-forwarder

The whole region, CLI only. An account-level subscription policy covers every log group in the region, including log groups created later. Its exclusion list must name the function's own log group:

aws logs put-account-policy \
  --policy-name sanelog-forward-all \
  --policy-type SUBSCRIPTION_FILTER_POLICY \
  --policy-document '{"DestinationArn":"arn:aws:lambda:<region>:<account-id>:function:sanelog-cloudwatch-forwarder","FilterPattern":""}' \
  --selection-criteria 'LogGroupName NOT IN ["/aws/lambda/sanelog-cloudwatch-forwarder"]' \
  --scope ALL

Each run replaces the whole policy. To leave out more log groups (for example noisy ones), add them to the NOT IN list and run the command again.

To see the policy in force, run aws logs describe-account-policies --policy-type SUBSCRIPTION_FILTER_POLICY.

RDS and Aurora logs

An RDS instance or Aurora cluster writes its logs to CloudWatch Logs only when you turn on log exports for it. The Configuration tab of a database in the RDS console shows which logs it exports.

Console.

1. Open the cluster or instance in RDS > Databases and choose Modify.
2. Under Log exports, tick Error log, and any of the slow query, general and audit logs you want.
3. Choose Continue, choose Apply immediately and confirm. Changing log exports does not restart the database.
4. Wait for the log groups to appear under CloudWatch > Log groups: /aws/rds/cluster/<cluster>/error for the error log and /aws/rds/cluster/<cluster>/slowquery for the slow query log. A log group is created the first time its log has something to write, so the slow query log group can appear some time after you turn on the export.
5. Subscribe each log group as in step 4. Each exported log needs its own subscription filter.

CLI.

aws rds modify-db-cluster \
  --db-cluster-identifier <cluster> \
  --cloudwatch-logs-export-configuration '{"EnableLogTypes":["error","slowquery"]}' \
  --apply-immediately

For a standalone instance, use aws rds modify-db-instance with --db-instance-identifier instead.

Lines from these log groups arrive with the last part of the log group name as the program, which is the log type. For example, every line from the error log has the program error, whatever its level.

An Aurora MySQL Serverless v2 cluster can fill its error log with a note at every capacity change. The FAQ shows how to leave those notes out.

Checking delivery

The next line written to a subscribed log group should reach the log source within seconds. On the source's page, the Logs received card and the events chart show the new events, and the Dropped events (last 24 hours) card counts any events that were dropped (for example over the flood limit). In the Log Explorer, you can filter by the source and by program to watch one log group at a time.

To test without waiting for a real line, open the function in the Lambda console, choose Test, pick the cloudwatch-logs template event and run it. The template's sample lines arrive in the log source with the program testLogGroup.

If nothing arrives, look in the function's own log group, /aws/lambda/sanelog-cloudwatch-forwarder:

• A rejected batch is written there with SaneLog's response, which names the reason.
• An error there means the function ran but could not post to SaneLog. Check the environment variables.
• If the log group is empty, the function has never run. Check the subscription filter and, on the CLI path, the permission from step 3.

EventBridge service events

What arrives

EventBridge delivers each matching event as it happens. SaneLog turns it into an event with:

• Program: the AWS service that sent it, such as rds, guardduty, health or signin.
• Message: what happened, in one line.
• Level: a level that matches the service's own failure signal (listed below), or info.
• Attributes: detail_type, event_id, aws_account, region, resources and the event's full detail object, so you can search by any of its fields.

These services get a tailored message and level:

• RDS: the event message. Failures arrive at error, and failovers and low storage at warning.
• GuardDuty findings: the finding title. High severity arrives at error, medium at warning and low at notice.
• AWS Health: the event description. Issues arrive at warning and scheduled changes at notice. Scheduled maintenance for EC2, RDS and ElastiCache arrives this way.
• CloudWatch alarms: the alarm name and its change of state as a sentence. ALARM arrives at error, INSUFFICIENT_DATA at warning and OK at notice.
• ECS tasks: task stops with the reason. Stops caused by a failure or a non-zero exit code arrive at error, and Spot interruptions at warning. A task starting to run arrives at info, and other routine task changes arrive at debug because they are frequent.
• ECS deployments and services: the reason AWS gives. Errors arrive at error, warnings and rollbacks at warning, and completed deployments at notice.
• Step Functions: the execution name and status, with the error and cause when a run fails. Failed and timed-out runs arrive at error, aborted runs at warning and successful runs at notice.
• AWS Backup: the backup, copy or restore job's state and status message. Failed and expired jobs arrive at error, and aborted jobs and jobs completed with issues at warning.
• Glue: the job message. Failed and timed-out jobs arrive at error.
• Certificate Manager: the certificate name, with the days left when it is about to expire. Approaching expiry and revocation arrive at warning, and expiry and renewal action required at error.
• Console sign-ins: whether the sign-in succeeded, who signed in, from which IP address and whether MFA was used. Failed sign-ins arrive at warning and root sign-ins at notice.
• Cost anomalies: the service or other dimension, the dollar impact and the anomaly monitor. These always arrive at warning.

Any other AWS event arrives at info, with its event type as the message and the detail's message or description appended when it has one. To add a service, change only what the rule matches. Nothing needs to change in SaneLog.

Some details are left out of the attributes:

• Step Functions events do not keep the execution's input and output, which can be large and can hold sensitive data.
• An event whose detail has more than 100 fields keeps only the first 100, and very deeply nested parts are kept as text. SaneLog marks such an event with the attribute truncated.

Step 1: The connection

The connection holds the events source's token. EventBridge sends it in the Authorization header with every delivery.

Console. In your workload region, open EventBridge > Connections > Create connection. Then:

1. Set the connection name to sanelog. Under Configure invocation, leave the API type on Public.
2. Under Configure authorization, leave Custom configuration selected and choose the authorization type API Key. Set the API key name to Authorization, and the Value to Bearer followed by a space and the events source's token.
3. Leave the encryption on the AWS owned key, and create the connection.

CLI.

aws events create-connection \
  --name sanelog \
  --authorization-type API_KEY \
  --auth-parameters '{"ApiKeyAuthParameters":{"ApiKeyName":"Authorization","ApiKeyValue":"Bearer <token>"}}'

The output includes the connection's ARN, which you need in step 2.

Step 2: The API destination

The API destination is the SaneLog address that EventBridge posts to.

Console. Open EventBridge > API destinations > Create API destination. Then:

1. Set the name to sanelog-ingest.
2. Set the API destination endpoint to https://app.sanelog.com/ingest/aws-events.
3. Set the HTTP method to POST. The form starts on GET.
4. Set the Invocation rate limit per second to 10.
5. Under Connection configuration, keep Use an existing connection and pick sanelog. Then create the destination.

CLI.

aws events create-api-destination \
  --name sanelog-ingest \
  --connection-arn arn:aws:events:<region>:<account-id>:connection/sanelog/<id> \
  --invocation-endpoint https://app.sanelog.com/ingest/aws-events \
  --http-method POST \
  --invocation-rate-limit-per-second 10

The output includes the destination's ARN, which you need in step 3.

Step 3: The rule

The rule decides which events are sent. This pattern sends every AWS service event in the region, including event types AWS adds later. It leaves out the API call records that CloudTrail publishes (why):

{
  "source": [{ "prefix": "aws." }],
  "detail-type": [{ "anything-but": "AWS API Call via CloudTrail" }]
}

To choose services one by one instead, create one rule per service with a pattern such as {"source": ["aws.rds"]}, all pointing at the same API destination.

Console. Open EventBridge > Rules > Create rule and pick the Advanced builder, which is the step-by-step form. (The Enhanced builder is a drag-and-drop canvas that creates the same rule.) Then:

1. Define rule detail. Set the name to aws-events-to-sanelog and the event bus to default, and leave the rule enabled.
2. Build event pattern. Skip the sample event. Under Creation method, choose Custom pattern (JSON editor) and paste the pattern above into the editor. The pattern form cannot express a prefix match or an exclusion, so the JSON editor is the only method that works here. Choosing it switches the event source above it to Other, which is correct for a pattern that covers every service.
3. Select targets. The target type starts on AWS service. Switch it to EventBridge API destination, keep Use an existing API destination and choose sanelog-ingest. Leave the header and query string parameters empty.
4. Under Execution role, keep Create default role. The console creates a role that can invoke this destination and nothing else. (The CLI path creates this role as an extra step.)
5. Under Additional settings, leave the target input on Matched events and the retry policy on its defaults.
6. Skip the tags, review the rule and create it.

CLI. The target needs a role that EventBridge can assume to invoke the destination. Create the role, then the rule and its target:

cat > events-trust-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "events.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role \
  --role-name sanelog-eventbridge-role \
  --assume-role-policy-document file://events-trust-policy.json

aws iam put-role-policy \
  --role-name sanelog-eventbridge-role \
  --policy-name invoke-sanelog-ingest \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"events:InvokeApiDestination","Resource":"arn:aws:events:<region>:<account-id>:api-destination/sanelog-ingest/*"}]}'

aws events put-rule \
  --name aws-events-to-sanelog \
  --event-bus-name default \
  --event-pattern '{"source":[{"prefix":"aws."}],"detail-type":[{"anything-but":"AWS API Call via CloudTrail"}]}'

aws events put-targets \
  --rule aws-events-to-sanelog \
  --event-bus-name default \
  --targets '[{"Id":"sanelog","Arn":"arn:aws:events:<region>:<account-id>:api-destination/sanelog-ingest/<id>","RoleArn":"arn:aws:iam::<account-id>:role/sanelog-eventbridge-role"}]'

Step 4: Repeat in us-east-1

Switch the console or the CLI to us-east-1 and repeat steps 1 to 3 with the same names, the same token and the same pattern.

On the CLI path, skip the create-role command in step 3. IAM roles are global, so sanelog-eventbridge-role already exists, but its policy only allows the destination in your workload region. Replace the policy with one that allows the destination in every region, then run put-rule and put-targets as in step 3:

aws iam put-role-policy \
  --role-name sanelog-eventbridge-role \
  --policy-name invoke-sanelog-ingest \
  --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"events:InvokeApiDestination","Resource":"arn:aws:events:*:<account-id>:api-destination/sanelog-ingest/*"}]}'

AWS delivers events from its global services to the us-east-1 event bus, whatever region your workloads run in. Console sign-ins arrive there, and so do expiry warnings for certificates used by CloudFront.

Skip this step if your workload region is already us-east-1.

Sign-ins need a CloudTrail trail

Console sign-ins reach EventBridge only if the account has a CloudTrail trail with logging turned on. The same applies to every other event type whose name ends in "via CloudTrail". Without a trail, sign-ins still show in CloudTrail's Event history, but they are never delivered to an event bus.

Other service events do not need a trail. A rule that delivers RDS events does not show that sign-ins will arrive.

A minimal trail is enough. Management events on an account's first trail are free, and storage in the trail's S3 bucket is the only running cost. CloudTrail always writes its log files to a bucket, so you cannot skip the bucket.

Console. In your workload region, open CloudTrail > Trails > Create trail. The trail covers all regions by default, so it records sign-ins wherever they happen.

1. Choose trail attributes. Set the trail name to sanelog-management-events and keep Create new S3 bucket.
2. Untick Log file SSE-KMS encryption. If it stays ticked, the wizard requires a KMS key alias before you can continue, and the key adds a monthly cost that this trail does not need.
3. Leave the Additional settings as they are and the CloudWatch Logs box unticked, then choose Next.
4. Choose log events. Under Event type, keep Management events ticked, and leave Data events, Insights events and Network activity events unticked. This setup needs none of them, and data events are billed for every operation they record.
5. In the Management events card, under API activity, keep both Read and Write ticked, and tick Exclude AWS KMS events. This keeps the most frequent management calls out of the bucket and does not affect sign-ins. Choose Next.
6. Review and create. Check the summary and create the trail. Logging starts at once.

CLI. The bucket needs a bucket policy that lets CloudTrail write to it. The console writes this policy for you, and the CLI does not. With a bucket and its policy ready, run:

aws cloudtrail create-trail \
  --name sanelog-management-events \
  --s3-bucket-name <bucket> \
  --is-multi-region-trail

aws cloudtrail start-logging --name sanelog-management-events

Sign-in events can take a few minutes to start arriving after the trail is created.

Why API call records are left out

With a trail in place, EventBridge also receives an AWS API Call via CloudTrail event for every management call that changes something in the account. That includes the KMS, STS and S3 calls that workloads make whenever they store or encrypt data. The rule's anything-but line leaves these events out, for three reasons:

• Their number grows with application traffic, not with anything that needs attention.
• They count toward the source's flood limit, so they can crowd out the events that matter.
• Their request parameters hold details such as bucket names and object keys.

If you want to see API calls, send the trail to a CloudWatch Logs log group and subscribe that log group with the forwarder. A subscription filter pattern can then forward only the calls that returned an error.

Checking delivery

Service events are rare on a healthy account. Minutes or days can pass between them, and that does not mean anything is wrong.

To produce one now, create a manual snapshot of an RDS instance in the workload region, then delete it. Each action sends an RDS event, and neither interrupts the database.

To test the us-east-1 rule, sign in to the AWS console once. The sign-in arrives with the program signin.

If events do not arrive, open the rule's Monitoring tab in the EventBridge console. It counts matched events and failed invocations.

What EventBridge does with a failed delivery depends on SaneLog's answer:

• SaneLog is busy or unavailable, or the token is wrong or revoked. EventBridge retries the delivery for up to 24 hours, so a short interruption delays events rather than losing them. After a token fix, events from the last 24 hours still arrive.
• The source is paused, or the account has ended its free period or used its monthly allowance. EventBridge does not retry, and the delivery counts as a failed invocation.

A console sign-in usually arrives on the us-east-1 event bus, but AWS can record some IAM user sign-ins in another region, such as us-east-2, eu-north-1 or ap-southeast-2. If a test sign-in does not arrive, filter CloudTrail's Event history by the event name ConsoleLogin and check the event's region. Repeat steps 1 to 3 in that region to receive those sign-ins.

Security Hub republishes GuardDuty, Inspector and Config findings as its own events. If your rule matches both Security Hub and those services, each finding arrives twice. Match either Security Hub or the individual services, not both.

Removing the integration

Everything the steps above created lives in your AWS account, and you remove it there. Deleting the subscription filters, the subscription policy or the rule stops new data at once. Everything else can then be deleted in any order.

Console

For CloudWatch Logs, in each region you set it up in:

1. For every subscribed log group, open it under CloudWatch > Log groups, select the sanelog filter on the Subscription filters tab and delete it.
2. Open the function under Lambda > Functions and note its execution role, shown on the Configuration > Permissions tab. Then delete the function.
3. Delete that role under IAM > Roles.
4. Optionally delete the function's own log group, /aws/lambda/sanelog-cloudwatch-forwarder.

For EventBridge, in your workload region and in us-east-1:

1. Open the rule aws-events-to-sanelog under EventBridge > Rules and note the role shown for its target on the Targets tab. Then delete the rule.
2. Delete the API destination sanelog-ingest and then the connection sanelog, under EventBridge > API destinations and Connections.
3. Delete the role you noted under IAM > Roles. IAM roles are global, so the rule in us-east-1 may use the same role. Delete it once, after both rules are gone.

CLI

aws logs delete-subscription-filter --log-group-name <log-group> --filter-name sanelog
aws logs delete-account-policy --policy-name sanelog-forward-all --policy-type SUBSCRIPTION_FILTER_POLICY
aws lambda delete-function --function-name sanelog-cloudwatch-forwarder
aws iam detach-role-policy --role-name sanelog-forwarder-role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam delete-role --role-name sanelog-forwarder-role

aws events remove-targets --rule aws-events-to-sanelog --ids sanelog
aws events delete-rule --name aws-events-to-sanelog
aws events delete-api-destination --name sanelog-ingest
aws events delete-connection --name sanelog
aws iam delete-role-policy --role-name sanelog-eventbridge-role --policy-name invoke-sanelog-ingest
aws iam delete-role --role-name sanelog-eventbridge-role

Run the delete-subscription-filter command once for each subscribed log group. Run the EventBridge commands (remove-targets to delete-connection) in each region where you created the rule. Run the IAM commands once.

What to keep or remove separately

• The CloudTrail trail (sanelog-management-events) and its S3 bucket may serve other purposes. Delete them under CloudTrail > Trails and S3 only if nothing else uses them. Stopping the trail also stops console sign-ins reaching any other EventBridge rule.
• RDS log exports keep writing to CloudWatch Logs, which AWS bills for. Turn them off with Modify on the database if nothing else reads them.

Stopping delivery from SaneLog

You can also stop delivery from SaneLog by choosing Revoke token on the source's page. After that, the forwarder writes each batch to its own log group and drops it, and EventBridge records failed invocations, until you remove the AWS side or give it a new token.

Rotate token keeps the old token working until the time shown as Old token valid until, which gives you time to update the forwarder's environment variable and the connection.