Skip to main content

S3

The S3 backend is ingest-only to a S3 bucket, saving logs to a series of parquet files.

warning

S3 as a backend currently has limited functionality and support. Ingestion architecture and underlying storage mechanisms are subject to breaking changes.

Behavior and Configuration​

Configuration​

The following values are required when creating a S3 backend:

  • s3_bucket: (string, required) the name of an existing S3 bucket.
  • storage_region: (string, required) the name of the region where the bucket is located.
  • access_key_id: (string, required) used for auth.
  • secret_access_key: (string, required) used for auth.
  • batch_timeout: (integer, required) the maximum time in milliseconds to wait before flushing the batch. Values between 2000 and 5000 are recommended.
note

For non-AWS S3-compatible providers, set storage_region to the value the provider expects — for example auto for Cloudflare R2, or the region string shown in your provider's dashboard for Backblaze B2 and DigitalOcean Spaces.

Required Permissions​

The account tied to the access_key_id only needs s3:PutObject on the target bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::YOUR_BUCKET/*"
    }
  ]
}

For buckets encrypted with SSE-KMS, the credential additionally needs kms:GenerateDataKey on the KMS key.

Objects are written to keys of the form <source uuid>/<unix_microseconds>.parquet, where the source UUID has dashes replaced by underscores. The backend connection test also writes a small sentinel file to _connection_test.parquet at the bucket root, so a policy scoped to a key prefix must still allow writing that root key.

Implementation Details​

In order to optimize performance, the S3 backend will produce many small parquet files. In heavy workloads, these files will be generated when the event batch reaches approximately 8MB in size OR when the batch timeout is reached.

The parquet file schema is as follows:

  • id: The log event UUID.
  • event_message: The provided or generated event message of the log event, stored as String
  • body: The processed log event, stored as serialized JSON in a String column.
  • timestamp: Unix microsecond, stored as DateTime