Skip to main content

Define object filters for an S3 pull job

AI summary
Explains glob patterns to filter S3 objects for ingestion into Imply Lumi. Shows how to write efficient patterns and how to use date optimization to for faster processing.

About AI summaries.

The S3 pull integration in Imply Lumi configures event ingestion from an AWS S3 bucket. You define what objects to ingest from the bucket using glob patterns. This applies to S3 jobs created in Lumi with backfill or poll-based ingestion. Note that for SNS-based S3 pull ingestion, you apply the filter using event notifications on the Amazon side.

This topic lists the object filter patterns you can use in an S3 job.

Glob patterns​

The following table lists the glob patterns supported by Lumi:

PatternDescriptionExample
**Matches zero or more path segmentslogs/**, **/*.json
*Matches any characters except path separatorslogs/2025-10-*.json
?Matches exactly one character except path separatorslogs/demo-logs-?.json
[abc]Matches any character in the setlogs/[aeu]*_logs.*
[a-z]Matches any character in the rangelogs/[a-z]*_logs.*
[!abc]Matches any character not in the set (negation)logs/[!aeu]*_logs.*
{a,b,c}Matches any of the alternatives (brace expansion)logs/*.{bz2,gz}

Date filter​

Lumi optimizes discovery when the object filter contains a standardized date pattern ({YYYY}, {MM}, {DD}, {HH}) and when you specify Modified after or Modified before.

For example, to define a prefix equivalent to logs/access/2026/01/{01..31}:

  1. Specify object filter logs/access/{YYYY}/{MM}/{DD}/**
  2. Select the modified after date 2026/01/01
  3. Select the modified before date 2026/02/01

Refine ingestion scope​

The limit for a backfill job or individual task in poll-based ingestion is 1,000,000 objects.

If your object filter matches against a large number of objects in your S3 bucket, Lumi can take a long time in the discovery phase before moving onto processing.

Narrow the scope of discovery by using the most specific prefix that matches your objects. For example, the prefix logs/access/ is more specific than logs/. However, logs/**/*.json has the same prefix as logs/ so it doesn't reduce the number of objects scanned. Glob patterns only confer specificity in processing after Lumi discovers objects that match the prefix.

Examples​

Consider an example scenario of an S3 bucket organized with the following structure:

logs/
├── access/
│   └── 2026/
│       └── 01/
│           ├── 01/
│           └── 02/
├── firewall/
└── system/

A single job that uses the filter logs/** might cause bottlenecks due to the size of the job. The following examples show how to manage the scope of an S3 job.

Subdivide into multiple jobs​

Initiate three jobs, where each job defines one of the following prefixes:

logs/access/**
logs/firewall/**
logs/system/**

Ingest only firewall and system logs​

Only ingest what you plan to use. To ingest firewall and system logs only, specify the filter:

logs/{firewall,system}

Apply date filter for access logs​

If you want to only ingest access logs, you can include the structured date format in the filter:

logs/access/{YYYY}/{MM}/{DD}

If you don't include the date pattern in the object filter, such as logs/access/**, Lumi iterates through everything in the access/ folder then compares it against the modified date range. This significantly increases the number of objects Lumi must evaluate during discovery.

Learn more​

See the following topics for more information: