Define object filters for an S3 pull job
AI summary
About AI summaries.
The S3 pull integration in Imply Lumi configures event ingestion from an AWS S3 bucket. You define what objects to ingest from the bucket using glob patterns. This applies to S3 jobs created in Lumi with backfill ingestion. Note that for SNS-based S3 pull ingestion, you apply the filter using event notifications on the Amazon side.
This topic lists the patterns you can use in an S3 job.
When creating a job in the Lumi UI, you specify the pattern in the Object filter field.
In the API, include the pattern field in the request body.
Glob patterns
The following table lists the glob patterns supported by Lumi:
| Pattern | Description | Example |
|---|---|---|
** | Matches zero or more path segments | logs/**, **/*.json |
* | Matches any characters except path separators | logs/2025-10-*.json |
? | Matches exactly one character except path separators | logs/demo-logs-?.json |
[abc] | Matches any character in the set | logs/[aeu]*_logs.* |
[a-z] | Matches any character in the range | logs/[a-z]*_logs.* |
[!abc] | Matches any character not in the set (negation) | logs/[!aeu]*_logs.* |
{a,b,c} | Matches any of the alternatives (brace expansion) | logs/*.{bz2,gz} |
Date filter
Lumi can optimize discovery when the object filter contains a standardized date pattern (YYYY, MM, DD, HH) and when you specify Modified after or Modified before.
For example, if you limit objects to those modified after 2026/01/01 and before 2026/02/01, the date range applies on top of the object filter so that Lumi only discovers objects equivalent to logs/access/2026/01/{01..31}.
Reduce job scope
If your object filter matches against a large number of objects in your S3 bucket, it can take a long time for Lumi to discover what to ingest. For example, with billions of objects to iterate over, a job can take several hours in Discovery before moving onto ingestion in Processing.
To reduce the scope of discovery, modify the prefix in your object filter to be more specific, or create multiple jobs of smaller scope.
Only a more specific prefix reduces discovery scope.
Other filter patterns, such as glob wildcards, apply after Lumi iterates over all matching objects.
For example, logs/**/*.json requires evaluation of the same number of objects as logs/**.
Examples
Consider an example scenario of an S3 bucket organized with the following structure:
logs/
├── access/
│ └── 2026/
│ └── 01/
│ ├── 01/
│ └── 02/
├── firewall/
└── system/
A single job that uses the filter logs/** might cause bottlenecks due to the size of the job.
The following examples show how to manage the scope of an S3 job.
Subdivide into multiple jobs
Initiate three jobs, where each job defines one of the following prefixes:
logs/access/**
logs/firewall/**
logs/system/**
Ingest only firewall and system logs
Only ingest what you plan to use. To ingest firewall and system logs only, specify the filter:
logs/{firewall,system}
Apply date filter for access logs
If you want to only ingest access logs, you can include the structured date format in the filter:
logs/access/{YYYY}/{MM}/{DD}
If you don't include the date pattern in the object filter, such as logs/access/**, Lumi iterates through everything in the access/ folder then compares it against the modified date range.
This significantly increases the number of objects Lumi must evaluate during discovery.
Learn more
See the following topics for more information:
- Send events with S3 pull for an overview of S3 pull in Lumi.
- Backfill events with S3 pull for creating a backfill ingestion from Amazon S3.