Skip to main content

Recurring ingestion from S3 at fixed intervals

AI summary
Shows you how to configure Imply Lumi to automatically ingest data from Amazon S3 at regular intervals. Describes how a recurring S3 job creates tasks to poll your bucket at the frequency you set.

About AI summaries.

You can use the S3 pull integration to configure recurring ingestion from Amazon S3 to Imply Lumi. When you configure a polling interval, Lumi periodically checks the S3 bucket at the specified time interval and ingests newly discovered objects.

This topic provides details on configuring poll-based recurring ingestion with S3 pull.

Prerequisites​

Before you continue, review Send events with S3 pull and ensure that you:

  • Met the prerequisites
  • Configured AWS access
  • Created a Lumi IAM key

How a recurring job works​

A recurring job consists of multiple tasks, or instances when Lumi polls S3. In the job, you define the time interval between tasks as the polling interval. When the polling interval elapses, Lumi ingests objects created or modified since the last task.

By default, in the first task, Lumi evaluates new objects relative to the time the job was created. Subsequent tasks evaluate new objects from the last successful task.

Lumi assigns the system attribute correlationId to each event from a recurring job. Events from the same job have the same correlation ID. For more information, see S3 pull attributes.

The following diagram shows an example of how fixed interval recurring ingestion works:

Diagram of example recurring S3 job

Create a job​

Create a recurring S3 job using the Lumi UI:

  1. From the Lumi navigation menu, click Integrations > S3 pull.

  2. In Select the job type, click Recurring.

  3. Select your AWS access method and your Lumi IAM key.

  4. For the discovery method, select Polling.

  5. Enter the following details to configure the job. The job requires the bucket name and object filter; all other fields are optional.

    • Bucket name: Name of the S3 bucket containing the data.

    • Object filter: Glob pattern for object keys that defines which objects to include when ingesting data. The pattern must match the entire object key. See S3 object filters for examples.

      info

      As a best practice, use the most specific prefix that matches your objects.

    • Region: AWS region of your S3 bucket. By default, it assumes the same region as the Lumi environment.

    • Format override: Event format, such as CSV or JSON. By default, the job inherits the format selected on the IAM key. Otherwise, Lumi automatically detects the format.

    • Polling interval: Frequency that Lumi checks the S3 bucket for new objects.

    Poll-based job

  6. Click Start job. Note that ingestion might not begin immediately.

  7. In Preview incoming data, view the events coming into Lumi. Lumi automatically refreshes the preview pane to display the latest events.

  8. Click Explore events to see more events associated with the IAM key. Adjust the time filter to choose the range of data displayed.

View job status​

Lumi displays the status of the job after you submit it. Click View and manage job for more details on the Jobs page. There, you can see the progress of discovery and processing for the latest task in the job. You can also cancel the job and view past jobs. For more details, see View and manage jobs.

Recurring go to job

Learn more​

See the following topics for more information: