Ingest Kafka records and metadata

In Imply Polaris, you can ingest both event data and event metadata from an Apache Kafka topic. An event, or record, consumed from a Kafka topic contains metadata including the key, Kafka timestamp, and any headers sent with the event. This topic shows you how to ingest Kafka event data and its corresponding metadata.

Parse Kafka metadata for ingestion

To parse Kafka metadata for ingestion, follow these steps:

1. In the Parse data stage of creating an ingestion job, select the format of the event data in the Input format drop-down menu.

2. Click the Parse Kafka metadata toggle.

3. Polaris displays fields specific to the Kafka metadata format:

   • Timestamp column name: A name to use for the Kafka timestamp. This name becomes the input field name that you can use in mappings.
   • Key input format: Format to use to parse the Kafka key. If you store the key as a string, use the CSV input format and specify the key column header with the key name.
   • Header format type: Character encoding to use to parse data from the Kafka headers. When you elect to parse Kafka headers, you can define the header column prefix. Polaris prepends the prefix to the Kafka header. For example, with the default prefix kafka.header., you can use a header with key Header-1 in mappings with the input field name kafka.header.Header-1.

4. Select Continue to continue onto the Map source to table stage. If you use the Kafka timestamp as your input to the primary timestamp, __time, define the input expression as MILLIS_TO_TIMESTAMP("kafka.timestamp"). If you use a non-default timestamp column name, update the value in the function. Verify that the Kafka timestamp is within 30 days of ingestion time.

5. Click Start ingestion to begin the ingestion job.

Learn more

You can also configure the settings for ingesting Kafka records and associated metadata using the API.

For details on creating connections to ingest from Kafka sources, see the following docs:

• Confluent Cloud
• Apache Kafka