Skip to main content

Transform VPC flow logs

Amazon VPC flow logs are a feature of AWS Virtual Private Cloud (VPC) that capture metadata about the IP traffic going to and from network interfaces in your VPC.

Imply Lumi includes a predefined pipeline you can use to transform VPC flow logs. The pipeline extracts fields including source and destination IP addresses, ports, protocol, and action, and normalizes them into Lumi user attributes.

In this topic, you’ll learn how the predefined pipeline transforms flow logs.

See Transform events with pipelines for an overview of pipelines in Lumi. See Use predefined pipelines for details about predefined pipelines.

VPC flow log pipeline​

The VPC flow log pipeline identifies flow logs during ingestion by matching the pipeline conditions against the log details. The default filter expression is sourcetype=aws:cloudwatchlogs:vpcflow.

The processors in the pipeline convert log lines into structured Lumi events. The enriched events include:

  • Parsed network details
  • Normalized identifiers
  • Derived attributes
  • Contextual metadata
  • Mapped values

Lumi preserves the original log line in message. If a field can't be extracted, Lumi applies the default value unknown.

The following diagram shows a raw log event and how its fields appear in Lumi after processing by the predefined pipeline:

VPC flow log pipeline

View an example event transformed by the VPC flow log pipeline
{
  "message": "2 422682147474 eni-0635052fdd001dbfa 10.210.72.205 10.90.0.105 57752 9997 17 20 3303 1756736406 1756736427 REJECT OK",
  "account_id": "422682147474",
  "action": "blocked",
  "app": "AWS VPC Flow Logs",
  "aws_account_id": "422682147474",
  "bytes": 3303,
  "dst": "10.90.0.105",
  "dst_ip": "10.90.0.105",
  "dst_port": 9997,
  "duration": 21,
  "dvc": "eni-0635052fdd001dbfa",
  "end_time": 1756736427,
  "eventtype": "vpcflow ( cloud communicate network )",
  "host": "web-01",
  "index": "main",
  "interface_id": "eni-0635052fdd001dbfa",
  "log_status": "OK",
  "packets": 20,
  "protocol": "ip",
  "protocol_code": 17,
  "protocol_full_name": "User Datagram",
  "protocol_version": "IPV4",
  "source": "vpc-example.log",
  "sourcetype": "aws:cloudwatchlogs:vpcflow",
  "src": "10.210.72.205",
  "src_ip": "10.210.72.205",
  "src_port": 57752,
  "start_time": 1756736406,
  "transport": "udp",
  "user_id": "422682147474",
  "vendor_account": "422682147474",
  "vendor_product": "AWS VPC Flow Logs",
  "version": 2,
  "vpcflow_action": "REJECT",
  "eventId": "e_h5cc5116a-1757079660000-11-952-709-2",
  "receiver": "splunk.s3",
  "team": "Unassigned",
  "env": "Unknown",
  "iamKeyId": "75700224-47b0-498a-b76e-4f0aab3a73e7",
  "processor": "ec 20250905.1661.0",
  "collector": "ec 20250905.1661.0"
}

Use a VPC flow log pipeline​

To use a VPC flow log pipeline:

  1. Add a predefined pipeline and select the VPC Flow Log Processing pipeline.

  2. The pipeline is ready to use without customization if it meets your requirements. If you want to customize the conditions or processors, duplicate the pipeline and customize the copy.

  3. Edit the pipeline details as required. Make sure the filter expression in the pipeline conditions matches your setup.

  4. Reorder the list of pipelines so your VPC flow log pipeline is in the correct position, and ensure it's enabled. Disable or delete any pipelines you don't want to use.

  5. Send flow logs to Lumi using your chosen ingestion integration.

  6. Check the Lumi integration to preview incoming data. In this example an S3 ingest action added the logs:

    VPC flow log ingestion

  7. Query Lumi to see the enriched flow log events. The event details pane shows the pipeline that processed the event:

    VPC flow log event query

Learn more​

See the following topics for more information: