Skip to main content

Ingest cloud object metadata

When your ingestion source is an S3 connection or Azure Blob Storage connection, you can ingest metadata about the objects, for example to create a new column to store an object's file path in the S3 bucket.

Metadata fields

You configure the options to ingest object metadata in the Parse data stage of the ingestion job. Select the toggle to include object metadata. You can then check any of the following options:

  • Full S3 URI: Ingest the complete URI string from the system input field __file_uri.
  • Bucket name: Ingest the bucket or container name from the system input field __file_bucket.
  • Object key (path): Ingest the object key from the system input field __file_path.

The following table lists examples for the metadata fields:

Metadata fieldAmazon S3Azure Blob Storage


The following screenshot displays a sample of data ready for ingestion with the URI, bucket name, and path included for ingestion.

S3 object metadata parse

You can change the column name in the Map source to table step. Select the column and update the Name field in the column editor. In the Input expression field, you can apply an input expression to transform the metadata input field, such as string parsing, concatenation, or converting the string to lower case format.

S3 object metadata map

Learn more

See the following topics for more information: