Ingest cloud object metadata
When your ingestion source is an S3 connection or Azure Blob Storage connection, you can ingest metadata about the objects, for example to create a new column to store an object's file path in the S3 bucket.
Metadata fields
You configure the options to ingest object metadata in the Parse data stage of the ingestion job. Select the toggle to include object metadata. You can then check any of the following options:
- Full S3 URI: Ingest the complete URI string from the system input field
__file_uri. - Bucket name: Ingest the bucket or container name from the system input field
__file_bucket. - Object key (path): Ingest the object key from the system input field
__file_path.
The following table lists examples for the metadata fields:
| Metadata field | Amazon S3 | Azure Blob Storage |
|---|---|---|
__file_uri | s3://bucket-name/logs/file01.json.gz | azureStorage://container-name/logs/file01.json.gz |
__file_bucket | bucket-name | container-name |
__file_path | logs/file01.json.gz | logs/file01.json.gz |
Example
The following screenshot displays a sample of data ready for ingestion with the URI, bucket name, and path included for ingestion.
You can change the column name in the Map source to table step. Select the column and update the Name field in the column editor. In the Input expression field, you can apply an input expression to transform the metadata input field, such as string parsing, concatenation, or converting the string to lower case format.
Learn more
See the following topics for more information:
- Connect to Amazon S3 for reference on connecting from Amazon S3 to Polaris.
- Create an ingestion job to learn more about ingestion jobs.
- Create a connection to learn about creating connections to ingestion sources.