• Developer guide
  • API reference

›Ingestion sources

Getting started

  • Introduction to Imply Polaris
  • Quickstart
  • Execute a POC
  • Create a dashboard
  • Navigate the console
  • Key concepts

Tables and data

  • Overview
  • Introduction to tables
  • Table schema
  • Create an ingestion job
  • Timestamp expressions
  • Data partitioning
  • Introduction to rollup
  • Approximation algorithms
  • Replace data

Ingestion sources

  • Ingestion sources overview
  • Supported data formats
  • Create a connection
  • Ingest from files
  • Ingest data from a table
  • Ingest from S3
  • Ingest from Kafka and MSK
  • Ingest from Kinesis
  • Ingest from Confluent Cloud
  • Kafka Connector for Imply Polaris
  • Push event data
  • Connect to Confluent Schema Registry

Analytics

  • Overview
  • Manage data cubes
  • Visualize data
  • Data cube dimensions
  • Data cube measures
  • Dashboards
  • Visualizations reference
  • Set up alerts
  • Set up reports
  • Embed visualizations
  • Query data

Monitoring

  • Overview

Management

  • Overview
  • Pause and resume a project

Billing

  • Overview
  • Polaris plans
  • Estimate project costs

Usage

  • Overview

Security

    Polaris access

    • Overview
    • Invite users to your organization
    • Manage users
    • Permissions reference
    • Manage user groups
    • Enable SSO
    • SSO settings reference
    • Map IdP groups

    Secure networking

    • Connect to AWS
    • Create AWS PrivateLink connection

Developer guide

  • Overview
  • Authentication

    • Overview
    • Authenticate with API keys
    • Authenticate with OAuth
  • Manage users and groups
  • Migrate deprecated resources
  • Create a table
  • Define a schema
  • Upload files
  • Create an ingestion job
  • Ingestion sources

    • Ingest from files
    • Ingest from a table
    • Get ARN for AWS access
    • Ingest from Amazon S3
    • Ingest from Kafka and MSK
    • Ingest from Amazon Kinesis
    • Ingest from Confluent Cloud
    • Push event data
    • Kafka Connector for Imply Polaris
    • Kafka Connector reference
  • Filter data to ingest
  • Ingest nested data
  • Ingest and query sketches
  • Specify data schema
  • Query data
  • Update a project
  • Link to BI tools
  • Connect over JDBC
  • Query parameters reference
  • API documentation

    • OpenAPI reference
    • Query API

Product info

  • Release notes
  • Known limitations
  • Druid extensions

Ingest from Amazon Kinesis

You can create a connection to Amazon Kinesis, a real-time data processing platform provided by Amazon Web Services, to ingest data into Imply Polaris.

Create a unique connection for each Kinesis stream from which you want to ingest data.

Kinesis connection information

A Polaris connection to Amazon Kinesis requires the following:

  • Information about the Kinesis data stream to ingest from.

    • Stream name: The name of the Kinesis stream that contains the data to ingest.
    • AWS endpoint: The endpoint of the Kinesis stream, such as kinesis.us-east-1.amazonaws.com. There is no limitation on a region for the Kinesis data stream.
  • Authorization to access the Kinesis data streams. For more information, see Secure connections to AWS and the AWS documentation on Managing access to resources.

    • ARN of IAM role: The Amazon Resource Name (ARN) of the AWS assumed role to use for access. For example, arn:aws:iam:::123456789012:role/kinesis-access-role.

    • Trust policy attached to the IAM role: Authorizing access to your Kinesis data from Polaris requires both the correct permissions defined in the IAM policy attached to your IAM role as well as a trust policy added to your IAM role to allow Polaris to assume the role. For more information, see Trust policy.

    • IAM policy with access permissions: In order to grant Polaris access to view and ingest data from your Kinesis streams, attach to the IAM role a permissions policy that lists your Kinesis stream resources and includes the following actions:

      • kinesis:ListStreams to view your Kinesis data streams.
      • kinesis:DescribeStream to obtain information about your Kinesis data streams.
      • kinesis:ListShards to list the shards in a stream.
      • kinesis:GetShardIterator to get a shard iterator. A shard iterator specifies the position from which to start reading data records from a shard in a stream.
      • kinesis:GetRecords to get data records from a Kinesis data stream's shard.

Example IAM policy

The following example shows an IAM policy that can be attached to your IAM role. The policy grants the role the listed permissions for Polaris to view and obtain data from your Kinesis data stream. Replace KINESIS ARN with the ARN for your Kinesis data stream—for example, arn:aws:kinesis:*:111122223333:stream/my-stream.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:ListStreams",
        "kinesis:DescribeStream"
        ],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:ListShards",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords"
      ],
      "Resource": ["KINESIS ARN"]
    }
  ]
}

Example Kinesis connection

The following screenshot shows an example connection created in the UI. For more information on creating connections in the UI, see Create a connection.

Kinesis connection UI

To successfully test the connection to Kinesis, ensure your Kinesis stream contains data.

Learn more

To learn how to ingest data from Amazon Kinesis using the Polaris API, see Ingest data from Amazon Kinesis by API.

← Ingest from Kafka and MSKIngest from Confluent Cloud →
  • Kinesis connection information
  • Example IAM policy
  • Example Kinesis connection
  • Learn more
Key links
Try ImplyApache Druid siteImply GitHub
Get help
Stack OverflowSupportContact us
Learn more
BlogApache Druid docs
Copyright © 2023 Imply Data, Inc