• Developer guide
  • API reference

›Monitoring

Getting started

  • Introduction to Imply Polaris
  • Quickstart
  • Execute a POC
  • Create a dashboard
  • Navigate the console
  • Customize Polaris
  • Key concepts

Tables and data

  • Overview
  • Introduction to tables
  • Table schema
  • Ingestion jobs

    • Create an ingestion job
    • Ingest using SQL
    • Job auto-discovery
    • Timestamp expressions
    • SQL ingestion reference
    • Ingestion status reference
  • Data partitioning
  • Introduction to rollup
  • Replace data
  • Ingestion use cases

    • Approximation algorithms
    • Ingest earliest or latest value

Ingestion sources

  • Ingestion sources overview
  • Supported data formats
  • Create a connection
  • Ingest from files
  • Ingest data from a table
  • Ingest from S3
  • Ingest from Kafka and MSK
  • Ingest from Kinesis
  • Ingest from Confluent Cloud
  • Kafka Connector for Imply Polaris
  • Push event data
  • Connect to Confluent Schema Registry
  • Ingestion source reference

Analytics

  • Overview
  • Manage data cubes
  • Visualize data
  • Data cube dimensions
  • Data cube measures
  • Dashboards
  • Visualizations reference
  • Set up alerts
  • Set up reports
  • Embed visualizations

Querying

  • Overview
  • Time series functions

Monitoring

  • Overview
  • Monitoring dashboards
  • Monitor performance metrics
  • Integrate with Datadog
  • Integrate with Prometheus
  • Integrate with Elastic stack
  • Metrics reference

Management

  • Overview
  • Pause and resume a project

Usage and Billing

  • Billing structure overview
  • Polaris plans
  • Add a payment method
  • Monitor account usage

Security

    Polaris access

    • Overview
    • Invite users to your organization
    • Manage users
    • Permissions reference
    • Manage user groups
    • Enable SSO
    • SSO settings reference
    • Map IdP groups

    Secure networking

    • Connect to AWS
    • Create AWS PrivateLink connection

Developer guide

  • Overview
  • Security

    • Overview
    • Authenticate with API keys
    • Authenticate with OAuth
    • Manage users and groups
    • Restrict an embedding link
  • Migrate deprecated resources
  • Create a table
  • Upload files
  • Ingestion jobs

    • Create an ingestion job
    • Create a streaming ingestion job
    • Ingest using SQL
    • View and manage jobs

    Ingestion sources

    • Ingest from files
    • Ingest from a table
    • Get ARN for AWS access
    • Ingest from Amazon S3
    • Ingest from Kafka and MSK
    • Ingest from Amazon Kinesis
    • Ingest from Confluent Cloud
    • Push event data
    • Kafka Connector for Imply Polaris
    • Kafka Connector reference

    Ingestion use cases

    • Filter data to ingest
    • Ingest nested data
    • Ingest and query sketches
    • Specify data schema
    • Ingest Kafka metadata

    Analytics

    • Query data
    • Connect over JDBC
    • Link to BI tools
    • Query parameters reference
  • Update a project
  • API documentation

    • OpenAPI reference
    • Query API

    Migrations

    • Migrate from Hybrid

Product info

    Release notes

    • 2023
    • 2022
  • Known limitations
  • Druid extensions

Metrics reference

You can use the Imply Polaris Metrics API to emit ingestion and query performance metrics into third-party monitoring tools.

Polaris exposes the following types of metrics. For more information on metric types, refer to the official OpenMetrics documentation.

  • Gauge: A measurement that can increase, decrease, or stay constant over time.
  • Summary: A measurement used to sample observations, such as request duration and response sizes.

This topic contains a list of performance metrics and their labels available in the Metrics API.

Labels

A label is a key-value attribute of a metric. Labels differentiate the characteristics of the resource that is being measured.

All Polaris metrics share the region label. It represents the deployment region of the project for which metrics are being reported.

Some metrics may include the following additional labels:

  • job_id: The ID of the ingestion job.
  • quantile: The quantile measurement, where 0.5 is the median, 0.98 corresponds to the 98th percentile, and +Inf correspond to the maximum value of the metric.
  • table: The table that the metric applies to.
  • tables: The list of queried tables that the metric applies to.
  • type: The type of ingestion.

Query metrics

Polaris exposes the following query metrics.

query_count

The total number of queries issued over one minute.

  • Type: gauge
  • Labels: tables, region

query_time_ms

The time, in milliseconds, that it took to complete a query.

  • Type: summary
  • Labels: tables, region, quantile

Polaris provides query_count and query_time_ms metrics for tables that have been queried. If you haven't queried any tables, these metrics are not available. Configure your monitoring tools to zero fill missing data in the UI.

A query that omits the table name—for example, SELECT 1234—or fails SQL planning—for example, SELECT * FROM table WHERE nonExistentColumn='abc'—has its tables label set to tables=[].

The order of elements in the tables list is not guaranteed. For example, if you have two queries SELECT * from A, B and SELECT * from B, A, the metrics for these queries can be split across two lists tables=[A, B] and tables=[B, A].

Kafka ingestion metrics

Polaris exposes the following metrics for Apache Kafka and push streaming ingestion jobs.

ingest_job_count

The total number of ingestion jobs.

  • Type: gauge
  • Labels: table, region, type

ingest_kafka_lag

The total lag between the offsets consumed by the Kafka streaming job and offsets in the Kafka topic across all partitions.

The lag reflects the time from when Polaris returns an HTTP 200 status code and the time the data is ingested into Polaris. The lag doesn’t account for the time between an application sending a request and Polaris responding with a 200.

  • Type: summary
  • Labels: table, region, job_id, quantile

ingest_events_processed

The number of events successfully processed during ingestion.

  • Type: gauge
  • Labels: table, region, job_id

ingest_events_thrownAway

The number of events rejected during ingestion because they are either null, filtered out, or outside the retention period.

  • Type: gauge
  • Labels: table, region, job_id

ingest_events_unparseable

The number of events rejected because the events are unparseable.

  • Type: gauge
  • Labels: table, region, job_id

Table metrics

Polaris does not compute table metrics every minute. These metrics are only emitted when calculated. Configure your monitoring tools to use the previous value instead of 0.

table_bytes_compacted

The total size, in bytes, of the compacted data in the table.

  • Type: gauge
  • Labels: table, region

table_intervals

The total number of intervals in the table.

  • Type: gauge
  • Labels: table, region

table_intervals_compacted

The number of intervals in the table that have been compacted.

  • Type: gauge
  • Labels: table, region

table_size

The total size of the table in bytes.

  • Type: gauge
  • Labels: table, region
← Integrate with Elastic stackOverview →
  • Labels
  • Query metrics
  • Kafka ingestion metrics
  • Table metrics
Key links
Try ImplyApache Druid siteImply GitHub
Get help
Stack OverflowSupportContact us
Learn more
BlogApache Druid docs
Copyright © 2023 Imply Data, Inc