Skip to main content

Metrics reference

You can use the Imply Polaris Metrics export API to emit ingestion and query performance metrics into third-party monitoring tools.

Polaris exports the following types of metrics. For more information on metric types, refer to the official OpenMetrics documentation.

  • Gauge: Measurement that can increase, decrease, or stay constant over time.
  • Summary: Measurement used to sample observations, such as request duration and response sizes.

This topic contains a list of performance metrics and their labels available in the Metrics export API.

Labels​

A label is a key-value attribute of a metric. Labels differentiate the characteristics of the resource that is being measured.

All Polaris metrics expose the following labels:

  • project_id: The unique identifier for the project.
  • region: The deployment region of the project for which metrics are being exposed.

Some metrics may expose the following additional labels:

  • job_id: ID of the ingestion job.
  • quantile: Quantile measurement, where 0.5 is the median, 0.98 corresponds to the 98th percentile, and +Inf correspond to the maximum value of the metric.
  • table: Table that the metric applies to.
  • tables: List of queried tables that the metric applies to.
  • type: Type of ingestion.
  • native_query_type: Type of the native query. For available native query types, see Available queries.

Streaming ingestion metrics​

This section describes the metrics Polaris emits for Apache Kafka, Amazon Kinesis, and push streaming ingestion jobs.

ingestion_job_count

The number of streaming ingestion jobs.

  • Type: gauge
  • Labels: region, project_id, table, type

ingest_events_processed

The number of events successfully processed during streaming ingestion.

  • Type: gauge
  • Labels: region, project_id, table, type, job_id

ingest_events_thrownAway

The number of events rejected during streaming ingestion because the events are either null, filtered out, or outside the retention period.

  • Type: gauge
  • Labels: region, project_id, table, type, job_id

ingest_events_unparseable

The number of events rejected during streaming ingestion because the events are unparseable.

  • Type: gauge
  • Labels: region, project_id, table, type, job_id

Kafka ingestion​

Polaris emits the following streaming ingestion metric for Apache Kafka.

ingest_kafka_lag

The total lag between the offsets consumed by the Kafka streaming job and offsets in the Kafka topic across all partitions.

The lag reflects the time from when Polaris returns an HTTP 200 status code and the time the data is ingested into Polaris. The lag doesn’t account for the time between an application sending a request and Polaris responding with a 200.

  • Type: summary
  • Labels: region, project_id, table, job_id, quantile

Kinesis ingestion​

Polaris emits the following streaming ingestion metric for Amazon Kinesis.

ingest_kinesis_lag_time_ms

The total lag time, in milliseconds, between the current message sequence number consumed by the Kinesis streaming job and latest sequence number in Kinesis across all shards.

  • Type: summary
  • Labels: region, project_id, table, job_id, quantile

Project capacity metrics​

Polaris emits the following project capacity metrics. You can use these metrics to set alerting thresholds and remediation actions on projects nearing capacity.

project_max_size_bytes

The maximum size of the project in bytes.

  • Type: gauge
  • Labels: region, project_id

project_current_size_bytes

The current size of the project in bytes.

  • Type: gauge
  • Labels: region, project_id

Query metrics​

Polaris emits the following SQL query metrics.

query_count

The number of SQL queries.

  • Type: gauge
  • Labels: region, project_id, tables

query_time_ms

The time it took a SQL query to complete in milliseconds.

  • Type: summary
  • Labels: region, project_id, tables, quantile

Polaris provides query_count and query_time_ms metrics for tables that have been queried. If you haven't queried any tables, these metrics are not available. Configure your monitoring tools to zero fill missing data in the UI.

A query that omits the table name—for example, SELECT 1234—or fails SQL planning—for example, SELECT * FROM table WHERE nonExistentColumn='abc'—has its tables label set to tables=[].

The order of elements in the tables list is not guaranteed. For example, if you have two queries SELECT * from A, B and SELECT * from B, A, the metrics for these queries can be split across two lists tables=[A, B] and tables=[B, A].

Table metrics​

Polaris emits table metrics only when calculated. Configure your monitoring tools to use the previous value instead of 0.

table_bytes_compacted

The size of the compacted data in the table in bytes.

  • Type: gauge
  • Labels: region, project_id, table

table_intervals

The number of intervals in the table.

  • Type: gauge
  • Labels: region, project_id, table

table_intervals_compacted

The number of intervals in the table that have been compacted.

  • Type: gauge
  • Labels: region, project_id, table

table_size

The size of the table in bytes.

  • Type: gauge
  • Labels: region, project_id, table

Legacy metrics​

Applications that use legacy Druid features can gain visibility into those features by emitting the legacy metrics described in this section. By default, Polaris doesn't emit legacy metrics. To include them, set the includeLegacyMetrics query parameter to true.

For example:

https://ORGANIZATION_NAME.api.imply.io/v1/metrics/export?includeLegacyMetrics=true

Native query metrics​

All native queries metrics come with the native_ prefix to distinguish them from SQL query metrics.

When enabled, Polaris emits the following native query metrics:

native_query_time_ms

The time it took a native query to complete in milliseconds.

  • Type: summary
  • Labels: region, project_id, tables, native_query_type, quantile

native_query_count

The number of native queries.

  • Type: gauge
  • Labels: region, project_id, tables