Metrics reference

You can use the Imply Polaris Metrics API to emit ingestion and query performance metrics into third-party monitoring tools.

Polaris exposes the following types of metrics. For more information on metric types, refer to the official OpenMetrics documentation.

  • Gauge: A measurement that can increase, decrease, or stay constant over time.
  • Summary: A measurement used to sample observations, such as request duration and response sizes.

This topic contains a list of performance metrics and their labels available in the Metrics API.

Labels

A label is a key-value attribute of a metric. Labels differentiate the characteristics of the resource that is being measured.

All Polaris metrics share the region label. It represents the deployment region of the project for which metrics are being reported.

Some metrics may include the following additional labels:

  • job_id: The ID of the ingestion job.
  • quantile: The quantile measurement, where 0.5 is the median, 0.98 corresponds to the 98th percentile, and +Inf correspond to the maximum value of the metric.
  • table: The table that the metric applies to.
  • tables: The list of queried tables that the metric applies to.
  • type: The type of ingestion.

Query metrics

Polaris exposes the following query metrics.

query_count

The total number of queries issued over one minute.

  • Type: gauge
  • Labels: tables, region

query_time_ms

The time, in milliseconds, that it took to complete a query.

  • Type: summary
  • Labels: tables, region, quantile

Polaris provides query_count and query_time_ms metrics for tables that have been queried. If you haven't queried any tables, these metrics are not available. Configure your monitoring tools to zero fill missing data in the UI.

A query that omits the table name—for example, SELECT 1234—or fails SQL planning—for example, SELECT * FROM table WHERE nonExistentColumn='abc'—has its tables label set to tables=[].

The order of elements in the tables list is not guaranteed. For example, if you have two queries SELECT * from A, B and SELECT * from B, A, the metrics for these queries can be split across two lists tables=[A, B] and tables=[B, A].

Kafka ingestion metrics

Polaris exposes the following metrics for Apache Kafka and push streaming ingestion jobs.

ingest_job_count

The total number of ingestion jobs.

  • Type: gauge
  • Labels: table, region, type

ingest_kafka_lag

The total lag between the offsets consumed by the Kafka streaming job and offsets in the Kafka topic across all partitions.

The lag reflects the time from when Polaris returns an HTTP 200 status code and the time the data is ingested into Polaris. The lag doesn’t account for the time between an application sending a request and Polaris responding with a 200.

  • Type: summary
  • Labels: table, region, job_id, quantile

ingest_events_processed

The number of events successfully processed during ingestion.

  • Type: gauge
  • Labels: table, region, job_id

ingest_events_thrownAway

The number of events rejected during ingestion because they are either null, filtered out, or outside the retention period.

  • Type: gauge
  • Labels: table, region, job_id

ingest_events_unparseable

The number of events rejected because the events are unparseable.

  • Type: gauge
  • Labels: table, region, job_id

Table metrics

Polaris does not compute table metrics every minute. These metrics are only emitted when calculated. Configure your monitoring tools to use the previous value instead of 0.

table_bytes_compacted

The total size, in bytes, of the compacted data in the table.

  • Type: gauge
  • Labels: table, region

table_intervals

The total number of intervals in the table.

  • Type: gauge
  • Labels: table, region

table_intervals_compacted

The number of intervals in the table that have been compacted.

  • Type: gauge
  • Labels: table, region

table_size

The total size of the table in bytes.

  • Type: gauge
  • Labels: table, region
← Integrate with Elastic stackOverview →