Skip to main content

Use dashboards to monitor Polaris

Imply Polaris provides built-in dashboards and data cubes for monitoring query performance and streaming ingestion jobs. You can use these tools to probe into detailed metrics and evaluate performance for your Polaris projects.

This topic provides an overview of the monitoring capabilities available in the Polaris UI. For information on how to import metrics into third-party monitoring systems, see Monitor performance metrics.

Access monitoring pages​

Members of the Organization Admin, Project Admin, and Data Manager groups and users with the AccessMonitoring permission assigned to their profile can view the contents of the monitoring dashboards and data cubes. For information on permissions, see Permissions reference.

Access the monitoring pages from the left sidebar of the Polaris UI: User queries, Streaming, and Detailed metrics.

User queries​

The User queries page provides a dashboard for monitoring query performance.

Streaming dashboard

In this dashboard, you can analyze the following:

  • User activity: Track the number of distinct query users and top query users. You can filter by user to investigate performance issues for specific users.
  • Query performance: Track the 98th percentile of query execution times, average query latency, total number of queries executed, and total number of failed queries.
  • Query processing details: Evaluate the average and 98th percentile of query wait times to determine whether to scale your project in response to high concurrent load issues.
  • Segment details: Assess the number of segments scanned and the segment scan times. High segment scan times usually indicate that your segment files are too large, which can be resolved by data partitioning. A high number of scanned segments usually means that your data is too fragmented, and you may benefit from configuring data rollup.

Streaming​

The Streaming ingestion page provides a dashboard for monitoring streaming ingestion. By default, this dashboard displays job details in an aggregate view across all tables. To filter by a specific table, click the plus icon in the filters bar and select Table from the drop-down.

You can use the steaming ingestion dashboard to assess lag. In the context of Polaris, lag refers to the delay between the latest data records ingested into a stream and the corresponding records successfully processed by Polaris.

Streaming dashboard

The dashboard reports the following streaming ingestion metrics:

  • Average Kafka lag (events): Average lag in messages between the offsets consumed by the Apache Kafka stream and the latest offsets in Kafka brokers across all partitions. Minimum emission period for this metric is a minute.
  • Average Kinesis lag (milliseconds): Average lag time in milliseconds between the current message sequence number consumed by the Amazon Kinesis stream and the latest sequence number in Kinesis across all shards. Minimum emission period for this metric is a minute.
  • Events processed: Number of events processed per emission period. Events processed successfully are persisted as output rows.
  • Events arrived too late (Later than 30 days): Number of events rejected because the events are either null, filtered out, or outside the retention period. If an event is missing a timestamp or the timestamp cannot be parsed, the event is discarded.
  • Unparseable events: Number of events discarded because Polaris can't parse them.
  • Rows output: Number of rows output from processed events. For tables with rollup enabled, you should expect to see fewer output rows than incoming rows.

Detailed metrics​

The Detailed metrics page provides a data cube you can use to investigate specific metrics with the option to filter by table, query type, and query ID.

Detailed metrics data cube

Learn more​

See the following topics for more information: