Streaming metrics reference

This topic provides a reference for the streaming metrics available in Imply Polaris. You can use the streaming dashboards to monitor and analyze issues concerning lag, event processing, and real-time queries.

See Monitoring metrics for prerequities and information on how to navigate the metrics dashboards. For information on all monitoring features available in Polaris, see the Monitoring overview.

Click Streaming in the Monitoring section of the left pane to access the dashboards.

Lag

This section describes the lag dashboards. Lag refers to the period of time between publication of the latest data records into a stream and successful processing by Polaris.

Average Kafka lag (events)

Shows the average lag per Kafka stream, shown as the number of events Polaris ingested into each table. The metric is blank if there are no active Kafka streams.

Average Kinesis lag (milliseconds)

Shows the average Kinesis lag time in milliseconds.

Event processing

This section describes the event processing dashboards. Polaris handles streamed events in one of three ways:

• Processes successfully
• Filters out
• Processes with an exception

Events processed

Shows the number of events Polaris successfully processed from data streams, grouped by table. Note that Polaris includes events processed with errors in both this metric and in the Events with processing exceptions metric.

Events filtered out

Shows the number of events Polaris filtered out, grouped by table. Polaris filters messages out if they're empty, outside the message rejection period, or explicitly filtered out in the job spec.

Events with processing exceptions

Shows the number of events Polaris couldn't process or processed with errors, grouped by table. Note that Polaris includes events processed with errors in both this metric and in the Events processed metric.

Rows output

Shows the number of rows output from events processed by Polaris, grouped by table. For tables with rollup enabled, you should expect to see fewer output rows than input rows.

Real-time queries

This section describes the real-time queries dashboards.

Queries

Shows the number of queries Polaris executed against tables containing data recently read from a stream.

p98 query time

Shows the 98th percentile of time taken for Polaris to complete a real-time query, grouped by table.

p98 query wait time

Shows the 98th percentile of time taken for Polaris to run a real-time query, grouped by table.

Query CPU time

Shows the CPU time Polaris spent processing a real-time query, in milliseconds, grouped by table.