// This assembly is included in the following assemblies:
//
// metrics/assembly_metrics-prometheus-alertmanager.adoc

[id='ref-metrics-alertmanager-examples-{context}']

= Alerting rule examples

Example alerting rules for Kafka and ZooKeeper metrics are provided with {ProductName} for use in a xref:proc-metrics-deploying-prometheus-{context}[Prometheus deployment].

General points about the alerting rule definitions:

* A `for` property is used with the rules to determine the period of time a condition must persist before an alert is triggered.
* A tick is a basic ZooKeeper time unit, which is measured in milliseconds and configured using the `tickTime` parameter of `Kafka.spec.zookeeper.config`. For example, if ZooKeeper `tickTime=3000`, 3 ticks (3 x 3000) equals 9000 milliseconds.
* The availability of the `ZookeeperRunningOutOfSpace` metric and alert is dependent on the Kubernetes configuration and storage implementation used. Storage implementations for certain platforms may not be able to supply the information on available space required for the metric to provide an alert.

.Kafka alerting rules

`UnderReplicatedPartitions`:: Gives the number of partitions for which the current broker is the lead replica but which have fewer replicas than the `min.insync.replicas` configured for their topic.
This metric provides insights about brokers that host the follower replicas. Those followers are not keeping up with the leader.
Reasons for this could include being (or having been) offline, and over-throttled interbroker replication.
An alert is raised when this value is greater than zero, providing information on the under-replicated partitions for each broker.

`AbnormalControllerState`:: Indicates whether the current broker is the controller for the cluster.
The metric can be 0 or 1.
During the life of a cluster, only one broker should be the controller and the cluster always needs to have an active controller.
Having two or more brokers saying that they are controllers indicates a problem.
If the condition persists, an alert is raised when the sum of all the values for this metric on all brokers is not equal to 1, meaning that there is no active controller (the sum is 0) or more than one controller (the sum is greater than 1).

`UnderMinIsrPartitionCount`:: Indicates that the minimum number of in-sync replicas (ISRs) for a lead Kafka broker, specified using `min.insync.replicas`, that must acknowledge a write operation has not been reached.
The metric defines the number of partitions that the broker leads for which the in-sync replicas count is less than the minimum in-sync.
An alert is raised when this value is greater than zero, providing information on the partition count for each broker that did not achieve the minimum number of acknowledgments.

`OfflineLogDirectoryCount`:: Indicates the number of log directories which are offline (for example, due to a hardware failure) so that the broker cannot store incoming messages anymore.
An alert is raised when this value is greater than zero, providing information on the number of offline log directories for each broker.

`KafkaRunningOutOfSpace`:: Indicates the remaining amount of disk space that can be used for writing data.
An alert is raised when this value is lower than 5GiB, providing information on the disk that is running out of space for each persistent volume claim.
The threshold value may be changed in `prometheus-rules.yaml`.

.ZooKeeper alerting rules

`AvgRequestLatency`:: Indicates the amount of time it takes for the server to respond to a client request.
An alert is raised when this value is greater than 10 (ticks), providing the actual value of the average request latency for each server.

`OutstandingRequests`:: Indicates the number of queued requests in the server.
This value goes up when the server receives more requests than it can process.
An alert is raised when this value is greater than 10, providing the actual number of outstanding requests for each server.

`ZookeeperRunningOutOfSpace`:: Indicates the remaining amount of disk space that can be used for writing data to ZooKeeper.
An alert is raised when this value is lower than 5GiB., providing information on the disk that is running out of space for each persistent volume claim.
