// file included in the following:
//
// kroxylicious-operator/index.adoc

[id='con-prometheus-metrics-proxy-{context}']
= Overview of proxy metrics

[role="_abstract"]

The proxy provides metrics for both connections and messages.
These metrics are categorized into downstream (client-side) and upstream (broker-side) groups
They allow users to assess the impact of the proxy and its filters on their Kafka system.

* Connection metrics count the connections made from the downstream (incoming connections from the clients) and the connection made by the proxy to upstream (outgoing connections to the Kafka brokers).
* Message metrics count the number of {kafka-protocol}#protocol_messages[Kafka protocol request and response messages] that flow through the proxy.

== Connection metrics

Connection metrics count the TCP connections made from the client to the proxy (`kroxylicious_client_to_proxy_request_total`) and from the proxy to the broker (`kroxylicious_proxy_to_server_connections_total`).
These metrics count connection _attempts_, so the connection count is incremented even if the connection attempt ultimately fails.

In addition to the count metrics, there are error metrics.
* If an error occurs whilst the proxy is accepting a connection from the client the `kroxylicious_client_to_proxy_errors_total` metric is incremented by one.
* If an error occurs whilst the proxy is attempting a connection to a broker the `kroxylicious_proxy_to_server_errors_total` metric is incremented by one.

Connection and connection error metrics include the following labels: `virtual_cluster` (the virtual cluster's name) and `node_id` (the broker's node ID).
When the client connects to the boostrap endpoint of the virtual cluster, a node ID value of `bootstrap` is recorded.

The `kroxylicious_client_to_proxy_errors_total` metric also counts connection errors that occur before a virtual cluster has been identified.
For these specific errors, the `virtual_cluster` and `node_id` labels are set to an empty string ("").

NOTE: Error conditions signaled _within_ the Kafka protocol response (such as `RESOURCE_NOT_FOUND` or `UNKNOWN_TOPIC_ID`) are not classed as errors by these metrics.

.Connection metrics for client and broker interactions
|===
|Metric Name |Type |Labels|Description

|`kroxylicious_client_to_proxy_connection_total`
|Counter
|`virtual_cluster`, `node_id`
|Incremented by one every time a connection is accepted from a client by the proxy. +
 This metric counts all connection attempts that reach the proxy, even those that end in error.

|`kroxylicious_client_to_proxy_errors_total`
|Counter
|`virtual_cluster`, `node_id`
|Incremented by one every time a connection is closed due to any downstream error.

|`kroxylicious_proxy_to_server_connections_total`
|Counter
|`virtual_cluster`, `node_id`
|Incremented by one every time a connection is made to the server from the proxy. +
 This metric counts all connections attempted to the broker, even those that end in error.

|`kroxylicious_proxy_to_server_errors_total`
|Counter
|`virtual_cluster`, `node_id`
|Incremented by one every time a connection is closed due to any upstream error.
|===

== Message metrics

Message metrics count, and record the sizes of, the Kafka protocol requests and responses that flow through the proxy.

Use these metrics to help understand:
* the number of messages flowing through the proxy.
* the overall volume of data through the proxy.
* the effect the filters are having on the messages.

* Downstream metrics
** `kroxylicious_client_to_proxy_request_total` counts requests as they arrive from the client.
** `kroxylicious_proxy_to_client_response_total` counts responses as they are returned to the client.
** `kroxylicious_client_to_proxy_request_size_bytes` is incremented by the size of each request as it arrives from the client.
** `kroxylicious_proxy_to_client_response_size_bytes` is incremented by the size of each response as it is returned to the client.

* Upstream metrics
** `kroxylicious_proxy_to_server_request_total` counts requests as they go to the broker.
** `kroxylicious_proxy_to_server_response_total` counts responses as they are returned by the broker.
** `kroxylicious_proxy_to_server_request_size_bytes` is incremented by the size of each request as it goes to the broker.
** `kroxylicious_proxy_to_server_response_size_bytes` is incremented by the size of each response as it is returned by the broker.

The size recorded is the encoded size of the protocol message. It includes the 4 byte {kafka-protocol}#protocol_common[message size].

Filters can alter the flow of messages through the proxy or the content of the message.
This is apparent through the metrics.
* If a filter sends a short-circuit, or closes a connection the downstream message counters will exceed the upstream counters.
* If a filter changes the size of the message, the downstream size metrics will be different to the upstream size metrics.

.Downstream and upstream message metrics in the proxy
image::../_assets/monitoring-message-counters.svg["Conceptual diagram showing the downstream and upstream message metrics within the proxy, illustrating how they respond to message transit through it."]

Message metrics include the following labels: `virtual_cluster` (the virtual cluster's name), `node_id` (the broker's node ID), `api_key` (the message type), `api_version`, and `decoded` (a flag indicating if the message was decoded by the proxy).

When the client connects to the boostrap endpoint of the virtual cluster, metrics are recorded with a node ID value of `bootstrap`.

.Kafka message metrics for proxy request and response flow
|===
|Metric Name |Type |Labels|Description

|`kroxylicious_client_to_proxy_request_total`
|Counter
|`virtual_cluster`, `node_id`, `api_key`, `api_version`, `decoded`
|Incremented by one every time a request arrives at the proxy from a client.

|`kroxylicious_proxy_to_server_request_total`
|Counter
|`virtual_cluster`, `node_id`, `api_key`, `api_version`, `decoded`
|Incremented by one every time a request goes from the proxy to a server.

|`kroxylicious_server_to_proxy_response_total`
|Counter
|`virtual_cluster`, `node_id`, `api_key`, `api_version`, `decoded`
|Incremented by one every time a response arrives at the proxy from a server.

|`kroxylicious_proxy_to_client_response_total`
|Counter
|`virtual_cluster`, `node_id`, `api_key`, `api_version`, `decoded`
|Incremented by one every time a response goes from the proxy to a client.

|`kroxylicious_client_to_proxy_request_total`
|Distribution
|`virtual_cluster`, `node_id`, `api_key`, `api_version`, `decoded`
|Incremented by the size of the message each time a request arrives at the proxy from a client.

|`kroxylicious_proxy_to_server_request_total`
|Distribution
|`virtual_cluster`, `node_id`, `api_key`, `api_version`, `decoded`
|Incremented by the size of the message each time a request goes from the proxy to a server.

|`kroxylicious_server_to_proxy_response_total`
|Distribution
|`virtual_cluster`, `node_id`, `api_key`, `api_version`, `decoded`
|Incremented by the size of the message each time a response arrives at the proxy from a server.

|`kroxylicious_proxy_to_client_response_total`
|Distribution
|`virtual_cluster`, `node_id`, `api_key`, `api_version`, `decoded`
|Incremented by the size of the message each time a response goes from the proxy to a client.

|===
