// Module included in the following assemblies:
//
// assembly-cruise-control-concepts.adoc

[id='con-optimization-proposals-{context}']

= Optimization proposals overview

[role="_abstract"]
Configure a `KafkaRebalance` resource to generate optimization proposals and apply the suggested changes.  
An _optimization proposal_ is a summary of proposed changes that would produce a more balanced Kafka cluster, with partition workloads distributed more evenly among the brokers.

Each optimization proposal is based on the set of xref:con-optimization-goals-{context}[optimization goals] that was used to generate it, subject to any configured link:{BookURLConfiguring}#property-cruise-control-broker-capacity-reference[capacity limits on broker resources].

All optimization proposals are _estimates_ of the impact of a proposed rebalance.
You can approve or reject a proposal.
You cannot approve a cluster rebalance without first generating the optimization proposal.

You can run optimization proposals in one of the following rebalancing modes:

* `full`
* `add-brokers`
* `remove-brokers`

[id='con-optimization-proposals-modes-{context}']
== Rebalancing modes

You specify a rebalancing mode using the `spec.mode` property of the `KafkaRebalance` custom resource.

`full`:: The `full` mode runs a full rebalance by moving replicas across all the brokers in the cluster.
This is the default mode if the `spec.mode` property is not defined in the `KafkaRebalance` custom resource.

`add-brokers`:: The `add-brokers` mode is used after scaling up a Kafka cluster by adding one or more brokers.
Normally, after scaling up a Kafka cluster, new brokers are used to host only the partitions of newly created topics.
If no new topics are created, the newly added brokers are not used and the existing brokers remain under the same load.
By using the `add-brokers` mode immediately after adding brokers to the cluster, the rebalancing operation moves replicas from existing brokers to the newly added brokers.
You specify the new brokers as a list using the `spec.brokers` property of the `KafkaRebalance` custom resource.

`remove-brokers`:: The `remove-brokers` mode is used before scaling down a Kafka cluster by removing one or more brokers.
If you scale down a Kafka cluster, brokers are shut down even if they host replicas.
This can lead to under-replicated partitions and possibly result in some partitions being under their minimum ISR (in-sync replicas).
To avoid this potential problem, the `remove-brokers` mode moves replicas off the brokers that are going to be removed.
When these brokers are not hosting replicas anymore, you can safely run the scaling down operation.
You specify the brokers you're removing as a list in the `spec.brokers` property in the `KafkaRebalance` custom resource.

In general, use the `full` rebalance mode to rebalance a Kafka cluster by spreading the load across brokers.
Use the `add-brokers` and `remove-brokers` modes only if you want to scale your cluster up or down and rebalance the replicas accordingly.

The procedure to run a rebalance is actually the same across the three different modes.
The only difference is with specifying a mode through the `spec.mode` property and, if needed, listing brokers that have been added or will be removed through the `spec.brokers` property.


[[contents-optimization-proposals]]
== The results of an optimization proposal

When an optimization proposal is generated, a summary and broker load is returned.

Summary:: The summary is contained in the `KafkaRebalance` resource. The summary provides an overview of the proposed cluster rebalance and indicates the scale of the changes involved.
A summary of a successfully generated optimization proposal is contained in the `Status.OptimizationResult` property of the `KafkaRebalance` resource.
The information provided is a summary of the full optimization proposal.
Broker load:: The broker load is stored in a ConfigMap that contains data as a JSON string. The broker load shows before and after values for the proposed rebalance, so you can see the impact on each of the brokers in the cluster.

== Manually approving or rejecting an optimization proposal

An optimization proposal summary shows the proposed scope of changes.

You can use the name of the `KafkaRebalance` resource to return a summary from the command line.

.Returning an optimization proposal summary
[source,shell,subs=+quotes]
----
kubectl describe kafkarebalance _<kafka_rebalance_resource_name>_ -n _<namespace>_
----

You can also use the `jq` command line JSON parser tool.

.Returning an optimization proposal summary using jq
[source,shell,subs=+quotes]
----
`kubectl get kafkarebalance -o json | jq _<jq_query>_`.
----

Use the summary to decide whether to approve or reject an optimization proposal.

Approving an optimization proposal:: You approve the optimization proposal by setting the `strimzi.io/rebalance` annotation of the `KafkaRebalance` resource to `approve`.
Cruise Control applies the proposal to the Kafka cluster and starts a cluster rebalance operation.
Rejecting an optimization proposal:: If you choose not to approve an optimization proposal,
you can xref:proc-generating-optimization-proposals-str[change the optimization goals] or xref:rebalance_tuning_options[update any of the rebalance performance tuning options], and then generate another proposal.
You can generate a new optimization proposal for a `KafkaRebalance` resource by setting the `strimzi.io/rebalance` annotation to `refresh`.

Use optimization proposals to assess the movements required for a rebalance.
For example, a summary describes inter-broker and intra-broker movements.
Inter-broker rebalancing moves data between separate brokers.
Intra-broker rebalancing moves data between disks on the same broker when you are using a JBOD storage configuration.
Such information can be useful even if you don't go ahead and approve the proposal.

You might reject an optimization proposal, or delay its approval, because of the additional load on a Kafka cluster when rebalancing.

In the following example, the proposal suggests the rebalancing of data between separate brokers.
The rebalance involves the movement of 55 partition replicas, totaling 12MB of data, across the brokers.
Though the inter-broker movement of partition replicas has a high impact on performance, the total amount of data is not large.
If the total data was much larger, you could reject the proposal, or time when to approve the rebalance to limit the impact on the performance of the Kafka cluster.

Rebalance performance tuning options can help reduce the impact of data movement.
If you can extend the rebalance period, you can divide the rebalance into smaller batches.
Fewer data movements at a single time reduces the load on the cluster.

.Example optimization proposal summary
[source,yaml]
----
Name:         my-rebalance
Namespace:    myproject
Labels:       strimzi.io/cluster=my-cluster
Annotations:  API Version:  kafka.strimzi.io/v1alpha1
Kind:         KafkaRebalance
Metadata:
# ...
Status:
  Conditions:
    Last Transition Time:  2022-04-05T14:36:11.900Z
    Status:                ProposalReady
    Type:                  State
  Observed Generation:     1
  Optimization Result:
    Data To Move MB:  0
    Excluded Brokers For Leadership:
    Excluded Brokers For Replica Move:
    Excluded Topics:
    Intra Broker Data To Move MB:         12
    Monitored Partitions Percentage:      100
    Num Intra Broker Replica Movements:   0
    Num Leader Movements:                 24
    Num Replica Movements:                55
    On Demand Balancedness Score After:   82.91290759174306
    On Demand Balancedness Score Before:  78.01176356230222
    Recent Windows:                       5
  Session Id:                             a4f833bd-2055-4213-bfdd-ad21f95bf184
----

The proposal will also move 24 partition leaders to different brokers.
This requires a change to the ZooKeeper configuration, which has a low impact on performance.

The balancedness scores are measurements of the overall balance of the Kafka cluster before and after the optimization proposal is approved.
A balancedness score is based on optimization goals.
If all goals are satisfied, the score is 100.
The score is reduced for each goal that will not be met.
Compare the balancedness scores to see whether the Kafka cluster is less balanced than it could be following a rebalance.

== Automatically approving an optimization proposal

To save time, you can automate the process of approving optimization proposals.
With automation, when you generate an optimization proposal it goes straight into a cluster rebalance.

To enable the optimization proposal auto-approval mechanism, create the `KafkaRebalance` resource with the `strimzi.io/rebalance-auto-approval` annotation set to `true`.
If the annotation is not set or set to `false`, the optimization proposal requires manual approval.

.Example rebalance request with auto-approval mechanism enabled
[source,yaml]
----
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
  annotations:
    strimzi.io/rebalance-auto-approval: "true"
spec:
  mode: # any mode
  # ...
----

You can still check the status when automatically approving an optimization proposal.
The status of the `KafkaRebalance` resource moves to `Ready` when the rebalance is complete.

== Optimization proposal summary properties

The following table explains the properties contained in the optimization proposal's summary section.

.Properties contained in an optimization proposal summary
[cols="35,65",options="header",stripes="none",separator=¦]
|===

m¦JSON property
¦Description

m¦numIntraBrokerReplicaMovements
¦The total number of partition replicas that will be transferred between the disks of the cluster's brokers.

*Performance impact during rebalance operation*: Relatively high, but lower than `numReplicaMovements`.

m¦excludedBrokersForLeadership
¦Not yet supported. An empty list is returned.

m¦numReplicaMovements
¦The number of partition replicas that will be moved between separate brokers.

*Performance impact during rebalance operation*: Relatively high.

m¦onDemandBalancednessScoreBefore, onDemandBalancednessScoreAfter
¦A measurement of the overall _balancedness_ of a Kafka Cluster, before and after the optimization proposal was generated.

The score is calculated by subtracting the sum of the `BalancednessScore` of each violated soft goal from 100. Cruise Control assigns a `BalancednessScore` to every optimization goal based on several factors, including priority--the goal's position in the list of `default.goals` or user-provided goals.

The `Before` score is based on the current configuration of the Kafka cluster.
The `After` score is based on the generated optimization proposal.

m¦intraBrokerDataToMoveMB
¦The sum of the size of each partition replica that will be moved between disks on the same broker (see also `numIntraBrokerReplicaMovements`).

*Performance impact during rebalance operation*: Variable. The larger the number, the longer the cluster rebalance will take to complete. Moving a large amount of data between disks on the same broker has less impact than between separate brokers (see `dataToMoveMB`).

m¦recentWindows
¦The number of metrics windows upon which the optimization proposal is based.

m¦dataToMoveMB
¦The sum of the size of each partition replica that will be moved to a separate broker (see also `numReplicaMovements`).

*Performance impact during rebalance operation*: Variable. The larger the number, the longer the cluster rebalance will take to complete.

m¦monitoredPartitionsPercentage
¦The percentage of partitions in the Kafka cluster covered by the optimization proposal. Affected by the number of `excludedTopics`.

m¦excludedTopics
¦If you specified a regular expression in the `spec.excludedTopicsRegex` property in the `KafkaRebalance` resource, all topic names matching that expression are listed here.
These topics are excluded from the calculation of partition replica/leader movements in the optimization proposal.

m¦numLeaderMovements
¦The number of partitions whose leaders will be switched to different replicas. This involves a change to ZooKeeper configuration.

*Performance impact during rebalance operation*: Relatively low.

m¦excludedBrokersForReplicaMove
¦Not yet supported. An empty list is returned.

|===

== Broker load properties

The broker load is stored in a ConfigMap (with the same name as the KafkaRebalance custom resource) as a JSON formatted string. This JSON string consists of a JSON object with keys for each broker IDs linking to a number of metrics for each broker.
Each metric consist of three values.
The first is the metric value before the optimization proposal is applied, the second is the expected value of the metric after the proposal is applied, and the third is the difference between the first two values (after minus before).

NOTE: The ConfigMap appears when the KafkaRebalance resource is in the `ProposalReady` state and remains after the rebalance is complete.

You can use the name of the ConfigMap to view its data from the command line.

.Returning ConfigMap data
[source,shell,subs=+quotes]
----
kubectl describe configmaps _<my_rebalance_configmap_name>_ -n _<namespace>_
----

You can also use the `jq` command line JSON parser tool to extract the JSON string from the ConfigMap.

.Extracting the JSON string from the ConfigMap using jq
[source,shell,subs=+quotes]
----
kubectl get configmaps _<my_rebalance_configmap_name>_ -o json | jq '.["data"]["brokerLoad.json"]|fromjson|.'
----

The following table explains the properties contained in the optimization proposal's broker load ConfigMap:

[cols="35,65",options="header",stripes="none"]
|======================================================================================================

| JSON property               | Description

m| leaders                     | The number of replicas on this broker that are partition leaders.

m| replicas                    | The number of replicas on this broker.

m| cpuPercentage               | The CPU utilization as a percentage of the defined capacity.

m| diskUsedPercentage          | The disk utilization as a percentage of the defined capacity.

m| diskUsedMB                  | The absolute disk usage in MB.

m| networkOutRate              | The total network output rate for the broker.

m| leaderNetworkInRate         | The network input rate for all partition leader replicas on this broker.

m| followerNetworkInRate       | The network input rate for all follower replicas on this broker.

m| potentialMaxNetworkOutRate  | The hypothetical maximum network output rate that would be realized if this broker became the leader of all the replicas it currently hosts.

|======================================================================================================

== Cached optimization proposal

Cruise Control maintains a _cached optimization proposal_ based on the configured default optimization goals.
Generated from the workload model, the cached optimization proposal is updated every 15 minutes to reflect the current state of the Kafka cluster.
If you generate an optimization proposal using the default optimization goals, Cruise Control returns the most recent cached proposal.

To change the cached optimization proposal refresh interval, edit the `proposal.expiration.ms` setting in the Cruise Control deployment configuration.
Consider a shorter interval for fast changing clusters, although this increases the load on the Cruise Control server.

[role="_additional-resources"]
.Additional resources

* xref:con-optimization-goals-{context}[]
* xref:proc-generating-optimization-proposals-{context}[]
* xref:proc-approving-optimization-proposal-{context}[]
