// Module included in the following assemblies:
//
// assembly-security.adoc

[id='proc-recovering-expired-ca-certs-{context}']

= Manually recovering from expired Cluster Operator-managed CA certificates

[role="_abstract"]
The Cluster Operator automatically renews the cluster and clients CA certificates when their renewal periods begin. 
Nevertheless, unexpected operational problems or disruptions may prevent the renewal process, such as prolonged downtime of the Cluster Operator or unavailability of the Kafka cluster. 
If CA certificates expire, Kafka cluster components cannot communicate with each other and the Cluster Operator cannot renew the CA certificates without manual intervention. 

To promptly perform a recovery, follow the steps outlined in this procedure in the order given. 
You can recover from expired cluster and clients CA certificates. 
The process involves deleting the secrets containing the expired certificates so that new ones are generated by the Cluster Operator. 
For more information on the secrets managed in Strimzi, see xref:con-certificates-str[].

NOTE: If you are using your own CA certificates and they expire, the process is similar, but you need to xref:renewing-your-own-ca-certificates-{context}[renew the CA certificates] rather than use certificates generated by the Cluster Operator.

.Prerequisites

* xref:deploying-cluster-operator-str[The Cluster Operator must be deployed.]
* A Kafka cluster in which CA certificates and private keys are installed.
* The OpenSSL TLS management tool to check the period of validity for CA certificates.

In this procedure, we use a Kafka cluster named `my-cluster` within the `my-project` namespace.

.Procedure

. Delete the secret containing the expired CA certificate.
+
.Deleting the Cluster CA secret
[source,shell]
----
kubectl delete secret my-cluster-cluster-ca-cert -n my-project
----
+
.Deleting the Clients CA secret
[source,shell]
----
kubectl delete secret my-cluster-clients-ca-cert -n my-project
----

. Wait for the Cluster Operator to generate new certificates. 
+
* A new CA cluster certificate to verify the identity of the Kafka brokers is created in a secret of the same name (`my-cluster-cluster-ca-cert`).
* A new CA clients certificate to verify the identity of Kafka users is created in a secret of the same name (`my-cluster-clients-ca-cert`).

. Check the period of validity for the new CA certificates.
+
.Checking the period of validity for the new cluster CA certificate
[source,shell]
----
kubectl get secret my-cluster-cluster-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates
----
+
.Checking the period of validity for the new clients CA certificate
[source,shell]
----
kubectl get secret my-cluster-clients-ca-cert -n my-project -o=jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates
----
+
The command returns a `notBefore` and `notAfter` date, which is the valid start and end date for the CA certificate.

. Delete the component pods and secrets that use the CA certificates. 
+
--
.. Delete the ZooKeeper secret.
.. Wait for the Cluster Operator to detect the missing ZooKeeper secret and recreate it.
.. Delete all ZooKeeper pods.
.. Delete the Kafka secret.
.. Wait for the Cluster Operator to detect the missing Kafka secret and recreate it.
.. Delete all Kafka pods. 
--
+
If you are only recovering the clients CA certificate, you only need to delete the Kafka secret and pods.
+
You can use the following `kubectl` command to find resources and also verify that they have been removed.
+
[source,shell]
----
kubectl get <resource_type> --all-namespaces | grep <kafka_cluster_name>
----
+
Replace `<resource_type>` with the type of the resource, such as `Pod` or `Secret`.

. Wait for the Cluster Operator to detect the missing Kafka and ZooKeeper pods and recreate them with the updated CA certificates.
+
On reconciliation, the Cluster Operator automatically updates other components to trust the new CA certificates.

. Verify that there are no issues related to certificate validation in the Cluster Operator log.
. Update client configurations to trust the new cluster CA certificate.
+
See:
+
--
* xref:configuring-internal-clients-to-trust-cluster-ca-str[]
* xref:configuring-external-clients-to-trust-cluster-ca-str[]
--