--- reviewers: - sig-cluster-lifecycle title: Certificate Management with kubeadm content_type: task weight: 80 --- {{< feature-state for_k8s_version="v1.15" state="stable" >}} Client certificates generated by [kubeadm](/docs/reference/setup-tools/kubeadm/) expire after 1 year. This page explains how to manage certificate renewals with kubeadm. It also covers other tasks related to kubeadm certificate management. The Kubernetes project recommends upgrading to the latest patch releases promptly, and to ensure that you are running a supported minor release of Kubernetes. Following this recommendation helps you to stay secure. ## {{% heading "prerequisites" %}} You should be familiar with [PKI certificates and requirements in Kubernetes](/docs/setup/best-practices/certificates/). You should be familiar with how to pass a [configuration](/docs/reference/config-api/kubeadm-config.v1beta4/) file to the kubeadm commands. This guide covers the usage of the `openssl` command (used for manual certificate signing, if you choose that approach), but you can use your preferred tools. Some of the steps here use `sudo` for administrator access. You can use any equivalent tool. ## Using custom certificates {#custom-certificates} By default, kubeadm generates all the certificates needed for a cluster to run. You can override this behavior by providing your own certificates. To do so, you must place them in whatever directory is specified by the `--cert-dir` flag or the `certificatesDir` field of kubeadm's `ClusterConfiguration`. By default this is `/etc/kubernetes/pki`. If a given certificate and private key pair exists before running `kubeadm init`, kubeadm does not overwrite them. This means you can, for example, copy an existing CA into `/etc/kubernetes/pki/ca.crt` and `/etc/kubernetes/pki/ca.key`, and kubeadm will use this CA for signing the rest of the certificates. ## Choosing an encryption algorithm {#choosing-encryption-algorithm} kubeadm allows you to choose an encryption algorithm that is used for creating public and private keys. That can be done by using the `encryptionAlgorithm` field of the kubeadm configuration: ```yaml apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration encryptionAlgorithm: ``` `` can be one of `RSA-2048` (default), `RSA-3072`, `RSA-4096` or `ECDSA-P256`. ## Choosing certificate validity period {#choosing-cert-validity-period} kubeadm allows you to choose the validity period of CA and leaf certificates. That can be done by using the `certificateValidityPeriod` and `caCertificateValidityPeriod` fields of the kubeadm configuration: ```yaml apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration certificateValidityPeriod: 8760h # Default: 365 days × 24 hours = 1 year caCertificateValidityPeriod: 87600h # Default: 365 days × 24 hours * 10 = 10 years ``` The values of the fields follow the accepted format for [Go's `time.Duration` values](https://pkg.go.dev/time#ParseDuration), with the longest supported unit being `h` (hours). ## External CA mode {#external-ca-mode} It is also possible to provide only the `ca.crt` file and not the `ca.key` file (this is only available for the root CA file, not other cert pairs). If all other certificates and kubeconfig files are in place, kubeadm recognizes this condition and activates the "External CA" mode. kubeadm will proceed without the CA key on disk. Instead, run the controller-manager standalone with `--controllers=csrsigner` and point to the CA certificate and key. There are various ways to prepare the component credentials when using external CA mode. ### Manual preparation of component credentials [PKI certificates and requirements](/docs/setup/best-practices/certificates/) includes information on how to prepare all the required by kubeadm component credentials manually. This guide covers the usage of the `openssl` command (used for manual certificate signing, if you choose that approach), but you can use your preferred tools. ### Preparation of credentials by signing CSRs generated by kubeadm kubeadm can [generate CSR files](#signing-csr) that you can sign manually with tools like `openssl` and your external CA. These CSR files will include all the specification for credentials that components deployed by kubeadm require. ### Automated preparation of component credentials by using kubeadm phases Alternatively, it is possible to use kubeadm phase commands to automate this process. - Go to a host that you want to prepare as a kubeadm control plane node with external CA. - Copy the external CA files `ca.crt` and `ca.key` that you have into `/etc/kubernetes/pki` on the node. - Prepare a temporary [kubeadm configuration file](/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file) called `config.yaml` that can be used with `kubeadm init`. Make sure that this file includes any relevant cluster wide or host-specific information that could be included in certificates, such as, `ClusterConfiguration.controlPlaneEndpoint`, `ClusterConfiguration.certSANs` and `InitConfiguration.APIEndpoint`. - On the same host execute the commands `kubeadm init phase kubeconfig all --config config.yaml` and `kubeadm init phase certs all --config config.yaml`. This will generate all required kubeconfig files and certificates under `/etc/kubernetes/` and its `pki` sub directory. - Inspect the generated files. Delete `/etc/kubernetes/pki/ca.key`, delete or move to a safe location the file `/etc/kubernetes/super-admin.conf`. - On nodes where `kubeadm join` will be called also delete `/etc/kubernetes/kubelet.conf`. This file is only required on the first node where `kubeadm init` will be called. - Note that some files such `pki/sa.*`, `pki/front-proxy-ca.*` and `pki/etc/ca.*` are shared between control plane nodes, You can generate them once and [distribute them manually](/docs/setup/production-environment/tools/kubeadm/high-availability/#manual-certs) to nodes where `kubeadm join` will be called, or you can use the [`--upload-certs`](/docs/setup/production-environment/tools/kubeadm/high-availability/#stacked-control-plane-and-etcd-nodes) functionality of `kubeadm init` and `--certificate-key` of `kubeadm join` to automate this distribution. Once the credentials are prepared on all nodes, call `kubeadm init` and `kubeadm join` for these nodes to join the cluster. kubeadm will use the existing kubeconfig and certificate files under `/etc/kubernetes/` and its `pki` sub directory. ## Certificate expiry and management {#check-certificate-expiration} {{< note >}} `kubeadm` cannot manage certificates signed by an external CA. {{< /note >}} You can use the `check-expiration` subcommand to check when certificates expire: ```shell kubeadm certs check-expiration ``` The output is similar to this: ```console CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Dec 30, 2020 23:36 UTC 364d no apiserver Dec 30, 2020 23:36 UTC 364d ca no apiserver-etcd-client Dec 30, 2020 23:36 UTC 364d etcd-ca no apiserver-kubelet-client Dec 30, 2020 23:36 UTC 364d ca no controller-manager.conf Dec 30, 2020 23:36 UTC 364d no etcd-healthcheck-client Dec 30, 2020 23:36 UTC 364d etcd-ca no etcd-peer Dec 30, 2020 23:36 UTC 364d etcd-ca no etcd-server Dec 30, 2020 23:36 UTC 364d etcd-ca no front-proxy-client Dec 30, 2020 23:36 UTC 364d front-proxy-ca no scheduler.conf Dec 30, 2020 23:36 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Dec 28, 2029 23:36 UTC 9y no etcd-ca Dec 28, 2029 23:36 UTC 9y no front-proxy-ca Dec 28, 2029 23:36 UTC 9y no ``` The command shows expiration/residual time for the client certificates in the `/etc/kubernetes/pki` folder and for the client certificate embedded in the kubeconfig files used by kubeadm (`admin.conf`, `controller-manager.conf` and `scheduler.conf`). Additionally, kubeadm informs the user if the certificate is externally managed; in this case, the user should take care of managing certificate renewal manually/using other tools. The `kubelet.conf` configuration file is not included in the list above because kubeadm configures kubelet for [automatic certificate renewal](/docs/tasks/tls/certificate-rotation/) with rotatable certificates under `/var/lib/kubelet/pki`. To repair an expired kubelet client certificate see [Kubelet client certificate rotation fails](/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert). {{< note >}} On nodes created with `kubeadm init` from versions prior to kubeadm version 1.17, there is a [bug](https://github.com/kubernetes/kubeadm/issues/1753) where you manually have to modify the contents of `kubelet.conf`. After `kubeadm init` finishes, you should update `kubelet.conf` to point to the rotated kubelet client certificates, by replacing `client-certificate-data` and `client-key-data` with: ```yaml client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem client-key: /var/lib/kubelet/pki/kubelet-client-current.pem ``` {{< /note >}} ## Automatic certificate renewal kubeadm renews all the certificates during control plane [upgrade](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/). This feature is designed for addressing the simplest use cases; if you don't have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure. If you have more complex requirements for certificate renewal, you can opt out from the default behavior by passing `--certificate-renewal=false` to `kubeadm upgrade apply` or to `kubeadm upgrade node`. ## Manual certificate renewal You can renew your certificates manually at any time with the `kubeadm certs renew` command, with the appropriate command line options. If you are running cluster with a replicated control plane, this command needs to be executed on all the control-plane nodes. This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in `/etc/kubernetes/pki`. `kubeadm certs renew` uses the existing certificates as the authoritative source for attributes (Common Name, Organization, subject alternative name) and does not rely on the `kubeadm-config` ConfigMap. Even so, the Kubernetes project recommends keeping the served certificate and the associated values in that ConfigMap synchronized, to avoid any risk of confusion. After running the command you should restart the control plane Pods. This is required since dynamic certificate reload is currently not supported for all components and certificates. [Static Pods](/docs/tasks/configure-pod-container/static-pod/) are managed by the local kubelet and not by the API Server, thus kubectl cannot be used to delete and restart them. To restart a static Pod you can temporarily remove its manifest file from `/etc/kubernetes/manifests/` and wait for 20 seconds (see the `fileCheckFrequency` value in [KubeletConfiguration struct](/docs/reference/config-api/kubelet-config.v1beta1/)). The kubelet will terminate the Pod if it's no longer in the manifest directory. You can then move the file back and after another `fileCheckFrequency` period, the kubelet will recreate the Pod and the certificate renewal for the component can complete. `kubeadm certs renew` can renew any specific certificate or, with the subcommand `all`, it can renew all of them: ```shell # If you are running cluster with a replicated control plane, this command # needs to be executed on all the control-plane nodes. kubeadm certs renew all ``` ### Copying the administrator certificate (optional) {#admin-certificate-copy} Clusters built with kubeadm often copy the `admin.conf` certificate into `$HOME/.kube/config`, as instructed in [Creating a cluster with kubeadm](/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/). On such a system, to update the contents of `$HOME/.kube/config` after renewing the `admin.conf`, you could run the following commands: ```shell sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` ## Renew certificates with the Kubernetes certificates API This section provides more details about how to execute manual certificate renewal using the Kubernetes certificates API. {{< caution >}} These are advanced topics for users who need to integrate their organization's certificate infrastructure into a kubeadm-built cluster. If the default kubeadm configuration satisfies your needs, you should let kubeadm manage certificates instead. {{< /caution >}} ### Set up a signer The Kubernetes Certificate Authority does not work out of the box. You can configure an external signer such as [cert-manager](https://cert-manager.io/docs/configuration/ca/), or you can use the built-in signer. The built-in signer is part of [`kube-controller-manager`](/docs/reference/command-line-tools-reference/kube-controller-manager/). To activate the built-in signer, you must pass the `--cluster-signing-cert-file` and `--cluster-signing-key-file` flags. If you're creating a new cluster, you can use a kubeadm [configuration file](/docs/reference/config-api/kubeadm-config.v1beta4/): ```yaml apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration controllerManager: extraArgs: - name: "cluster-signing-cert-file" value: "/etc/kubernetes/pki/ca.crt" - name: "cluster-signing-key-file" value: "/etc/kubernetes/pki/ca.key" ``` ### Create certificate signing requests (CSR) See [Create CertificateSigningRequest](/docs/reference/access-authn-authz/certificate-signing-requests/#create-certificatessigningrequest) for creating CSRs with the Kubernetes API. ## Renew certificates with external CA This section provide more details about how to execute manual certificate renewal using an external CA. To better integrate with external CAs, kubeadm can also produce certificate signing requests (CSRs). A CSR represents a request to a CA for a signed certificate for a client. In kubeadm terms, any certificate that would normally be signed by an on-disk CA can be produced as a CSR instead. A CA, however, cannot be produced as a CSR. ### Renewal by using certificate signing requests (CSR) Renewal of ceritficates is possible by generating new CSRs and signing them with the external CA. For more details about working with CSRs generated by kubeadm see the section [Signing certificate signing requests (CSR) generated by kubeadm](#signing-csr). ## Certificate authority (CA) rotation {#certificate-authority-rotation} Kubeadm does not support rotation or replacement of CA certificates out of the box. For more information about manual rotation or replacement of CA, see [manual rotation of CA certificates](/docs/tasks/tls/manual-rotation-of-ca-certificates/). ## Enabling signed kubelet serving certificates {#kubelet-serving-certs} By default the kubelet serving certificate deployed by kubeadm is self-signed. This means a connection from external services like the [metrics-server](https://github.com/kubernetes-sigs/metrics-server) to a kubelet cannot be secured with TLS. To configure the kubelets in a new kubeadm cluster to obtain properly signed serving certificates you must pass the following minimal configuration to `kubeadm init`: ```yaml apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration serverTLSBootstrap: true ``` If you have already created the cluster you must adapt it by doing the following: - Find and edit the `kubelet-config` ConfigMap in the `kube-system` namespace. In that ConfigMap, the `kubelet` key has a [KubeletConfiguration](/docs/reference/config-api/kubelet-config.v1beta1/) document as its value. Edit the KubeletConfiguration document to set `serverTLSBootstrap: true`. - On each node, add the `serverTLSBootstrap: true` field in `/var/lib/kubelet/config.yaml` and restart the kubelet with `systemctl restart kubelet` The field `serverTLSBootstrap: true` will enable the bootstrap of kubelet serving certificates by requesting them from the `certificates.k8s.io` API. One known limitation is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically approved by the default signer in the kube-controller-manager - [`kubernetes.io/kubelet-serving`](/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers). This will require action from the user or a third party controller. These CSRs can be viewed using: ```shell kubectl get csr ``` ```console NAME AGE SIGNERNAME REQUESTOR CONDITION csr-9wvgt 112s kubernetes.io/kubelet-serving system:node:worker-1 Pending csr-lz97v 1m58s kubernetes.io/kubelet-serving system:node:control-plane-1 Pending ``` To approve them you can do the following: ```shell kubectl certificate approve ``` By default, these serving certificate will expire after one year. Kubeadm sets the `KubeletConfiguration` field `rotateCertificates` to `true`, which means that close to expiration a new set of CSRs for the serving certificates will be created and must be approved to complete the rotation. To understand more see [Certificate Rotation](/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/#certificate-rotation). If you are looking for a solution for automatic approval of these CSRs it is recommended that you contact your cloud provider and ask if they have a CSR signer that verifies the node identity with an out of band mechanism. {{% thirdparty-content %}} Third party custom controllers can be used: - [kubelet-csr-approver](https://github.com/postfinance/kubelet-csr-approver) Such a controller is not a secure mechanism unless it not only verifies the CommonName in the CSR but also verifies the requested IPs and domain names. This would prevent a malicious actor that has access to a kubelet client certificate to create CSRs requesting serving certificates for any IP or domain name. ## Generating kubeconfig files for additional users {#kubeconfig-additional-users} During cluster creation, `kubeadm init` signs the certificate in the `super-admin.conf` to have `Subject: O = system:masters, CN = kubernetes-super-admin`. [`system:masters`](/docs/reference/access-authn-authz/rbac/#user-facing-roles) is a break-glass, super user group that bypasses the authorization layer (for example, [RBAC](/docs/reference/access-authn-authz/rbac/)). The file `admin.conf` is also created by kubeadm on control plane nodes and it contains a certificate with `Subject: O = kubeadm:cluster-admins, CN = kubernetes-admin`. `kubeadm:cluster-admins` is a group logically belonging to kubeadm. If your cluster uses RBAC (the kubeadm default), the `kubeadm:cluster-admins` group is bound to the [`cluster-admin`](/docs/reference/access-authn-authz/rbac/#user-facing-roles) ClusterRole. {{< warning >}} Avoid sharing the `super-admin.conf` or `admin.conf` files. Instead, create least privileged access even for people who work as administrators and use that least privilege alternative for anything other than break-glass (emergency) access. {{< /warning >}} You can use the [`kubeadm kubeconfig user`](/docs/reference/setup-tools/kubeadm/kubeadm-kubeconfig) command to generate kubeconfig files for additional users. The command accepts a mixture of command line flags and [kubeadm configuration](/docs/reference/config-api/kubeadm-config.v1beta4/) options. The generated kubeconfig will be written to stdout and can be piped to a file using `kubeadm kubeconfig user ... > somefile.conf`. Example configuration file that can be used with `--config`: ```yaml # example.yaml apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration # Will be used as the target "cluster" in the kubeconfig clusterName: "kubernetes" # Will be used as the "server" (IP or DNS name) of this cluster in the kubeconfig controlPlaneEndpoint: "some-dns-address:6443" # The cluster CA key and certificate will be loaded from this local directory certificatesDir: "/etc/kubernetes/pki" ``` Make sure that these settings match the desired target cluster settings. To see the settings of an existing cluster use: ```shell kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}" ``` The following example will generate a kubeconfig file with credentials valid for 24 hours for a new user `johndoe` that is part of the `appdevs` group: ```shell kubeadm kubeconfig user --config example.yaml --org appdevs --client-name johndoe --validity-period 24h ``` The following example will generate a kubeconfig file with administrator credentials valid for 1 week: ```shell kubeadm kubeconfig user --config example.yaml --client-name admin --validity-period 168h ``` ## Signing certificate signing requests (CSR) generated by kubeadm {#signing-csr} You can create certificate signing requests with `kubeadm certs generate-csr`. Calling this command will generate `.csr` / `.key` file pairs for regular certificates. For certificates embedded in kubeconfig files, the command will generate a `.csr` / `.conf` pair where the key is already embedded in the `.conf` file. A CSR file contains all relevant information for a CA to sign a certificate. kubeadm uses a [well defined specification](/docs/setup/best-practices/certificates/#all-certificates) for all its certificates and CSRs. The default certificate directory is `/etc/kubernetes/pki`, while the default directory for kubeconfig files is `/etc/kubernetes`. These defaults can be overridden with the flags `--cert-dir` and `--kubeconfig-dir`, respectively. To pass custom options to `kubeadm certs generate-csr` use the `--config` flag, which accepts a [kubeadm configuration](/docs/reference/config-api/kubeadm-config.v1beta4/) file, similarly to commands such as `kubeadm init`. Any specification such as extra SANs and custom IP addresses must be stored in the same configuration file and used for all relevant kubeadm commands by passing it as `--config`. {{< note >}} This guide uses the default Kubernetes directory `/etc/kubernetes`, which requires a super user. If you are following this guide and are using directories that you can write to (typically, this means running `kubeadm` with `--cert-dir` and `--kubeconfig-dir`) then you can omit the `sudo` command. You must then copy the files that you produced over to within the `/etc/kubernetes` directory so that `kubeadm init` or `kubeadm join` will find them. {{< /note >}} ### Preparing CA and service account files On the primary control plane node, where `kubeadm init` will be executed, call the following commands: ```shell sudo kubeadm init phase certs ca sudo kubeadm init phase certs etcd-ca sudo kubeadm init phase certs front-proxy-ca sudo kubeadm init phase certs sa ``` This will populate the folders `/etc/kubernetes/pki` and `/etc/kubernetes/pki/etcd` with all self-signed CA files (certificates and keys) and service account (public and private keys) that kubeadm needs for a control plane node. {{< note >}} If you are using an external CA, you must generate the same files out of band and manually copy them to the primary control plane node in `/etc/kubernetes`. Once all CSRs are signed, you can delete the root CA key (`ca.key`) as noted in the [External CA mode](#external-ca-mode) section. {{< /note >}} For secondary control plane nodes (`kubeadm join --control-plane`) there is no need to call the above commands. Depending on how you setup the [High Availability](/docs/setup/production-environment/tools/kubeadm/high-availability) cluster, you either have to manually copy the same files from the primary control plane node, or use the automated `--upload-certs` functionality of `kubeadm init`. ### Generate CSRs The `kubeadm certs generate-csr` command generates CSRs for all known certificates managed by kubeadm. Once the command is done you must manually delete `.csr`, `.conf` or `.key` files that you don't need. #### Considerations for kubelet.conf {#considerations-kubelet-conf} This section applies to both control plane and worker nodes. If you have deleted the `ca.key` file from control plane nodes ([External CA mode](#external-ca-mode)), the active kube-controller-manager in this cluster will not be able to sign kubelet client certificates. If no external method for signing these certificates exists in your setup (such as an [external signer](#set-up-a-signer)), you could manually sign the `kubelet.conf.csr` as explained in this guide. Note that this also means that the automatic [kubelet client certificate rotation](/docs/tasks/tls/certificate-rotation/#enabling-client-certificate-rotation) will be disabled. If so, close to certificate expiration, you must generate a new `kubelet.conf.csr`, sign the certificate, embed it in `kubelet.conf` and restart the kubelet. If this does not apply to your setup, you can skip processing the `kubelet.conf.csr` on secondary control plane and on workers nodes (all nodes that call `kubeadm join ...`). That is because the active kube-controller-manager will be responsible for signing new kubelet client certificates. {{< note >}} You must process the `kubelet.conf.csr` file on the primary control plane node (the host where you originally ran `kubeadm init`). This is because `kubeadm` considers that as the node that bootstraps the cluster, and a pre-populated `kubelet.conf` is needed. {{< /note >}} #### Control plane nodes Execute the following command on primary (`kubeadm init`) and secondary (`kubeadm join --control-plane`) control plane nodes to generate all CSR files: ```shell sudo kubeadm certs generate-csr ``` If external etcd is to be used, follow the [External etcd with kubeadm](/docs/setup/production-environment/tools/kubeadm/high-availability/#external-etcd-nodes) guide to understand what CSR files are needed on the kubeadm and etcd nodes. Other `.csr` and `.key` files under `/etc/kubernetes/pki/etcd` can be removed. Based on the explanation in [Considerations for kubelet.conf](#considerations-kubelet-conf) keep or delete the `kubelet.conf` and `kubelet.conf.csr` files. #### Worker nodes Based on the explanation in [Considerations for kubelet.conf](#considerations-kubelet-conf), optionally call: ```shell sudo kubeadm certs generate-csr ``` and keep only the `kubelet.conf` and `kubelet.conf.csr` files. Alternatively skip the steps for worker nodes entirely. ### Signing CSRs for all certificates {{< note >}} If you are using external CA and already have CA serial number files (`.srl`) for `openssl`, you can copy such files to a kubeadm node where CSRs will be processed. The `.srl` files to copy are `/etc/kubernetes/pki/ca.srl`, `/etc/kubernetes/pki/front-proxy-ca.srl` and `/etc/kubernetes/pki/etcd/ca.srl`. The files can be then moved to a new node where CSR files will be processed. If a `.srl` file is missing for a CA on a node, the script below will generate a new SRL file with a random starting serial number. To read more about `.srl` files see the [`openssl`](https://www.openssl.org/docs/man3.0/man1/openssl-x509.html) documentation for the `--CAserial` flag. {{< /note >}} Repeat this step for all nodes that have CSR files. Write the following script in the `/etc/kubernetes` directory, navigate to the directory and execute the script. The script will generate certificates for all CSR files that are present in the `/etc/kubernetes` tree. ```bash #!/bin/bash # Set certificate expiration time in days DAYS=365 # Process all CSR files except those for front-proxy and etcd find ./ -name "*.csr" | grep -v "pki/etcd" | grep -v "front-proxy" | while read -r FILE; do echo "* Processing ${FILE} ..." FILE=${FILE%.*} # Trim the extension if [ -f "./pki/ca.srl" ]; then SERIAL_FLAG="-CAserial ./pki/ca.srl" else SERIAL_FLAG="-CAcreateserial" fi openssl x509 -req -days "${DAYS}" -CA ./pki/ca.crt -CAkey ./pki/ca.key ${SERIAL_FLAG} \ -in "${FILE}.csr" -out "${FILE}.crt" sleep 2 done # Process all etcd CSRs find ./pki/etcd -name "*.csr" | while read -r FILE; do echo "* Processing ${FILE} ..." FILE=${FILE%.*} # Trim the extension if [ -f "./pki/etcd/ca.srl" ]; then SERIAL_FLAG=-CAserial ./pki/etcd/ca.srl else SERIAL_FLAG=-CAcreateserial fi openssl x509 -req -days "${DAYS}" -CA ./pki/etcd/ca.crt -CAkey ./pki/etcd/ca.key ${SERIAL_FLAG} \ -in "${FILE}.csr" -out "${FILE}.crt" done # Process front-proxy CSRs echo "* Processing ./pki/front-proxy-client.csr ..." openssl x509 -req -days "${DAYS}" -CA ./pki/front-proxy-ca.crt -CAkey ./pki/front-proxy-ca.key -CAcreateserial \ -in ./pki/front-proxy-client.csr -out ./pki/front-proxy-client.crt ``` ### Embedding certificates in kubeconfig files Repeat this step for all nodes that have CSR files. Write the following script in the `/etc/kubernetes` directory, navigate to the directory and execute the script. The script will take the `.crt` files that were signed for kubeconfig files from CSRs in the previous step and will embed them in the kubeconfig files. ```bash #!/bin/bash CLUSTER=kubernetes find ./ -name "*.conf" | while read -r FILE; do echo "* Processing ${FILE} ..." KUBECONFIG="${FILE}" kubectl config set-cluster "${CLUSTER}" --certificate-authority ./pki/ca.crt --embed-certs USER=$(KUBECONFIG="${FILE}" kubectl config view -o jsonpath='{.users[0].name}') KUBECONFIG="${FILE}" kubectl config set-credentials "${USER}" --client-certificate "${FILE}.crt" --embed-certs done ``` ### Performing cleanup {#post-csr-cleanup} Perform this step on all nodes that have CSR files. Write the following script in the `/etc/kubernetes` directory, navigate to the directory and execute the script. ```bash #!/bin/bash # Cleanup CSR files rm -f ./*.csr ./pki/*.csr ./pki/etcd/*.csr # Clean all CSR files # Cleanup CRT files that were already embedded in kubeconfig files rm -f ./*.crt ``` Optionally, move `.srl` files to the next node to be processed. Optionally, if using external CA remove the `/etc/kubernetes/pki/ca.key` file, as explained in the [External CA node](#external-ca-mode) section. ### kubeadm node initialization Once CSR files have been signed and required certificates are in place on the hosts you want to use as nodes, you can use the commands `kubeadm init` and `kubeadm join` to create a Kubernetes cluster from these nodes. During `init` and `join`, kubeadm uses existing certificates, encryption keys and kubeconfig files that it finds in the `/etc/kubernetes` tree on the host's local filesystem.