cert-manager quickstart

October 23, 2022

This article shows how to install and configure cert-manager to obtain locally issued X.509 certificates by using the CA issuer of cert-manager.

1. About cert-manager

cert-manager is an X.509 certificate manager. It runs as a Kubernetes controller that accepts certificate requests in the form of Kubernetes custom resources, and produces Kubernetes Secrets holding the private key, the signed X.509 certificate, and the CA certificates that signed the certificate.

You request a certificate from cert-manager by creating a Kubernetes custom resource that contains details such as the subject names, key usages, key algorithm, key size, certificate validity duration, a reference to an issuer, truststore format, output secret’s name, etc. cert-manager reads this resource, generates a key pair, creates a certificate signing request (CSR), submits the CSR to the issuer, reads the certificate issued by the issuer and creates a Kubernetes Secret holding the private key, the issued certificate and the CA certificates that signed the certificate. Applications mount this Kubernetes Secret as a volume and use the key and certificates for purposes such as TLS.

When cert-manager receives a request for a certificate, it creates a certificate signing request (CSR) and submits it to an issuer. The issuer communicates with the CA to obtain a signed X.509 certificate. cert-manager has built-in issuers for CA services such as HashiCorp Vault, AWS Private CA, Venafi, ACME CA services (e.g. Let’s Encrypt), etc. There also exist external issuers, contributed by the community, for CA services such as Google Cloud CA Service, Cloudflare Origin CA, etc. You can even write your own external issuer for integration with your custom CA service. A single installation of cert-manager can support multiple issuers simultaneously.

cert-manager was created by Jetstack. Jetstack is now a part of Venafi. cert-manager was donated to CNCF in the year 2020. It is currently at Incubating maturity level in CNCF. cert-manager is an open source project. Jetstack provides paid customer support. Jetstack also offers an enterprise version of cert-manager that offers additional features on top of what is available in the open source version.

2. Prerequisites

  1. OpenSSL CLI.

  2. A working Kubernetes cluster. See Kubernetes on Windows Subsystem for Linux (WSL) 2 using kind and Docker Engine.

  3. Helm CLI. See Installing Helm.

  4. jq

I was using the following at the time of writing this article:

  1. OpenSSL 1.1.1f

  2. kind version 0.16.0

  3. Kubernetes version 1.25.2

  4. Docker Engine - Community version 20.10.18.

  5. WSL distro Ubuntu 20.04.5 LTS.

  6. Helm 3.10.0

  7. jq 1.6

3. Install cert-manager

helm repo add jetstack https://charts.jetstack.io

helm repo update

helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --version v1.10.0 \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true \(1)
  --set featureGates=LiteralCertificateSubject=true \(2)
  --set "webhook.extraArgs={--feature-gates=LiteralCertificateSubject=true}" \(3)
  --set "extraArgs={--enable-certificate-owner-ref=true}"(4)
1 CRD resources will be installed as part of the Helm chart. When cert-manager is uninstalled, CRD resources will be deleted causing all installed custom resources to be deleted.
2 Enable LiteralCertificateSubject feature gate on cert-manager controller. Read Literal Certificate Subjects.
3 Enable LiteralCertificateSubject feature gate on cert-manager webhook. Read Literal Certificate Subjects.
4 Enable deletion of the Kubernetes Secret generated by cert-manager when the corresponding Certificate resource is deleted. Read Cleaning up Secrets when Certificates are deleted.

4. Generate keys and certificates for the issuing CA

We will be using the following certificate chain:

certificate-chain

  • End entity certificates will be signed by the issuing CA.

  • The certificate of the issuing CA will be signed by an intermediate CA.

  • The certificate of the intermediate CA will be signed by a self-signed root CA.

end entity: user of PKI certificates and/or end user system that is the subject of a certificate

. . .

End entity certificates are issued to subjects that are not authorized to issue certificates.

— www.rfc-editor.org/rfc/rfc5280#section-3

4.1. Generate key and certificate for the root CA

Generate a key pair.

openssl genpkey \
    -algorithm RSA \
    -pkeyopt rsa_keygen_bits:4096 \
    -outform PEM \
    -out root-ca-key.pem

Generate a self-signed certificate.

openssl req \
    -new \
    -x509 \
    -key root-ca-key.pem \
    -keyform PEM \
    -sha256 \
    -subj "/CN=root-ca/OU=Foo/O=Bar" \
    -days 3650 \
    -addext "keyUsage = keyCertSign,cRLSign" \
    -utf8 \
    -batch \
    -outform PEM \
    -out root-ca-crt.pem

4.2. Generate key and certificate for the intermediate CA

Generate a key pair.

openssl genpkey \
    -algorithm RSA \
    -pkeyopt rsa_keygen_bits:4096 \
    -outform PEM \
    -out intermediate-ca-key.pem

Generate a certificate signing request (CSR).

openssl req \
    -new \
    -key intermediate-ca-key.pem \
    -keyform PEM \
    -subj "/CN=intermediate-ca/OU=Foo/O=Bar" \
    -sha256 \
    -utf8 \
    -batch \
    -outform PEM \
    -out intermediate-ca.csr

Get the certificate signed by the root CA.

cat <<EOF > intermediate-ca.cnf
[ req ]
distinguished_name     = req_distinguished_name
req_extensions = v3_ca

[ req_distinguished_name ]

[ v3_ca ]
subjectKeyIdentifier=hash
basicConstraints = critical,CA:TRUE
keyUsage = critical,keyCertSign,cRLSign
authorityKeyIdentifier = keyid
EOF

openssl x509 \
    -req \
    -in intermediate-ca.csr \
    -inform PEM \
    -extfile intermediate-ca.cnf \
    -extensions v3_ca \
    -CA root-ca-crt.pem \
    -CAform PEM \
    -CAkey root-ca-key.pem \
    -CAkeyform PEM \
    -CAcreateserial \
    -sha256 \
    -days 1825 \
    -outform PEM \
    -out intermediate-ca-crt.pem

Cleanup.

rm \
    root-ca-key.pem \
    root-ca-crt.srl \
    intermediate-ca.cnf \
    intermediate-ca.csr

4.3. Generate key and certificate for the issuing CA

Generate a key pair.

openssl genpkey \
    -algorithm RSA \
    -pkeyopt rsa_keygen_bits:4096 \
    -outform PEM \
    -out issuer-ca-key.pem

Generate a certificate signing request (CSR).

openssl req \
    -new \
    -key issuer-ca-key.pem \
    -keyform PEM \
    -subj "/CN=issuer-ca/OU=Foo/O=Bar" \
    -sha256 \
    -utf8 \
    -batch \
    -out issuer-ca.csr \
    -outform PEM

Get the certificate signed by the intermediate CA.

cat <<EOF > issuer-ca.cnf
[ req ]
distinguished_name     = req_distinguished_name
req_extensions = v3_ca

[ req_distinguished_name ]

[ v3_ca ]
subjectKeyIdentifier=hash
basicConstraints = critical,CA:TRUE
keyUsage = critical,keyCertSign,cRLSign
authorityKeyIdentifier = keyid
EOF

openssl x509 \
    -req \
    -in issuer-ca.csr \
    -inform PEM \
    -extfile issuer-ca.cnf \
    -extensions v3_ca \
    -CA intermediate-ca-crt.pem \
    -CAform PEM \
    -CAkey intermediate-ca-key.pem \
    -CAkeyform PEM \
    -CAcreateserial \
    -sha256 \
    -days 730 \
    -outform PEM \
    -out issuer-ca-crt.pem

Cleanup.

rm \
    intermediate-ca-key.pem \
    intermediate-ca-crt.srl \
    issuer-ca.cnf \
    issuer-ca.csr

5. Deploy the issuing CA

We will use the CA issuer of cert-manager as the issuing CA.

The CA issuer represents a Certificate Authority whose certificate and private key are stored inside the cluster as a Kubernetes Secret.

— cert-manager.io/docs/configuration/ca/

CA issuers are generally either for trying cert-manager out or else for advanced users with a good idea of how to run a PKI. To be used safely in production, CA issuers introduce complex planning requirements around rotation, trust store distribution and disaster recovery.

— cert-manager.io/docs/configuration/ca/

The CA issuer of cert-manager signs certificates locally using its private key.

Create a Kubernetes namespace for the issuer.

kubectl create namespace ns1

Create a Kubernetes secret holding the private key of the issuing CA and the certificates of the issuing CA, intermediate CA and root CA.

cat issuer-ca-crt.pem > tls.crt
cat intermediate-ca-crt.pem >> tls.crt
cat root-ca-crt.pem >> tls.crt

kubectl create secret generic \
    ca-key-pair \
    --namespace=ns1 \
    --from-file=tls.crt=./tls.crt \
    --from-file=tls.key=./issuer-ca-key.pem

Create a CA issuer of cert-manager to be used as the issuing CA.

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ca-issuer
  namespace: ns1
spec:
  ca:
    secretName: ca-key-pair
EOF

Cleanup.

rm \
    issuer-ca-key.pem \
    issuer-ca-crt.pem \
    intermediate-ca-crt.pem \
    root-ca-crt.pem \
    tls.crt

6. Request a certificate

6.1. Submit a certificate request

Request cert-manager to issue an end entity certificate.

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-service-cert-request
  namespace: ns1
spec:
  isCA: false
  usages:
    - server auth
    - client auth
  privateKey:
    algorithm: RSA
    encoding: PKCS8
    size: 4096
  literalSubject: "CN=ExampleService,O=ExampleOrg,OU=ExampleOrgUnit,STREET=1st street,L=Madhapur,ST=Hyderabad,C=IN,SERIALNUMBER=1234567890"
  dnsNames:
    - ExampleService.ns1.svc.cluster.local
    - ExampleService.ns1
  emailAddresses:
    - ExampleService@ExampleOrg.com
  duration: 240h # 10d
  renewBefore: 120h # 5d
  issuerRef:
    kind: Issuer
    group: cert-manager.io
    name: ca-issuer
  secretName: example-service-key-cert
EOF

6.2. Verify the status of certificate request

Command
kubectl -n ns1 \
    get certificate example-service-cert-request \
    -o jsonpath="{.status}" | jq .
Output
{
  "conditions": [
    {
      "lastTransitionTime": "2022-10-23T07:25:01Z",
      "message": "Certificate is up to date and has not expired",
      "observedGeneration": 1,
      "reason": "Ready",
      "status": "True",(1)
      "type": "Ready"(1)
    }
  ],
  "notAfter": "2022-11-02T07:25:01Z",
  "notBefore": "2022-10-23T07:25:01Z",
  "renewalTime": "2022-10-28T07:25:01Z",
  "revision": 1
}
1 The certificate has been issued and is ready to be used.

7. Inspect the issued certificate

Inspect the Kubernetes Secret generated by cert-manager.

Command
kubectl get secret \
    example-service-key-cert \
    -n ns1 \
    -o yaml
Output
apiVersion: v1
kind: Secret
type: kubernetes.io/tls
metadata:
  name: example-service-key-cert
  namespace: ns1
  annotations:
    cert-manager.io/alt-names: ExampleService.ns1.svc.cluster.local,ExampleService.ns1
    cert-manager.io/certificate-name: example-service-cert-request
    cert-manager.io/common-name: ExampleService
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: cert-manager.io
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: ca-issuer
    cert-manager.io/uri-sans: ""
  ownerReferences: (1)
  - apiVersion: cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Certificate
    name: example-service-cert-request
    uid: 8652a900-65b1-4f95-b30b-8d85f8424d67
  resourceVersion: "58088"
  uid: 8c25906d-f8f6-4150-9d57-fafc50a117e1
  creationTimestamp: "2022-10-23T09:41:57Z"
data:
  ca.crt: [REDACTED] (2)
  tls.crt: [REDACTED] (3)
  tls.key: [REDACTED] (4)
1 Because we passed the argument --enable-certificate-owner-ref=true to cert-manager, the Certificate resource owns this Secret. Therefore, Kubernetes will delete this Secret when the Certificate resource is deleted.
2 ca.crt holds the certificate of the root CA.
3 tls.crt holds the certificates of the end entity, the issuing CA and the intermediate CA.
4 tls.key holds the private key of the end entity.

Inspect tls.crt in the generated secret.

Command
kubectl -n ns1 \
    get secret example-service-key-cert \
    -o "jsonpath={.data['tls\.crt']}" \
    | base64 -d \
    | openssl crl2pkcs7 -nocrl -certfile /dev/stdin \
    | openssl pkcs7 -print_certs -text -noout \
    | egrep  "(Certificate:|Subject:|Issuer:)"
Output
Certificate: (1)
        Issuer: CN=issuer-ca, OU=Foo, O=Bar
        Subject: serialNumber=1234567890, C=IN, ST=Hyderabad, L=Madhapur/street=1st street, OU=ExampleOrgUnit, O=ExampleOrg, CN=ExampleService
Certificate: (2)
        Issuer: CN=intermediate-ca, OU=Foo, O=Bar
        Subject: CN=issuer-ca, OU=Foo, O=Bar
Certificate: (3)
        Issuer: CN=root-ca, OU=Foo, O=Bar
        Subject: CN=intermediate-ca, OU=Foo, O=Bar
1 End entity certificate signed by the issuing CA.
2 Issuing CA certificate signed by the intermediate CA.
3 Intermediate CA certificate signed by the root CA.

Inspect ca.crt in the generated secret.

Command
kubectl -n ns1 \
    get secret example-service-key-cert \
    -o "jsonpath={.data['ca\.crt']}" \
    | base64 -d \
    | openssl crl2pkcs7 -nocrl -certfile /dev/stdin \
    | openssl pkcs7 -print_certs -text -noout \
    | egrep  "(Certificate:|Subject:|Issuer:)"
Output
Certificate: (1)
        Issuer: CN=root-ca, OU=Foo, O=Bar
        Subject: CN=root-ca, OU=Foo, O=Bar
1 Self-signed certificate of the root CA.

Inspect tls.key in the generated secret.

Command
kubectl -n ns1 \
    get secret example-service-key-cert \
    -o "jsonpath={.data['tls\.key']}" \
    | base64 -d \
    | openssl pkey -text -noout
Output
RSA Private-Key: (4096 bit, 2 primes)
modulus:
    [REDACTED]
publicExponent: [REDACTED]
privateExponent:
    [REDACTED]
prime1:
    [REDACTED]
prime2:
    [REDACTED]
exponent1:
    [REDACTED]
exponent2:
    [REDACTED]
coefficient:
    [REDACTED]

8. Cleanup

8.1. Delete the certificate request

kubectl -n ns1 \
    delete certificate example-service-cert-request

Notice that deletion of the Certificate resource also causes the deletion of the corresponding Kubernetes Secret that was generated by cert-manager. This is because we passed the argument --enable-certificate-owner-ref=true to cert-manager when cert-manager was installed.

$ kubectl -n ns1 \
    get secret example-service-key-cert

Error from server (NotFound): secrets "example-service-key-cert" not found (1)
1 The Secret was not found because it was already deleted by Kubernetes in response to the deletion of the Certificate resource.

8.2. Uninstall the issuing CA

kubectl -n ns1 delete issuer ca-issuer

kubectl -n ns1 delete secret ca-key-pair

kubectl delete namespace ns1

8.3. Uninstall cert-manager

helm uninstall cert-manager -n cert-manager

9. References

9.1. cert-manager

  1. cert-manager docs > Installing with Helm

  2. cert-manager docs > Changes since v1.9.0 > Literal Certificate Subjects

  3. cert-manager docs > Cleaning up Secrets when Certificates are deleted

  4. cert-manager docs > CA issuer

  5. cert-manager docs > Usage > Certificate Resources

  6. cert-manager docs > API reference > Certificate

  7. GitHub > cert-manager/cert-manager > Error getting keypair for CA issuer: certificate is not a CA #279

9.2. OpenSSL

  1. golinuxcloud.com > OpenSSL create certificate chain with Root & Intermediate CA

  2. linux.die.net > openssl > genpkey

  3. linux.die.net > openssl > req

  4. linux.die.net > openssl > x509

  5. linux.die.net > openssl > crl2pkcs7

  6. linux.die.net > openssl > pkcs7

  7. GitHub > openssl/openssl > "error, no objects specified in config file" when creating CSR with ECDSA key & config file #3536

  8. Stack Exchange > Missing X509 extensions with an openssl-generated certificate

  9. Server Fault > How to view all ssl certificates in a bundle?

Appendix A: Acronyms

Acronym Meaning

CA

Certificate authority

CRD

Kubernetes CustomResourceDefinitions

CSR

Certificate signing request

PEM

Privacy-Enhanced Mail

PKCS

Public Key Cryptography Standards

RSA

Rivest–Shamir–Adleman

TLS

Transport Layer Security

WSL

Windows Subsystem for Linux