mutualTLS certificates appear to expire after 24 hours

[cer]

I have a Kind-based cluster running on my MacBook.
After 24 hours, the cluster stops working.
It appears that the mutualTLS certificates are not being rotated.
Suggestions?

customer-service-primary

[ 20304.745819s]  WARN ThreadId(01) outbound:proxy{addr=10.244.0.39:3306}:forward{addr=10.244.0.39:3306}: rustls::conn: Sending fatal alert BadCertificate
[ 20304.954333s] ERROR ThreadId(02) daemon:tap: rustls::conn: TLS alert received: AlertMessagePayload {
    level: Fatal,
    description: BadCertificate,
}

default/customer-service-mysql-0:

linkerd-proxy [ 23240.876707s] ERROR ThreadId(01) inbound:server{port=4143}: rustls::conn: TLS alert received: AlertMessagePayload {                                                                                                              │
│ linkerd-proxy     level: Fatal,                                                                                                                                                                                                                   │
│ linkerd-proxy     description: BadCertificate,                                                                                                                                                                                                    │
│ linkerd-proxy }                                                                                                                                                                                                                                   │

[mateiidavid]

Hey @cer! Which certificates do you expect will be rotated? The proxy (leaf) certificates, or the issuer certificate? Only the leaf certificates get automatically renewed every 24h. Issuer (and root) certificates:

• Either get rotated manually (see doc page)
• Or get rotated through a third-party PKI/cert automation tool (e.g. cert manager) (see doc page)

[cer]

@mateiidavid Thanks for replying.

I'm expecting the proxy certs to be rotated. But as far as I can tell, after 24 hours the proxies can no longer communicate with one another or the control plane.

The Issuer and Root certs are created with step ... --not-after 8760h

[mateiidavid]

Hm. Interesting. Can we verify the validity of one the proxy certificates? You can get a leaf certificate through linkerd identity <pod-name> [-n <namespace>]. Additionally, it would be helpful to check the identity controller logs (kubectl logs -n linkerd <identity-pod> identity).

We can also pull out events (e.g. all successful CSR emit an IssuedLeafCertificate event). They are typically emitted on the ServiceAccount object. I'd do kubectl get events -n <my-namespace> and filter it since kubectl describe doesn't seem to always return events when a service account is targeted.

Hopefully, one of these commands will give us some useful information.

[cer]

The cluster has been running for 20 hours.

Current time is Tue May 30 22:15:56 UTC 2023

linkerd identity -n default customer-service-primary-f7ddbb669-8g5xw

POD customer-service-primary-f7ddbb669-8g5xw (1 of 1)

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 32 (0x20)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=identity.linkerd.cluster.local
        Validity
            Not Before: May 30 02:06:29 2023 UTC
            Not After : May 31 02:07:09 2023 UTC

kubectl logs -n linkerd linkerd-identity-55b765b456-6d6c8 just shows log messages like:

time="2023-05-30T01:53:19Z" level=info msg="issued certificate for default.default.serviceaccount.identity.linkerd.cluster.local until 2023-05-31 01:53:39 +0000 UTC: 616e0ad020694bf258e6fb9519e7a0f9c02801afa3ec9bbc0925a05de79eba21"
time="2023-05-30T02:03:48Z" level=info msg="issued certificate for default.default.serviceaccount.identity.linkerd.cluster.local until 2023-05-31 02:04:08 +0000 UTC: 77d46605f434af474211eed585694a283411baba60f86d72a110215d3a88e024"
time="2023-05-30T02:06:49Z" level=info msg="issued certificate for default.default.serviceaccount.identity.linkerd.cluster.local until 2023-05-31 02:07:09 +0000 UTC: 4b11c9b0e0b217d5956822c5029e48ede8dd273eb40dcbcba34b1fae2e73ef58"

I'll try to keep the cluster running until 24 hours have passed.

One question: when are the leaf certificates rotated

[cer]

Quick question: are pods required to have a non-default service account by any chance?

[cer]

24 hours have passed and the cluster is failing.

Current time: Wed May 31 01:49:15 UTC 2023

Add database pod's certificate has expired:

linkerd identity -n default customer-service-mysql-0

POD customer-service-mysql-0 (1 of 1)

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 20 (0x14)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=identity.linkerd.cluster.local
        Validity
            Not Before: May 30 01:44:07 2023 UTC
            Not After : May 31 01:44:47 2023 UTC

kubectl logs -n linkerd linkerd-identity-55b765b456-6d6c8 does not show any new log entries.

None of the pods have updated certificates.

[jscheel]

I am experiencing this issue as well. I followed all the steps here: https://linkerd.io/2.13/tasks/automatically-rotating-control-plane-tls-credentials/. My dev cluster fails after 24 hours with Cert Expired errors on all the linkerd proxies.