[BUG] TLS certificates cannot be renewed without downtime

[malayh]

What is the bug?

The TLS certificates are mounted using subPath, so even if the secrets update the pods don't receive the renewed certificate without restarting all the pods and manually triggering a security update that puts the new certificate across all cluster nodes. This means you can't renew the TLS certs without downtime.

volumeMounts:
        - mountPath: /usr/share/opensearch/config/tls-transport/ca.crt
          name: transport-ca
          subPath: ca.crt
        - mountPath: /usr/share/opensearch/config/tls-transport/tls.key
          name: transport-key
          subPath: tls.key
        - mountPath: /usr/share/opensearch/config/tls-transport/tls.crt
          name: transport-cert
          subPath: tls.crt
        - mountPath: /usr/share/opensearch/config/tls-http/ca.crt
          name: http-ca
          subPath: ca.crt
        - mountPath: /usr/share/opensearch/config/tls-http/tls.key
          name: http-key
          subPath: tls.key
        - mountPath: /usr/share/opensearch/config/tls-http/tls.crt
          name: http-cert
          subPath: tls.crt

How can one reproduce the bug?

Set up a cluster with a certificate that expires soon. Then renew the certificate and the pod won't receive the change.

What is the expected behavior?

When the secrets containing the TLS certs update, the pods should automatically receive the new certs and propagate the change across all the nodes without downtime.

Aditional context

I am using cert-manager to create all the certificates, which automatically renews. But the pods cluster doesn't update with the new certificates.

[prudhvigodithi]

[Triage]
AFAIK the volumeMounts by default should put the updated files inside the pod, but OpenSearch might require restart to pick up the new certs, adding @cwperks here for more details. But @malayh how about you try rolling upgrade which should not cause any downtime.
Thanks

[malayh]

@prudhvigodithi How can I manually trigger a rolling update?