[BUG] Operator certificate generation / renewals not working

[albgus]

I have recently updated the OpenSearch operator to version 2.6.0. This seems to have actually triggered some sort of certificate genration process, as seen in the log entries. However, it seems that only the admin certificate was updated, the http and transport certificate is still the same old version.

The operator has been logging this for hours with no apparent progress.

{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchactiongroup","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchActionGroup","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchcomponenttemplate","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchComponentTemplate","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchindextemplate","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchIndexTemplate","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.132Z","msg":"Starting workers","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.224Z","msg":"Starting workers","controller":"opensearchrole","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchRole","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.225Z","msg":"Starting workers","controller":"opensearchtenant","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchTenant","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.228Z","msg":"Starting workers","controller":"opensearchuserrolebinding","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchUserRoleBinding","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.228Z","msg":"Starting workers","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.228Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:43:40.229Z","msg":"Starting workers","controller":"opensearchuser","controllerGroup":"opensearch.opster.io","controllerKind":"OpensearchUser","worker count":1}
{"level":"info","ts":"2024-05-16T12:43:40.244Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:43:40.244Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997","interface":"http"}
{"level":"info","ts":"2024-05-16T12:43:40.848Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"d511df94-9763-429c-b28f-f7df986f5997"}
{"level":"info","ts":"2024-05-16T12:43:40.946Z","logger":"KubeAPIWarningLogger","msg":"would violate PodSecurity \"restricted:latest\": allowPrivilegeEscalation != false (container \"dashboards\" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container \"dashboards\" must set securityContext.capabilities.drop=[\"ALL\"]), runAsNonRoot != true (pod or container \"dashboards\" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container \"dashboards\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")"}
{"level":"info","ts":"2024-05-16T12:44:10.985Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:44:11.023Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:44:11.023Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21","interface":"http"}
{"level":"info","ts":"2024-05-16T12:44:11.173Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ce3f1596-3baf-4887-b0e8-a628fb891a21"}
{"level":"info","ts":"2024-05-16T12:44:41.274Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:44:41.293Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:44:41.294Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095","interface":"http"}
{"level":"info","ts":"2024-05-16T12:44:41.536Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"2d3f6e99-51e9-4975-aa28-d1b6eba80095"}
{"level":"info","ts":"2024-05-16T12:45:11.647Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:45:11.666Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:45:11.666Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2","interface":"http"}
{"level":"info","ts":"2024-05-16T12:45:11.924Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"ee9e9109-d68a-4882-a7b7-de318b2bffa2"}
{"level":"info","ts":"2024-05-16T12:45:42.029Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:45:42.055Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:45:42.055Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3","interface":"http"}
{"level":"info","ts":"2024-05-16T12:45:42.239Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"49f68700-b1a1-420c-8d0b-961c38e623e3"}
{"level":"info","ts":"2024-05-16T12:46:12.334Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:46:12.352Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:46:12.352Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7","interface":"http"}
{"level":"info","ts":"2024-05-16T12:46:12.488Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"3ddf126d-81dc-456b-b1b6-9e4842f72ba7"}
{"level":"info","ts":"2024-05-16T12:46:42.639Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:46:42.735Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:46:42.735Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690","interface":"http"}
{"level":"info","ts":"2024-05-16T12:46:42.923Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"db98e17b-7501-468d-b79c-743b7aa66690"}
{"level":"info","ts":"2024-05-16T12:47:13.027Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:47:13.044Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:47:13.045Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1","interface":"http"}
{"level":"info","ts":"2024-05-16T12:47:13.274Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"017ad2d7-2d7a-4251-8c73-bee0c2609db1"}
{"level":"info","ts":"2024-05-16T12:47:43.371Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"09e712b6-8591-4bff-bd0d-bb14a8a76c50","cluster":{"name":"opensearch","namespace":"opensearch-deployment"}}
{"level":"info","ts":"2024-05-16T12:47:43.389Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"09e712b6-8591-4bff-bd0d-bb14a8a76c50","interface":"transport"}
{"level":"info","ts":"2024-05-16T12:47:43.390Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch","namespace":"opensearch-deployment"},"namespace":"opensearch-deployment","name":"opensearch","reconcileID":"09e712b6-8591-4bff-bd0d-bb14a8a76c50","interface":"http"}

[pasztorl]

+1

[pasztorl]

I've checked with a new cluster install. 2.11.0 works as expected, 2.12.0 the same issue above.

[pasztorl]

update: if I retry multiple times sometimes works sometimes not, race condition?

[Jerrimikkihvatai]

+1 Catch the same error while testing cert renewal after trying this method. Tested on operator versions 2.5.0 and 2.6.0, opensearch 2.13.
The operator didn't recreate certs so I restarted pod and got the error. But certs were regenerated.
The error disappeared only after cluster redeploy.

[prudhvigodithi]

[Triage]
Thanks everyone, I assume this method posted here by @swoehrl-mw worked?
@albgus @Jerrimikkihvatai @pasztorl
@getsaurabh02

[AdaptiveStep]

Assume certmanager just refreshed its certs somehow.

The only thing that works for me when trying to update certs on opensearch is by:
1: first deleting the relevant secrets
2: deleting the entire opensearch crd instance object
3: re create the opensearch crd instance object

It then seems to use the new certs.

It seems like all the stuff in the statefulset is still there. So its a way to do this manually for me. But re-freshing certs with certmanager has been an issue.