[FEATURE] Containers mounting Secrets(internal_users.yml, etc) using subPath for securityconfig does not receive automated Secret updates

[YeonghyeonKO]

Is your feature request related to a problem?

When developers edit securityconfig-secret (K8s Secret of each OpenSearchCluster CR) for changing internal_users.yml or roles.yml, securityconfig-update (K8s Job) starts and soon finishes its Pod.

Since the Pod newly created by securityconfig reconciler of operator always have the recently edited data from securityconfig-secret, there's no problem to run the below arguments which are defined in the securityconfig-update Job:

ADMIN=/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh;
until $ADMIN -cacert /certs/ca.crt -cert /certs/tls.crt -key /certs/tls.key -f /usr/share/opensearch/config/opensearch-security/internal_users.yml -t internalusers -icl -nhnv -h dtp-test-opensearch-cluster.dtp-test-opensearch-cluster.svc.cluster.local -p 9200 || (( count++ >= 20 ));
do

After the command executed successfully,

However, if you try to look manifests from other Pods (i.e. OpenSearch Nodes such as data-1, master-0, etc), they don't have the exact data you've already made when editing securityconfig-secret. It's because Kubernetes has the limit of auto-updating by "eventually-consistent" way.

Note:
A container using a Secret as a subPath volume mount does not receive automated Secret updates.

volumeMounts:
  ...
  - mountPath: /usr/share/opensearch/config/opensearch-security/internal_users.yml
    name: securityconfig
    readOnly: true
    subPath: internal_users.yml

What alternatives have you considered?

If OpenSearch Cluster itself has a goal of updating data of securityconfig like internal_users.yml synchronously/eventually, one of the simplest way is restarting entire Pods (i.e. Nodes). Of course it can be dangerous and not discouraged just only for having newly updated securityconfig. Another way is using ConfigMap instead of Security but it may lead security risks. So the only alternative I've considered is separating the securityconfig-secret manifest into multiple Secrets, one for each subPath.

Do you have any additional context?

I think the maintainers of the opensearch-k8s-operator may have already discussed or gotten stuck in this point. Is there any room for consideration of my idea or have it been already considered? (Or the above issue may be resolved in the next major version : opensearch-project/security#1755 )

[swoehrl-mw]

Hi @YeonghyeonKO. I don't see this happening. The securityconfig files are only really used in the securityconfig-update-job, that they are also mounted into the opensearch pods is AFAIK more incidental and does not have an actual use.
And especially if the handling of the securityconfig is changed in the forseeable future, there is no need for this kind of change.