[BUG] Unable to configure tls, Pods not created

[sfisli]

What is the bug?

i'am trying to configure tls for opensearch operator, i followed the steps mentioned on the docs but the pods are not created ! and when i see controller logs there's nothing ( if i delete security:http section it works normally)

What is the expected behavior?

Cluster up and running with tls

Do you have any screenshots?

cluster.yaml:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: adeiz-opensearch-cluster
  namespace: labadeiz
spec:
  security:
    config:
      adminCredentialsSecret: # these are the admin credentials for the Operator to use
        name: admin-credentials-secret
      securityConfigSecret:  # this is the whole security configuration for OpenSearch
       name: securityconfig-secret
    tls:
      transport:
        generate: true
      http:
        generate: true
  general:
    setVMMaxMapCount: true
    serviceName: adeiz-opensearch-cluster
    version: 2.3.0
  dashboards:
    enable: true
    tls:
      enable: true
      generate: true
    opensearchCredentialsSecret: # doesn't have to be the same as adminCredentialsSecret
      name: admin-credentials-secret
    version: 2.3.0
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: nodes
      replicas: 2
      diskSize: "5Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "cluster_manager"
        - "data"

security-config.secret:

apiVersion: v1
kind: Secret
metadata:
  name: securityconfig-secret
type: Opaque
stringData:
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "$2y$10$R7I62JFTRSTELQD2O4tLQ.1TwW/oz8cvj87ug29YYsgq6r54aas1CdhL8W"
          reserved: true
          backend_roles:
          - "admin"
          description: "Demo admin user"
        dashboarduser:
          hash: "$2y$10$R7I62JFTRSTELQD2O4tLQ.1TwW/oz8cvj87ug29s54sssYYsgq6r1CdhL8W"
          reserved: true
          description: "Demo OpenSearch Dashboards user"
      tenants.yml: |-
        _meta:
          type: "tenants"
          config_version: 2

admin-credentials-secret:

apiVersion: v1
data:
  password: QGRlaXo1OQo=
  username: XWRtaszW4=
kind: Secret
metadata:
  name: admin-credentials-secret
  namespace: labadeiz
type: Opaque

kubectl describe opensearchclusters :

Name:         adeiz-opensearch-cluster
Namespace:    labadeiz
Labels:       <none>
Annotations:  <none>
API Version:  opensearch.opster.io/v1
Kind:         OpenSearchCluster
Metadata:
  Creation Timestamp:  2024-04-19T10:35:11Z
  Finalizers:
    Opster
  Generation:        2
  Resource Version:  95604313
  UID:               75da25af-7dda-46a2-be19-5fc6e680b5e4
Spec:
  Bootstrap:
    Resources:
  Conf Mgmt:
  Dashboards:
    Enable:  true
    Opensearch Credentials Secret:
      Name:    adeiz-opensearch-admin-password
    Replicas:  1
    Resources:
      Limits:
        Cpu:     200m
        Memory:  512Mi
      Requests:
        Cpu:     200m
        Memory:  512Mi
    Service:
      Type:  ClusterIP
    Tls:
      Ca Secret:
      Enable:    true
      Generate:  true
      Secret:
    Version:  2.3.0
  General:
    Http Port:  9200
    Monitoring:
    Service Name:          adeiz-opensearch-cluster
    Set VM Max Map Count:  true
    Version:               2.3.0
  Init Helper:
    Resources:
  Node Pools:
    Component:  nodes
    Disk Size:  5Gi
    Replicas:   2
    Resources:
      Limits:
        Cpu:     500m
        Memory:  2Gi
      Requests:
        Cpu:     500m
        Memory:  2Gi
    Roles:
      cluster_manager
      data
  Security:
    Config:
      Admin Credentials Secret:
        Name:  adeiz-opensearch-admin-password
      Admin Secret:
      Security Config Secret:
        Name:  securityconfig-secret
    Tls:
      Http:
        Ca Secret:
        Generate:  true
        Secret:
      Transport:
        Ca Secret:
        Generate:  true
        Secret:
Status:
  Components Status:
  Phase:  RUNNING
Events:   <none>

[swoehrl-mw]

Hi @sfisli Can you please check the logs of the operator if it is reporting anything (make sure to check the operator-controller-manager container)? There will likely be a log line about an error or something the operator is waiting for.
I tried applying your config on a local cluster, and aside from the secret having an illegal value (illegal base64 data at input byte 9) pods are being started.

[sfisli]

@swoehrl-mw now i got new error when i tried to configure certif with cert-manager and let'sencrypt:
opensearch-node-0:
[2024-04-22T15:17:51,121][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [adeiz-opensearch-cluster-nodes-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)

Config:
cluster.yaml :

kind: OpenSearchCluster
metadata:
  name: adeiz-opensearch-cluster
  namespace: labadeiz
spec:
  security:
     tls:
       transport:
         generate: false
         perNode: false
         secret: 
           name: opensearch-secret
      #   caSecret: 
      #     name: opensearch-secret
         nodesDn: ['CN=opensearch.mydomain.mycompany.com']
       http:
          generate: false
          secret:
            name: opensearch-secret # Name of the secret that contains the provided certificate
       #   caSecret:
       #     name: opensearch-secret  # Name of the secret that contains a CA the Operator should use
     config:
       adminCredentialsSecret: # these are the admin credentials for the Operator to use
         name: admin-credentials-secret
       securityConfigSecret:  # this is the whole security configuration for OpenSearch
         name: securityconfig-secret
  general:
    setVMMaxMapCount: true
    serviceName: adeiz-opensearch-cluster
    version: 2.3.0
  dashboards:
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    enable: true
    tls:
      enable: true
      generate: true
    version: 2.3.0
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: nodes
      replicas: 2
      diskSize: "3Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "1000m"
         limits:
           # memory: "2Gi"
            #cpu: "500m"
      roles:
        - "cluster_manager"
        - "data"

opensearch-secret.yaml:

apiVersion: v1
data:
  ca.crt: LS0tLS...
  tls.key: LS0tL...
  tls.crt: LS0tOZ
kind: Secret
metadata:
  name: opensearch-secret
type: kubernetes.io/tls

[therus000]

did u fix it ?

[sfisli]

the second issue ? not yet.

[swoehrl-mw]

@sfisli From the little information you've provided its impossible to tell what is happening. You'll need to check the logs of the securityconfig-update job if it has successfully run. And if not, check the operator logs if it is reporting any problems.

Also: You will need a cluster with at least 3 replicas, anything less is not supported by the operator and can lead to non-functioning clusters.

[digitalray]

@swoehrl-mw now i got new error when i tried to configure certif with cert-manager and let'sencrypt: opensearch-node-0: [2024-04-22T15:17:51,121][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [adeiz-opensearch-cluster-nodes-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)

Config: cluster.yaml :

kind: OpenSearchCluster
metadata:
  name: adeiz-opensearch-cluster
  namespace: labadeiz
spec:
  security:
     tls:
       transport:
         generate: false
         perNode: false
         secret: 
           name: opensearch-secret
      #   caSecret: 
      #     name: opensearch-secret
         nodesDn: ['CN=opensearch.mydomain.mycompany.com']
       http:
          generate: false
          secret:
            name: opensearch-secret # Name of the secret that contains the provided certificate
       #   caSecret:
       #     name: opensearch-secret  # Name of the secret that contains a CA the Operator should use
     config:
       adminCredentialsSecret: # these are the admin credentials for the Operator to use
         name: admin-credentials-secret
       securityConfigSecret:  # this is the whole security configuration for OpenSearch
         name: securityconfig-secret
  general:
    setVMMaxMapCount: true
    serviceName: adeiz-opensearch-cluster
    version: 2.3.0
  dashboards:
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    enable: true
    tls:
      enable: true
      generate: true
    version: 2.3.0
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: nodes
      replicas: 2
      diskSize: "3Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "1000m"
         limits:
           # memory: "2Gi"
            #cpu: "500m"
      roles:
        - "cluster_manager"
        - "data"

opensearch-secret.yaml:

apiVersion: v1
data:
  ca.crt: LS0tLS...
  tls.key: LS0tL...
  tls.crt: LS0tOZ
kind: Secret
metadata:
  name: opensearch-secret
type: kubernetes.io/tls

I am not seeing this issue with 2.14.0 or 2.11.0. Try to bump the versions