IPV6 Only Access

[safaci2000]

I have looked over https://github.com/orgs/strimzi/discussions/7148 and tried to implement the suggested solution. I wanted to spawn a new thread to avoid hijacking the conversation.

My Operator is installed with the following configuration:

USER-SUPPLIED VALUES:
extraEnvs:
- name: KUBERNETES_DISABLE_HOSTNAME_VERIFICATION
  value: "true"
- name: KUBERNETES_MASTER
  value: https://34.46.76.68

I don't have a master node/ control plane V6 address, so I'm stuck with the V4 address shown above.

when I connect the cluster from a V6 only network, I get the following errors:

%3|1719444266.764|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: 34.136.35.18:9094: Failed to connect to broker at 18.35.136.34.bc.googleusercontent.com:9094: Network is unreachable (after 3ms in state CONNECT)
% Error: Local: Broker transport failure: GroupCoordinator: 34.136.35.18:9094: Failed to connect to broker at 18.35.136.34.bc.googleusercontent.com:9094: Network is unreachable (after 3ms in state CONNECT). If I enable V4 addresses everything works fine.

This is my current configuration for my cluster. I've also tried setting V6 for the plain and tls listeners but that causes issues.

Listener configuration is not valid: [listener tls cannot configure ipFamilies because it is internal listener, listener tls cannot configure ipFamilyPolicy because it is internal listener, listener plain cannot configure ipFamilies because it is internal listener, listener plain cannot configure ipFamilyPolicy because it is internal listener]

Any help would be appreciated.

I've attached my cluster configuration as well if that's helpful.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: testing
  labels:
    app: kafka
spec:
  kafka:
    version: 3.7.0
    replicas: 6
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: tls
        configuration:
          ipFamilyPolicy: PreferDualStack
          ipFamilies:
              - IPv4
              - IPv6
    authorization:
      type: simple
      superUsers:
        - CN=kafka-stardust-admin
    readinessProbe:
      initialDelaySeconds: 30
      timeoutSeconds: 10
    livenessProbe:
      initialDelaySeconds: 30
      timeoutSeconds: 10
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.7"
    template:
      bootstrapService:
        ipFamilyPolicy: PreferDualStack
        ipFamilies:
          - IPv4
          - IPv6
      brokersService:
        ipFamilyPolicy: PreferDualStack
        ipFamilies:
          - IPv4
          - IPv6

      pod:
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                app: kafka
            matchLabelKeys:
              - strimzi.io/component-type
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - testing-kafka
                topologyKey: "kubernetes.io/hostname"
    resources:
      limits: 
        cpu: "4"
        memory: 34Gi
      requests: 
        cpu: "2"
        memory: 32Gi
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 2048Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
    readinessProbe:
      initialDelaySeconds: 30
      timeoutSeconds: 10
    livenessProbe:
      initialDelaySeconds: 30
      timeoutSeconds: 10
    template:
      clientService:
          ipFamilyPolicy: PreferDualStack
          ipFamilies:
              - IPv4
              - IPv6
      nodesService:
          ipFamilyPolicy: PreferDualStack
          ipFamilies:
              - IPv4
              - IPv6
      pod:
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                app: kafka
            matchLabelKeys:
              - strimzi.io/component-type
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - testing-zookeeper
                topologyKey: "kubernetes.io/hostname"
    resources:
      limits: 
        cpu: "2"
        memory: 3Gi
      requests: 
        cpu: "1"
        memory: 2Gi

[scholzj]

I don' think your problem has nothing to do with #7148 => that one is about the operator talking over IPv6 to the Kubernetes API. It has nothing to do with Kafka (plus it is old so it might not be needed anymore anyway). So you should not expect that to solve your issue.

It is hard to know for sure without understanding your environment. But it seems to me like your brokers advertise the IPv4 addresses wich your network cannot route. So to fix that, you might need to:

• Have a listener that uses IPv6 as its primary network and not IPv4
• Override the advertised listeners to advertise IPv6 addresses (but than you will likely have the same issue from IPv4-only network)
• If you want to use a single listener for both IPv4 and IPv6, you might need to:
  • Use a custom DNS names as advertised hostnames
  • Make sure the DNS names resolves to IPv4 address in the IPv4 network
  • Make sure the DNS names resolves to IPv6 address in the IPv6 network

(Again, it is a bit speculative what the problem is without detailed knowledge of your environment)

[safaci2000]

Fair enough but as far as the yaml I'm applying assuming v6 functionality is available the strimzi operator config looks correct?

[scholzj]

Yeah, it looks as I would expect it.

[safaci2000]

Thank you. One last question, if using zookeeper, there's also,

kafkas.spec.zookeeper.template where clientService and nodesService both have IPV6 configuration for ipFamilies and ipFamilyPolicy. If I was to use ZK instead of Kraft, would you suggest enabling both of those entities? Or how do they interact with the boostrap service (assuming I have a v4 only and v6 only version) as configured above?

[scholzj]

The ZooKeeper services are internal only. So your clients should not need to worry whether they use IPv6 or IPv4. They are used only by the operator and the Kafka brokers. But if you have dual stack cluster, I do not have any recommendation whether it is better to use IPv6 or IPv4 to be honest.

[safaci2000]

Thank you, well that clarifies all my questions about Strimzi. I'll see what I can figure out with GKE and report back if I find a solution.

[darwin67]

We're also on IPv6 only and we are seeing issues with the controller creating a quorum when deploying Kafka using this operator.

The error we're seeing is

2025-03-10 17:13:53,505 WARN [RaftManager id=3] Error connecting to node main-controller-5.main-kafka-brokers.kafka-main.svc.cluster.local:9090 (id: 5 rack: null) (org.apache.kafka.clients.NetworkClient) [kafka-3-raft-outbound-request-thread]
java.net.UnknownHostException: main-controller-5.main-kafka-brokers.kafka-main.svc.cluster.local

where both controllers and brokers can't connect to controllers somehow.

We were able to confirm that the DNS entry can be resolved to an IPv6 address so we know that DNS is not the issue, but instead we found that controllers are only bind to IPv4 when we ran netstat inside them.

sh-5.1$ netstat -lnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9090            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9999            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:34531           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN

Setting bootstrapService and brokersService to

ipFamilyPolicy: SingleStack
ipFamilies: [IPv6]

has no effect, which also kinda seems understandable since it's not clear if that affects the kraft controllers itself.

What's a good way to make sure all pods coming up are bind to an IPv6 address and not IPv4?

I'm a noob in Java, so might be a simple question, but wasn't obvious when I did some search.

[darwin67]

Seeing this when the container starts, where it's setting 0.0.0.0 for CONTROLPLANE-9090, and it's not clear how this can be changed to [::]:9090 instead.

##########
# Common listener configuration
##########
listener.security.protocol.map=CONTROLPLANE-9090:SSL
listeners=CONTROLPLANE-9090://0.0.0.0:9090
advertised.listeners=CONTROLPLANE-9090://main-controller-3.main-kafka-brokers.kafka-main.svc:9090
sasl.enabled.mechanisms=
ssl.endpoint.identification.algorithm=HTTPS

Seems to be generated by the operator and listeners can't be overrided based on the docs. So how do we go about this?

[darwin67]

I see that it's hardcoded here.

strimzi-kafka-operator/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/KafkaBrokerConfigurationBuilder.java

Line 233 in a5a7747

listeners.add(CONTROL_PLANE_LISTENER_NAME + "://0.0.0.0:9090");

Does this mean strimzi can't handle IPv6 only clusters?

[im-konge]

@darwin67 it would be great to keep the discussion on one place and not opening multiple threads, discussions etc.

[darwin67]

Sorry, about that. Consolidating in this thread.
https://cloud-native.slack.com/archives/CMH3Q3SNP/p1741634709885909

[darwin67]

TLDR if folks don't want to read the long thread in Slack.

There's a flag java.net.preferIPv4Stack that defaults to false in the JVM, but when you enable jmxOptions, the flag will be set to true.

strimzi-kafka-operator/docker-images/kafka-based/kafka/scripts/set_kafka_jmx_options.sh

Line 9 in a5a7747

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=$(hostname -I | awk '{print $1}') -Djava.net.preferIPv4Stack=true"

So if you don't know what JMX is like I do, and when glacing through the metrics section in the docs where the JMX keywords shows up multiple times, and end up enabling it without much thought, this connection error will happen.

You can either remove the jmxOption entirely, or add this to the jvmOptions if you actually need JMX.

jvmOptions:
  javaSystemProperties:
    - name: java.net.preferIPv4Stack # <==== set this to false to prioritize IPv6
      value: "false"
    - name: java.net.preferIPv6Addresses # <===== don't really need it but just in case
      value: "true"

Though it's unclear JMX will continue to work with IPv6 only or not.

NOTE

If you also somehow assumed jmxOptions needs to be set to enable the prometheus exporter, that's not the case. So it can be safely removed.