EKS worker upgrade to v1.27 breaks emissary-ingress NLB

[rootrahulagr]

What happened:

EKS worker upgrade to v1.27 breaks the emissary-ingress NLB. Basically, NLB target registration doesn't take place after worker AMI upgrade to v1.27

We have emissary-ingress component deployed on our clusters that creates an NLB and registers EKS worker nodes as targets. As soon as we upgrade the worker AMI to v1.27, node registration is not taking place in NLB target groups. It's not even initiating the registration. It start working again if we revert the worker nodes to use v1.26 AMI.

What you expected to happen:

Worker instances should get registered as targets to AWS NLB

How to reproduce it (as minimally and precisely as possible):
It should be reproducible by upgrading worker nodes to v1.27 while having emissary-ingress installed on cluster.

Anything else we need to know?:

While upgrading EKS workers to v1.27, one of the significant changes that we had to do was to use --hostname-override argument for kubelet to work. Not sure if it could have something to do with the issue in hand.

Environment:

• AWS Region: ap-southeast-2
• Instance Type(s): r5.xlarge
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.11
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.27
• AMI Version: 1.27.7
• Kernel (e.g. uname -a): 3.10.0-1160.105.1.el7.x86_74
• Release information (run cat /etc/eks/release on a node):

BASE_AMI_ID="ami-0b01946f9ea8b2834"
BUILD_TIME="Wed Jan 10 07:43:45 UTC 2024"
BUILD_KERNEL="3.10.0-1160.105.1.el7.x86_64"
ARCH="x86_64"

[rootrahulagr]

Hey @cartermckinnon , would you be able to help with this please?

[cartermckinnon]

This repository is for tracking issues with the EKS AMI, so I don't think this problem is in scope. Please open a ticket with AWS Support and they'll send it over to us as needed 👍