skip-outbound-ports causes a performance issue

[nevetS]

Bug Report

What is the issue?

In my environment, I have a php web application. That php application makes calls to an service outside of kubernetes on port 5000. With linkerd enabled, the sockets hang and the application ends up not responding to certain user actions and eventually the browser will timeout.

I tried manually applying config.linkerd.io/skip-outbound-ports: "5000" and the entire service became unresponsive. I tried applying it via linkerd inject --skip-outbound-ports=5000 and the service became somewhat responsive, but some pages don't respond at all and others are just very slow to load (a simple web page will take 25+ seconds to load). The difference between my manually applied vs. linkerd injected annotation was that linkerd put the annotation before `linkerd.io/inject: enabled" and I placed it after. The pages with performance problems are not doing any communication on port 5000. They do run mysql queries. Performance is bad enough that I can't get to the pages that actually do communicate on port 5000 to test.

Without the configuration to skip outbound ports, linkerd is a component that is not noticeable to the end user.

It's not surprising that the traffic on port 5000 isn't proxied cleanly. But I am very surprised that things break so badly when I try to skip the outbound port.

How can it be reproduced?

I'm not sure it can be easily reproduced

Logs, error output, etc

not quite sure what logging would be helpful here.

linkerd check output

linkerd check
kubernetes-api

√ can initialize the client
√ can query the Kubernetes API

kubernetes-version

√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence

√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config

√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity

√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls

√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days

linkerd-api

√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version

√ can determine the latest version
√ cli is up-to-date

control-plane-version

√ control plane is up-to-date
√ control plane and cli versions match

linkerd-prometheus

√ prometheus add-on service account exists
√ prometheus add-on config map exists
√ prometheus pod is running

linkerd-grafana

√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running

Status check results are √

Environment

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-eks-2ba888", GitCommit:"2ba888155c7f8093a1bc06e3336333fbdb27b3da", GitTreeState:"clean", BuildDate:"2020-07-17T18:48:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Possible solution

Additional context

The one thing I think is unique about this pod is that it leverages a multus CNI plugin for networking. The hosted software is licensed, so we inject an additional network interface into the pod with a fixed MAC. No traffic filters through this interface.

      annotations:
        k8s.v1.cni.cncf.io/networks: '[
            { "name": "macvlan-conf",
              "ips": [ "10.1.1.22/24" ],
              "mac": "06:a5:12:ee:aa:aa",
              "gateway": [ "10.1.1.1" ]
            }]'

[grampelberg]

@nevetS this is interesting, as skip-outbound-ports actually skips proxying entirely (bypasses iptables even). Are you skipping mysql as well?

[nevetS]

mysql is not added to the mesh. I wonder if that could be it - since I added skip-outbound-ports, maybe it overrode the default skip of 3306. Testing now...

[nevetS]

That did the trick. You are my favorite person today. Thank you.