An elegant way to upgrade/test for localhost vs. 0.0.0.0 so that migrating to 2.11 doesn't freeze all comms

[juandiegopalomino]

https://linkerd.slack.com/archives/C89RTCWJF/p1647020660446349

[juandiegopalomino]

For instance, hello world django will fail unless a user remembers to specify 0.0.0.0 : http://www.holeintheceiling.com/blog/2012/06/21/django-and-runserver-0-0-0-0/ (verified in other sources as well)

Tbh, this kinda goes against Linkerd's "it just works", doesn't it?

[juandiegopalomino]

Asking friends, this is also the default in java springboot

[mateiidavid]

Hey @juandiegopalomino, we talked a bit about it on Slack :)

Checking each individual app might turn out to be a bit cumbersome. I think perhaps the easiest way to catch it would be through liveness/readiness probes. If you want to check this behaviour without linkerd (so an app is unmeshed), you can use both tcp and http probes. If the app is meshed, you can use http probes -- although, you probably want to check the app itself, not that linkerd won't forward.

Here is an example nginx configuration that listens on localhost, taken from one of our tests:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
 nginx.conf: |-
    events {}
    http {
        server {
          listen 127.0.0.1:8080;
            location / {
                return 200;
            }
        }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 3
            periodSeconds: 3
            successThreshold: 1
            timeoutSeconds: 1
          volumeMounts:
            - name: nginx-config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-config
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - name: service
    port: 8080
  selector:
    app: nginx

In this case, the probe will be successful if the index path (/) of our nginx container returns a status code that's successful, i.e 200 <= status < 400. Since our nginx container listens on localhost, we get a failure:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  34s                default-scheduler  Successfully assigned default/nginx-57d8859757-pzdf8 to k3d-workshop-server-0
  Normal   Pulled     13s (x3 over 34s)  kubelet            Container image "nginx:alpine" already present on machine
  Normal   Created    13s (x3 over 34s)  kubelet            Created container nginx
  Normal   Started    13s (x3 over 33s)  kubelet            Started container nginx
  Warning  Unhealthy  4s (x9 over 28s)   kubelet            Liveness probe failed: Get "http://10.42.0.32:8080/": dial tcp 10.42.0.32:8080: connect: connection refused
  Normal   Killing    4s (x3 over 22s)   kubelet            Container nginx failed liveness probe, will be restarted
  Warning  BackOff    3s (x2 over 4s)    kubelet            Back-off restarting failed container

I believe this might do the trick, it's something I whipped up in a few minutes though so my assumptions might be wrong :)

For instance, hello world django will fail unless a user remembers to specify 0.0.0.0 : http://www.holeintheceiling.com/blog/2012/06/21/django-and-runserver-0-0-0-0/ (verified in other sources as well)

Tbh, this kinda goes against Linkerd's "it just works", doesn't it?

We understand such a breaking change can be a bit frustrating for users. Prior to our 2.11.0 release, we tried to document this as best as we can. It's important to note though, that linkerd does just work, and it's doing what it is supposed to do: forward connections from outside of the pod to your container.

If a container is bound on localhost (127.0.0.1) it should not be able to process requests from outside of the pod. Take your work machine for example. If you start a dev server, and it is bound on 127.0.0.1, without taking into account any firewalls and such, no device or user from outside of your machine's loopback interface would be able to connect to it. The loopback interface is meant for the machine only. The same applies to your pod; it has its own loopback interface, its own virtual ethernet interface, and so on. If you want your machine to accept internet connections, it needs to bind to whatever IP it is using (or 0.0.0.0 for any IP).

Looking at your Django link, the same thing is specified there:

only makes the system available via localhost which is of course useless for other machines on the network. To solve this I was looking up the ip for my machine and passing that to runsever:

In other words, if you don't bind on 0.0.0.0 or the pod's IP, you shouldn't be able to be accessed by other pods. Linkerd previously leaked this and it was revised to harden security and pave the way for authorization policies. Just because it worked before when it shouldn't have, does not mean we adhere to a "just works" philosophy. We believe it should be easy to configure and use, but it should still not work when configurations are wrong.

Hope this all clears it up. Again, sorry it's a bit of a tough process to do the upgrade. On the bright side, your container security will improve.