How to scale up workload replicas count based on inbound traffic?

[susanhuhu]

Feature Request

What problem are you trying to solve?

We have a zeroscaler service, which needs to scale up a service from 0 replica as soon as there is a reqeust coming in. One way to achieve this is to rely on prometheus metrics to check if there is request coming in, which always has some delay. We then developed a notification mode in our grpc load balancer solution, so whenever there is a request coming in, while the dst deployment has 0 replica, the load balancer will send out a notifcation. Now we are trying to use LinkerD, wondering if there is an easy way to achieve this notification functionality by writing a proxy plug in. So when LinkerD tries to find a dst deployment's pod to send request, but get 0 result back, it will send out a notification to a redis pubsub channel ( or other places like event hub).

How should the problem be solved?

Allowing customized plugin to the proxy, which will trigger notification on request coming in.

Any alternatives you've considered?

Cannot find an alternative solution with exiting LinkerD.

How would users interact with this feature?

Our zeroscaler service can consume the notification and trigger corresponding deployment to scale up immediately.

[grampelberg]

You can do this pretty trivially with tap, no reason to write a plugin for the proxy.

[susanhuhu]

great. would you share more information about tap and how this can be achieved with it please?

[susanhuhu]

I looked at documenation of tap, I cannot find a way to output its data to a channel. Seems it is designed to debug LinkerD.

[grampelberg]

You can either:

• Use the gRPC streaming API directly.
• Consume the CLI as a client library.

I would recommend going the gRPC route first, which would obviously allow you to consume the data with a channel. AFAIK there's no one who's integrated with that API, so you'll want to look through the linkerd codebase, in particular the web service and CLI.

[viniciusgarcia-hotmart]

Do you guys have some code sample or even a documentation of this gRPC streaming API?

[susanhuhu]

or please at least give a pointer to the source code of this API. :)

[grampelberg]

• Code
• Proto

[viniciusgarcia-hotmart]

Thanks @grampelberg!

[ihcsim]

I'm going to rename the title of this issue. Currently, the Linkerd proxy doesn't support a plugin model. I also don't think a proxy plugin is the right approach - which proxy will be responsible for sending out the notification if the workload replicas count is 0? Are the requests being queued somewhere while the replicas are being scaled up? What if the replicas take longer time to come up?

[susanhuhu]

the proxy on client side will be responsible for sending out the notification if the workload replicas count is 0. It doesn't matter if the requests queue or not, it doesn't matter how long it takes for the replicas to come up. All the proxy has to do is to send out notifcation so other systems like zeroscaler can consume it.

[susanhuhu]

with all that said, I will try the tap service and hopefully it can meet the need.

[susanhuhu]

@grampelberg, does tap service track failed requests? I tried 'linkerd tap -n test-host deploy/test-frontend' in our system, and I found if backend pods scale down to 0 replica, tap will not show any event, even the frontend still tries to send request (for sure they fail). Does tap service only track successful requets? I need the signal from the failed requests as that is how zeroscaler service knows it needs to scale up the backend deployment.

[grampelberg]

Yes, it tracks failed requests. I'm surprised that you're not seeing something, if you provide us with a YAML that replicates the behavior we can take a look.

…

On Thu, Aug 20, 2020 at 8:01 PM susanhuhu ***@***.***> wrote: @grampelberg <https://github.com/grampelberg>, does tap service track failed requests? I tried 'linkerd tap -n test-host deploy/test-frontend' in our system, and I found if backend pods scale down to 0 replica, tap will not show any event, even the frontend still tries to send request (for sure they fail). Does tap service only track successful requets? I need the signal from the failed requests as that is how zeroscaler service knows it needs to scale up the backend deployment. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4830 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALW6CIKHJLS5U2RWPAOR3SBXIOBANCNFSM4PVXJYHQ> .

[susanhuhu]

The frontend is a simple deploy for NGINX Ingress Controller, so not much intresting yaml content. Here is what I see during the testing, pls let me know if it is expected.

1. before backend was scaled down to 0 replica, I see the OUTBOUND tap event like below
   "source": {
   "ip": "xxxxx",
   "port": xxx,
   "metadata": {
   "control_plane_ns": "linkerd",
   "deployment": "test-host-nginx-ingress-controller",
   "namespace": "test-host",
   "pod": "test-host-nginx-ingress-controller-778c77b4ff-f57vx",
   "pod_template_hash": "778c77b4ff",
   "serviceaccount": "test-host-nginx-ingress",
   "tls": "loopback"
   }
   },
   "destination": {
   "ip": "10.240.5.xxx",
   "port": 80,
   "metadata": {
   "tls": "not_provided_by_service_discovery"
   }
   },
   "routeMeta": null,
   "proxyDirection": "OUTBOUND",

After backend pod is scaled down to 0 replica, I don't see any of such OUTBOUND event anymore. I do see some INBOUND event but those are from K8s health probe check. In the ingress controller log, I saw below linkerd logs
"INFO outbound:accept{peer.addr=10.240.2.xxx:xxxxx}:source{target.addr=10.240.5.xxx:80}:logical{addr=ppe.testhost.xxxx.com:80}: linkerd2_proxy_http::canonicalize: DNS refinement timed out
WARN outbound:accept{peer.addr=10.240.2.xxx:xxxxx}:source{target.addr=10.240.5.xxx:80}: linkerd2_app_core::errors: Failed to proxy request: error trying to connect: Connection refused (os error 111)"

So my questions are,

1. Is it expected that we don't see OUTBOUND events anymore after backend scales down to 0? If so, when linkerd gets proxy requests error (like the log shows), can it fire some error event?
2. If the client and backend uses grpc instead of http to communication, will the OUTBOUND event include destination deployment information? asking as one client may talk with mulitple backends, and we need a way to know which backend the request attemps to use.