Virtual Router to output logs to Cloudstack Mgmt Server, so Operator can monitor via Zabbix, Loki or other system

[btzq]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

VR, Cloudstack Mgmt

CLOUDSTACK VERSION

4.19.0

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

When a VR is created, the logs are only being output within the VM, which makes it very difficult to monitor.

When an issue occurs, it is likely the cloud operator would reboot or clean up the router, resulting the VR being recreated and logs being lost. But this is done to restore the services with as little downtime as possible.

It would be great to have the VR output logs back to a central location in Cloudstack Management. Then, using some sort of log monitoring tool (Zabbix, Grafana Loki, Wazuh SIEM), we can monitor logs of all VRs created to lookout for key words which may signify an issue.

This can also lead to potential future automations such as: 'If Keyword exist, cleanup VPC' via some self built automation service that talks to Cloudstack APIs.

STEPS TO REPRODUCE

NA

EXPECTED RESULTS

NA

ACTUAL RESULTS

Difficult for Cloud Operator to manage and monitor VR and the logs. 

[DaanHoogland]

@btzq , this is not the first time this comes up. There have been implementation attempt on this as well. Maybe it is good to have a discuss item to discuss how this would be implemented and how configurable it would be. (https://github.com/apache/cloudstack/discussions/new/choose)

[btzq]

I see @DaanHoogland , then can you help convert this ticket to a discussion?
We can continue discussions there.

[weizhouapache]

fyi, may refer to #9179

[btzq]

Hi @DaanHoogland @weizhouapache , ive taken a look at #9179 but it still looks like some way to go.

Is it possible to run a userdata/script to the VR upon each new creation/reboot?

I am thinking of running a script to auto download and install the Grafana Alloy or Zabbix Agent and monitor the logs from a fixed file path, and send it to our Grafana or Zabbix Server.

That might be a good short term solution.

[DaanHoogland]

if you have a configuration engine running (configd/chef/puppet/ansible/salt), I'd use that to install whatever you need on the agent. Think hard about security implications though.
You can use webhooks to add new VRs to your inventory.