Remove deprecated package autorest/adal from beats

[twozniak1]

Beats currently uses the go-autorest/adal package, which has officially reached End-of-Life (EOL). The package's EOL status is confirmed here, and it is also flagged as deprecated by several security tools.

Given the potential security and maintenance implications, do you have any plans to remove or replace this deprecated package? This update would help improve the security posture of the project and align it with actively maintained dependencies.

Thanks in advance for any information or roadmap details you can share on this topic!

[yang-wei]

This package depends on github.com/golang-jwt/jwt/v4 v4.5.0 which also contains a low severity CVE https://nvd.nist.gov/vuln/detail/CVE-2024-51744

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[chrisberkhout]

I've opened a PR to remove the direct dependency on autorest/adal by using azidentity for token fetching and refresh:
#42959.

However, 3 indirect dependencies remain...

1: Via autorest in the o365audit input

github.com/elastic/beats/v7/x-pack/filebeat/input/o365audit
↳ github.com/Azure/go-autorest/autorest
  ↳github.com/Azure/go-autorest/autorest/adal

The o365audit input uses autorest as its HTTP client. The adal part shouldn't be used, but
autorest does declare it as a dependency. The o365audit code depends heavily on autorest's request pipeline model, so removing autorest would be close to a full rewrite of the input. The input was never marked as GA and was deprecated in 8.14.0:

The o365audit input is deprecated. For collecting Microsoft Office 365 log data, please use the Microsoft 365 integration package. For more complex or user-specific use cases, similar functionality can be achieved using the CEL input .

The o365audit input is owned by @elastic/security-service-integrations.

2: Via azure-event-hubs-go in the azureeventhub input

github.com/elastic/beats/v7/x-pack/filebeat/input/azureeventhub
↳ github.com/Azure/azure-event-hubs-go/v3/storage
  ↳ github.com/Azure/go-autorest/autorest/adal

Only the v1 implementation of azureeventhub uses azure-event-hubs-go. The v2 implementation followed the recommended migration path by switching to azure-sdk-for-go, in #39796.

The azureeventhub input is owned by @elastic/obs-ds-hosted-services.

3: Via autorest in the app_insights metrics set in the azure module

github.com/elastic/beats/v7/x-pack/metricbeat/module/azure/app_insights
↳ github.com/Azure/go-autorest/autorest
  ↳ github.com/Azure/go-autorest/autorest/adal

The azure module is owned by @elastic/obs-ds-hosted-services.

[chrisberkhout]

Related: elastic/integrations#11775