Add `logging.with_fields` to beatreceiver config #42961

khushijain21 · 2025-02-28T11:19:29Z

Proposed commit message

It adds with_fields config to logging section of the beatreceiver - which allows for additional structured context to the logs emitted by beat process
For https://github.com/elastic/ingest-dev/issues/5139

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Part of https://github.com/elastic/ingest-dev/issues/5139

mergify · 2025-02-28T11:20:08Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

libbeat/docs/loggingconfig.asciidoc

mauri870

I had already a PR to bump elastic-agent-libs as part of a fix to all active branches at #42804.

If you do not mind, I would like to include the update to libs in that PR, in that case you might have to wait in order to merge this.

elasticmachine · 2025-03-03T04:30:10Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

auditbeat/auditbeat.reference.yml

cmacknz · 2025-03-03T20:54:28Z

libbeat/cmd/instance/beat.go

@@ -346,6 +346,14 @@ func NewBeatReceiver(settings Settings, receiverConfig map[string]interface{}, u
 		return nil, fmt.Errorf("error unpacking beats logging config: %w\n%v", err, b.Config.Logging)
 	}

+	if logpConfig.WithFields != nil {
+		var fields = []zapcore.Field{}
+		for key, value := range logpConfig.WithFields {


You should be able to write a test that ensures setting this produces logs with the expected fields. There are several system tests that run a beat and inspect the log files you could look at.

Since this is restricted to beatreceivers - will use the existing tests in elastic-agent to ensure this setting.

Related PR on elastic-agent: elastic/elastic-agent#7062

This is a Beats specific change in beat.go, it doesn't make sense to me to test it in agent because a PR in agent could never break this.

Agent will rely on this feature, but the elastic agent repository shouldn't be responsible for testing that it works on its own.

One way to think about this is a series of contracts that must be upheld:

Beat logs include arbitrary custom fields with logging.with_fields is set. You can specifically test that the component fields can be injected here.

That agent sets the logging.with_fields configuration of each beat receiver when its own log level changes.

Those two things tested independently should guarantee this works properly.

elasticmachine · 2025-03-04T08:01:11Z

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

elasticmachine · 2025-03-04T08:01:11Z

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

elasticmachine · 2025-03-04T08:01:11Z

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

elasticmachine · 2025-03-04T08:01:11Z

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

mergify · 2025-03-04T19:13:01Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b poc upstream/poc
git merge upstream/main
git push upstream poc

khushijain21 · 2025-03-05T09:35:44Z

/test

khushijain21 · 2025-03-05T13:46:04Z

/test

leehinman · 2025-03-05T19:24:22Z

libbeat/cmd/instance/beat.go

@@ -348,6 +348,14 @@ func NewBeatReceiver(settings Settings, receiverConfig map[string]interface{}, u
 		return nil, fmt.Errorf("error unpacking beats logging config: %w\n%v", err, b.Config.Logging)
 	}

+	if logpConfig.WithFields != nil {
+		var fields = []zapcore.Field{}


I agree this works, but I'm wondering why logp.ConfigureWithCore isn't doing this? It is the logp config that has the option, I would just expect it to happen there, much like the selectors get set there.

My bad, you are right. I will raise close this and open a separate PR on elastic-agent-libs. Thank you for pointing this

khushijain21 · 2025-03-06T04:56:26Z

Closed this in favor of elastic/elastic-agent-libs#290

khushijain21 requested review from a team as code owners February 28, 2025 11:19

khushijain21 requested review from mauri870 and VihasMakwana February 28, 2025 11:19

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 28, 2025

mergify bot assigned khushijain21 Feb 28, 2025

khushijain21 marked this pull request as draft February 28, 2025 11:20

mauri870 reviewed Feb 28, 2025

View reviewed changes

libbeat/docs/loggingconfig.asciidoc Outdated Show resolved Hide resolved

mauri870 reviewed Feb 28, 2025

View reviewed changes

libbeat/docs/loggingconfig.asciidoc Outdated Show resolved Hide resolved

mauri870 reviewed Feb 28, 2025

View reviewed changes

libbeat/docs/loggingconfig.asciidoc Outdated Show resolved Hide resolved

khushijain21 added 8 commits February 28, 2025 18:07

[otelconsumer]: Add support for conversion of type any

76d74c2

fix lint

863fc72

poc

c5ef3ef

first draft

9e0aefe

make update

c144433

remove files from previous commit

c62bf6d

make docs

8634ad4

add changelgo

21e2555

khushijain21 force-pushed the poc branch from 9958b4e to 21e2555 Compare February 28, 2025 12:41

update docs

f839834

khushijain21 marked this pull request as ready for review February 28, 2025 12:54

mauri870 requested changes Feb 28, 2025

View reviewed changes

khushijain21 added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Mar 3, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 3, 2025

make update

589f641

khushijain21 requested review from a team as code owners March 3, 2025 06:10

khushijain21 requested a review from a team as a code owner March 3, 2025 06:10

khushijain21 requested review from mauri870 and leehinman March 3, 2025 09:53

cmacknz reviewed Mar 3, 2025

View reviewed changes

auditbeat/auditbeat.reference.yml Outdated Show resolved Hide resolved

cmacknz reviewed Mar 3, 2025

View reviewed changes

Merge branch 'main' into poc

28403f4

khushijain21 changed the title ~~Add logging.with_fields to beat config~~ Add logging.with_fields to beatreceiver config Mar 4, 2025

khushijain21 added 2 commits March 4, 2025 18:35

remove it from public facing config

c9a394d

remove changelog

2765f2d

khushijain21 added 3 commits March 5, 2025 13:32

Merge remote-tracking branch 'refs/remotes/upstream/main' into poc

57d4f45

update notice

528ea82

add test

c6d6114

fix test

e657823

Merge branch 'main' into poc

24df137

leehinman reviewed Mar 5, 2025

View reviewed changes

khushijain21 closed this Mar 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `logging.with_fields` to beatreceiver config #42961

Add `logging.with_fields` to beatreceiver config #42961

khushijain21 commented Feb 28, 2025 •

edited

Loading

mergify bot commented Feb 28, 2025

mauri870 left a comment

elasticmachine commented Mar 3, 2025

cmacknz Mar 3, 2025

khushijain21 Mar 4, 2025 •

edited

Loading

cmacknz Mar 4, 2025

elasticmachine commented Mar 4, 2025

elasticmachine commented Mar 4, 2025

elasticmachine commented Mar 4, 2025

elasticmachine commented Mar 4, 2025

mergify bot commented Mar 4, 2025

khushijain21 commented Mar 5, 2025

khushijain21 commented Mar 5, 2025

leehinman Mar 5, 2025

khushijain21 Mar 6, 2025

khushijain21 commented Mar 6, 2025

Add logging.with_fields to beatreceiver config #42961

Add logging.with_fields to beatreceiver config #42961

Conversation

khushijain21 commented Feb 28, 2025 • edited Loading

Proposed commit message

Checklist

Related issues

mergify bot commented Feb 28, 2025

mauri870 left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 3, 2025

cmacknz Mar 3, 2025

Choose a reason for hiding this comment

khushijain21 Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

cmacknz Mar 4, 2025

Choose a reason for hiding this comment

elasticmachine commented Mar 4, 2025

elasticmachine commented Mar 4, 2025

elasticmachine commented Mar 4, 2025

elasticmachine commented Mar 4, 2025

mergify bot commented Mar 4, 2025

khushijain21 commented Mar 5, 2025

khushijain21 commented Mar 5, 2025

leehinman Mar 5, 2025

Choose a reason for hiding this comment

khushijain21 Mar 6, 2025

Choose a reason for hiding this comment

khushijain21 commented Mar 6, 2025

Add `logging.with_fields` to beatreceiver config #42961

Add `logging.with_fields` to beatreceiver config #42961

khushijain21 commented Feb 28, 2025 •

edited

Loading

khushijain21 Mar 4, 2025 •

edited

Loading