Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logging.with_fields to beatreceiver config #42961

Closed
wants to merge 18 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions NOTICE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12488,11 +12488,11 @@ SOFTWARE

--------------------------------------------------------------------------------
Dependency : github.com/elastic/elastic-agent-libs
Version: v0.18.1
Version: v0.18.7
Licence type (autodetected): Apache-2.0
--------------------------------------------------------------------------------

Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-libs@v0.18.1/LICENSE:
Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-libs@v0.18.7/LICENSE:

Apache License
Version 2.0, January 2004
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ require (
github.com/elastic/bayeux v1.0.5
github.com/elastic/ebpfevents v0.6.0
github.com/elastic/elastic-agent-autodiscover v0.9.0
github.com/elastic/elastic-agent-libs v0.18.1
github.com/elastic/elastic-agent-libs v0.18.7
github.com/elastic/elastic-agent-system-metrics v0.11.7
github.com/elastic/go-elasticsearch/v8 v8.17.0
github.com/elastic/go-quark v0.3.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -350,8 +350,8 @@ github.com/elastic/elastic-agent-autodiscover v0.9.0 h1:+iWIKh0u3e8I+CJa3FfWe9h0
github.com/elastic/elastic-agent-autodiscover v0.9.0/go.mod h1:5iUxLHhVdaGSWYTveSwfJEY4RqPXTG13LPiFoxcpFd4=
github.com/elastic/elastic-agent-client/v7 v7.15.0 h1:nDB7v8TBoNuD6IIzC3z7Q0y+7bMgXoT2DsHfolO2CHE=
github.com/elastic/elastic-agent-client/v7 v7.15.0/go.mod h1:6h+f9QdIr3GO2ODC0Y8+aEXRwzbA5W4eV4dd/67z7nI=
github.com/elastic/elastic-agent-libs v0.18.1 h1:dE6jf/D9bP8eRMQsV7KKpKV/G8zQzwMFBTj1w4e716c=
github.com/elastic/elastic-agent-libs v0.18.1/go.mod h1:rWdyrrAFzZwgNNi41Tsqhlt2c2GdXWhCEwcsnqISJ2U=
github.com/elastic/elastic-agent-libs v0.18.7 h1:C/63JieRiRIKBCOHnusIQ6yGBBmTU9rqcxneOw3zVX4=
github.com/elastic/elastic-agent-libs v0.18.7/go.mod h1:Repx7BMzE1v/gTipPogNIQeEnSGwOWGBC63h7h9c5aM=
github.com/elastic/elastic-agent-system-metrics v0.11.7 h1:1xm2okCM0eQZ4jivZgUFSlt6HAn/nPgKB/Fj8eLG6mY=
github.com/elastic/elastic-agent-system-metrics v0.11.7/go.mod h1:nzkrGajQA29YNcfP62gfzhxX9an3/xdQ3RmfQNw9YTI=
github.com/elastic/elastic-transport-go/v8 v8.6.0 h1:Y2S/FBjx1LlCv5m6pWAF2kDJAHoSjSRSJCApolgfthA=
Expand Down
8 changes: 8 additions & 0 deletions libbeat/cmd/instance/beat.go
Original file line number Diff line number Diff line change
Expand Up @@ -348,6 +348,14 @@ func NewBeatReceiver(settings Settings, receiverConfig map[string]interface{}, u
return nil, fmt.Errorf("error unpacking beats logging config: %w\n%v", err, b.Config.Logging)
}

if logpConfig.WithFields != nil {
var fields = []zapcore.Field{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree this works, but I'm wondering why logp.ConfigureWithCore isn't doing this? It is the logp config that has the option, I would just expect it to happen there, much like the selectors get set there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, you are right. I will raise close this and open a separate PR on elastic-agent-libs. Thank you for pointing this

for key, value := range logpConfig.WithFields {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to write a test that ensures setting this produces logs with the expected fields. There are several system tests that run a beat and inspect the log files you could look at.

Copy link
Contributor Author

@khushijain21 khushijain21 Mar 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is restricted to beatreceivers - will use the existing tests in elastic-agent to ensure this setting.

Related PR on elastic-agent: elastic/elastic-agent#7062

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a Beats specific change in beat.go, it doesn't make sense to me to test it in agent because a PR in agent could never break this.

Agent will rely on this feature, but the elastic agent repository shouldn't be responsible for testing that it works on its own.

One way to think about this is a series of contracts that must be upheld:

  1. Beat logs include arbitrary custom fields with logging.with_fields is set. You can specifically test that the component fields can be injected here.
  2. That agent sets the logging.with_fields configuration of each beat receiver when its own log level changes.

Those two things tested independently should guarantee this works properly.

fields = append(fields, zap.Any(key, value))
}
core = core.With(fields)
}

if err := logp.ConfigureWithCore(logpConfig, core); err != nil {
return nil, fmt.Errorf("error configuring beats logp: %w", err)
}
Expand Down
60 changes: 59 additions & 1 deletion x-pack/filebeat/tests/integration/otel_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,14 @@
package integration

import (
"bufio"
"context"
"crypto/tls"
"fmt"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
"time"

Expand Down Expand Up @@ -40,8 +42,8 @@ output:
hosts:
- localhost:9200
protocol: http
password: testing
username: admin
password: testing
index: %s
queue.mem.flush.timeout: 0s
processors:
Expand Down Expand Up @@ -170,3 +172,59 @@ func assertMapsEqual(t *testing.T, m1, m2 mapstr.M, ignoredFields []string, msg
}
require.Equal(t, "", cmp.Diff(flatM1, flatM2), "expected maps to be equal")
}

// This test ensures the functionality of `logging.with_fields` on beatreceivers
func TestBeatReceiverLoggingWithFields(t *testing.T) {
var filebeatBasicConfig = `
filebeat.inputs:
- type: filestream
id: filestream-input-id
enabled: true
paths:
- /tmp/flog.log
output:
elasticsearch:
hosts:
- localhost:9200
protocol: http
username: admin
password: testing
logging.with_fields:
component: filebeat-otel-test
path.home: %s

`
// start filebeat in otel mode
filebeatOTel := integration.NewBeat(
t,
"filebeat-otel",
"../../filebeat.test",
"otel",
)

tempDir := filebeatOTel.TempDir()
s := fmt.Sprintf(filebeatBasicConfig, tempDir)
filebeatOTel.WriteConfigFile(s)
filebeatOTel.Start()

require.Eventually(t, func() bool {
// we must open it each time in case no logs have arrived yet
f, err := os.Open(filepath.Join(tempDir, "stderr"))
if err != nil {
return false
}
defer f.Close()

r := bufio.NewScanner(f)

for r.Scan() {
line := r.Text()
if strings.Contains(line, `"name": "filebeatreceiver"`) {
fmt.Println(line)
return assert.True(t, strings.Contains(line, `"component": "filebeat-otel-test"`))
}
}
return false
}, 10*time.Second, 100*time.Millisecond, "additional logging context was not added")

}
Loading