[fbreceivier] inject log level #42798

VihasMakwana · 2025-02-20T10:17:16Z

This PR maps logging.level from beats to service::telemetry::logs::level, so that the log level is respected across config translation

Screenshots:

logging with warning level

config.yaml

filebeat.inputs:
  - type: filestream
    id: filestream-input-id
    enabled: true
    paths:
      - /Users/vihasmakwana/Desktop/Vihas/elastic/beats/x-pack/filebeat/logs/*.log

output:
  elasticsearch:
    hosts: ["https://localhost:9200"]
    username: elastic
    password: changeme

logging.level: warning

2. logging with debug level

config.yaml

filebeat.inputs:
  - type: filestream
    id: filestream-input-id
    enabled: true
    paths:
      - /Users/vihasmakwana/Desktop/Vihas/elastic/beats/x-pack/filebeat/logs/*.log

output:
  elasticsearch:
    hosts: ["https://localhost:9200"]
    username: elastic
    password: changeme

logging.level: debug

Closes https://github.com/elastic/ingest-dev/issues/5030

mergify · 2025-02-20T10:17:59Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @VihasMakwana? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

elasticmachine · 2025-03-03T14:13:06Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

khushijain21 · 2025-03-03T14:54:36Z

libbeat/otelbeat/providers/fbprovider/fbprovider.go

@@ -72,7 +72,11 @@ func (fmp *provider) Retrieve(_ context.Context, uri string, _ confmap.WatcherFu
 			},
 		},
 	}
-
+	// inject log level


can you move this to beatconverter.go? Since this is common for all beats

khushijain21 · 2025-03-03T14:56:41Z

libbeat/otelbeat/providers/fbprovider/fbprovider.go

+	// inject log level
+	level, _ := cfg.String("logs.level", -1)
+	if level != "" {
+		cfgMap["service::telemetry::logs::level"] = level


you may want to map beats' logging level to otel level first.

For example. logging.level supports values such as warning which should translate to WARN level on otel.

For ref:
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html#level
https://opentelemetry.io/docs/collector/internal-telemetry/#configure-internal-logs

AndersonQ

as Khushi said, please add a translation instead of just copying it to the OTel config.

VihasMakwana · 2025-03-04T09:58:46Z

@AndersonQ can you please re-review?

VihasMakwana · 2025-03-04T14:27:03Z

/test

AndersonQ

[Blocker]
Shouldn't it reject a wrong log level just as the beats already do?
getOTeLLogLevel could return an error if the level is invalid.

If I run this code with an invalid log level, it'll error and exit, but not because it's handled here, but because it's handled somewhere else.

config:

filebeat.inputs:
  - type: filestream
    id: filestream-input-id
    enabled: true
    paths:
      - /home/ainsoph/tmp/log.log

output:
  elasticsearch:
    hosts: [ "https://localhost:9200" ]
    username: elastic
    password: changeme
    ssl.verification_mode: none

logging.level: pastel

running it:

❯ go run . otel -c filebeat-otel.yml
2025-03-04T16:24:03.466+0100	info	service@v0.114.0/service.go:166	Setting up own telemetry...
2025-03-04T16:24:03.467+0100	info	telemetry/metrics.go:70	Serving metrics	{"address": "localhost:8888", "metrics level": "Normal"}
2025-03-04T16:24:03.467+0100	warn	elasticsearchexporter@v0.114.0/config.go:378	dedot has been deprecated: in the future, dedotting will always be performed in ECS mode only	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch"}
Error: failed to build pipelines: failed to create "filebeatreceiver" receiver for data type "logs": error creating filebeatreceiver: error unpacking beats logging config: invalid level 'pastel' accessing 'logging.level'
&{{{0xc00207c600} logging} <nil> 0xc00239c020}
Usage:
  filebeat otel [flags]

it fails on another place:

Error: failed to build pipelines: failed to create "filebeatreceiver" receiver for data type "logs": error creating filebeatreceiver: error unpacking beats logging config: invalid level 'pastel' accessing 'logging.level'

last but not least, could you add a test as well?

AndersonQ · 2025-03-04T15:21:22Z

libbeat/otelbeat/beatconverter/beatconverter.go

@@ -111,6 +111,12 @@ func (c converter) Convert(_ context.Context, conf *confmap.Conf) error {
 			beatReceiverConfigKey + "::output::otelconsumer": nil,
 		}

+		// inject log level
+		level, _ := config.MustNewConfigFrom(receiverCfg.ToStringMap()).String("logging.level", -1)


[Blocker]
Please, don't panic.
use the right method and handle the error gracefully

Suggested change

level, _ := config.MustNewConfigFrom(receiverCfg.ToStringMap()).String("logging.level", -1)

cfg, err := config.NewConfigFrom(receiverCfg.ToStringMap())

if err!=nil{

return fmt.Errorf("proper error message: %w", err)

}

level, err:=cfg.String("logging.level", -1)

if err!=nil{

return fmt.Errorf("proper error message: %w", err)

}```

I think we should be good now.

libbeat/otelbeat/beatconverter/logging.go

Co-authored-by: Anderson Queiroz <me@andersonq.me>

libbeat/otelbeat/beatconverter/beatconverter.go

VihasMakwana · 2025-03-06T10:47:42Z

@AndersonQ @khushijain21 please take a look when you can:

Test case added
We don't panic anymore
We return error if an unknown log level has been specified

khushijain21 · 2025-03-06T13:40:14Z

libbeat/otelbeat/beatconverter/beatconverter_test.go

@@ -279,6 +280,75 @@ service:

 	})

+	// t.Run("test log level",


Suggested change

// t.Run("test log level",

khushijain21 · 2025-03-06T13:40:26Z

LGTM

VihasMakwana · 2025-03-06T19:18:00Z

/test

inject log level

71c22dd

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 20, 2025

mergify bot assigned VihasMakwana Feb 20, 2025

VihasMakwana changed the title ~~inject log level~~ [fbreceivier] inject log level Mar 3, 2025

VihasMakwana marked this pull request as ready for review March 3, 2025 14:12

VihasMakwana requested a review from a team as a code owner March 3, 2025 14:12

VihasMakwana requested review from AndersonQ and leehinman March 3, 2025 14:12

VihasMakwana added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Mar 3, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 3, 2025

khushijain21 reviewed Mar 3, 2025

View reviewed changes

AndersonQ requested changes Mar 3, 2025

View reviewed changes

review comments

9f52f4f

VihasMakwana requested review from khushijain21 and AndersonQ March 4, 2025 06:53

khushijain21 approved these changes Mar 4, 2025

View reviewed changes

Merge branch 'main' into log-level-debug

566e5d5

AndersonQ requested changes Mar 4, 2025

View reviewed changes

Update libbeat/otelbeat/beatconverter/logging.go

56bc4ff

Co-authored-by: Anderson Queiroz <me@andersonq.me>

VihasMakwana requested a review from AndersonQ March 4, 2025 15:55

Merge branch 'main' into log-level-debug

af93ae8

khushijain21 reviewed Mar 5, 2025

View reviewed changes

libbeat/otelbeat/beatconverter/beatconverter.go Outdated Show resolved Hide resolved

comments and test case

0ac5848

VihasMakwana requested a review from khushijain21 March 6, 2025 10:47

khushijain21 reviewed Mar 6, 2025

View reviewed changes

khushijain21 approved these changes Mar 6, 2025

View reviewed changes

leehinman approved these changes Mar 6, 2025

View reviewed changes

AndersonQ approved these changes Mar 6, 2025

View reviewed changes

VihasMakwana and others added 2 commits March 6, 2025 22:01

remove comment

68e93e9

Merge branch 'main' into log-level-debug

dd2a83d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fbreceivier] inject log level #42798

[fbreceivier] inject log level #42798

VihasMakwana commented Feb 20, 2025 •

edited

Loading

mergify bot commented Feb 20, 2025

elasticmachine commented Mar 3, 2025

khushijain21 Mar 3, 2025

khushijain21 Mar 3, 2025

AndersonQ left a comment

VihasMakwana commented Mar 4, 2025

VihasMakwana commented Mar 4, 2025

AndersonQ left a comment

AndersonQ Mar 4, 2025

VihasMakwana Mar 6, 2025

VihasMakwana commented Mar 6, 2025

khushijain21 Mar 6, 2025

khushijain21 commented Mar 6, 2025

VihasMakwana commented Mar 6, 2025

-		level, _ := config.MustNewConfigFrom(receiverCfg.ToStringMap()).String("logging.level", -1)
+		cfg, err := config.NewConfigFrom(receiverCfg.ToStringMap())
+		if err!=nil{
+			return fmt.Errorf("proper error message: %w", err)
+		}
+		level, err:=cfg.String("logging.level", -1)
+		if err!=nil{
+			return fmt.Errorf("proper error message: %w", err)
+		}```

[fbreceivier] inject log level #42798

Are you sure you want to change the base?

[fbreceivier] inject log level #42798

Conversation

VihasMakwana commented Feb 20, 2025 • edited Loading

Screenshots:

mergify bot commented Feb 20, 2025

elasticmachine commented Mar 3, 2025

khushijain21 Mar 3, 2025

Choose a reason for hiding this comment

khushijain21 Mar 3, 2025

Choose a reason for hiding this comment

AndersonQ left a comment

Choose a reason for hiding this comment

VihasMakwana commented Mar 4, 2025

VihasMakwana commented Mar 4, 2025

AndersonQ left a comment

Choose a reason for hiding this comment

AndersonQ Mar 4, 2025

Choose a reason for hiding this comment

VihasMakwana Mar 6, 2025

Choose a reason for hiding this comment

VihasMakwana commented Mar 6, 2025

khushijain21 Mar 6, 2025

Choose a reason for hiding this comment

khushijain21 commented Mar 6, 2025

VihasMakwana commented Mar 6, 2025

VihasMakwana commented Feb 20, 2025 •

edited

Loading