Skip to content

Commit

Permalink
Merge branch 'main' into awss3unexpectedoef
Browse files Browse the repository at this point in the history
  • Loading branch information
gizas authored Mar 6, 2025
2 parents 8cb0777 + 1db0444 commit c677f73
Show file tree
Hide file tree
Showing 27 changed files with 4,733 additions and 4,379 deletions.
154 changes: 154 additions & 0 deletions CHANGELOG.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,160 @@
:issue: https://github.com/elastic/beats/issues/
:pull: https://github.com/elastic/beats/pull/

[[release-notes-9.0.0-beta1]]
=== Beats version 9.0.0-beta1
https://github.com/elastic/beats/compare/v8.17.2\...v9.0.0-beta1[View commits]

==== Breaking changes

*Affecting all Beats*

- Set default Kafka version to 2.1.0 in Kafka output and Filebeat. {pull}41662[41662]
- Replace default Ubuntu-based images with UBI-minimal-based ones. {pull}42150[42150]
- removed support for a single `-` to precede multi-letter command line arguments. Use `--` instead. {issue}42117[42117] {pull}42209[42209]

*Filebeat*

- Filebeat fails to start if there is any input with a duplicated ID. It logs the duplicated IDs and the offending inputs configurations. {pull}41731[41731]
- Filestream inputs with duplicated IDs will fail to start. An error is logged showing the ID and the full input configuration. {issue}41938[41938] {pull}41954[41954]
- Filestream inputs can define `allow_deprecated_id_duplication: true` to run keep the previous behaviour of running inputs with duplicated IDs. {issue}41938[41938] {pull}41954[41954]
- The Filestream input only starts to ingest a file when it is >= 1024 bytes in size. This happens because the fingerprint is the default file identity now. To restore the previous behaviour, set `file_identity.native: ~` and `prospector.scanner.fingerprint.enabled: false`. {issue}40197[40197] {pull}41762[41762]
- Filebeat fails to start when its configuration contains usage of the deprecated `log` or `container` inputs. However, they can still be used when `allow_deprecated_use: true` is set in their configuration. {pull}42295[42295]

*Osquerybeat*

- Upgrade osquery version to 5.13.1. {pull}40849[40849]

*Packetbeat*

- Use base-16 for reporting `serial_number` value in TLS fields in line with the ECS recommendation. {pull}41542[41542]

*Winlogbeat*

- Default to use raw API and delete older XML implementation. {pull}42275[42275]

==== Bugfixes

*Auditbeat*

- hasher: Add a cached hasher for upcoming backend. {pull}41952[41952]
- Split common tty definitions. {pull}42004[42004]

*Filebeat*

- Redact authorization headers in HTTPJSON debug logs. {pull}41920[41920]
- Further rate limiting fix in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}41977[41977]
- The `_id` generation process for S3 events has been updated to incorporate the LastModified field. This enhancement ensures that the `_id` is unique. {pull}42078[42078]
- Fix truncation of bodies in request tracing by limiting bodies to 10% of the maximum file size. {pull}42327[42327]
- [Journald] Fixes handling of `journalctl` restart. A known symptom was broken multiline messages when there was a restart of journalctl while aggregating the lines. {issue}41331[41331] {pull}42595[42595]

*Metricbeat*

- Fix bug where Metricbeat unintentionally triggers Windows ASR. {pull}42177[42177]
- Remove `hostname` field from ZooKeeper's `mntr` data stream. {pull}41887[41887]

*Packetbeat*

- Properly marshal nested structs in ECS fields, fixing issues with mixed cases in field names. {pull}42116[42116]

==== Added

*Auditbeat*

- Improve logging in system/socket. {pull}41571[41571]

*Filebeat*

- Added out of the box support for Amazon EventBridge notifications over SQS to S3 input. {pull}40006[40006]
- Update CEL mito extensions to v1.16.0. {pull}41727[41727]
- Filebeat's registry is now added to the Elastic-Agent diagnostics bundle. {issue}33238[33238] {pull}41795[41795]
- Add `unifiedlogs` input for MacOS. {pull}41791[41791]
- Add evaluation state dump debugging option to CEL input. {pull}41335[41335]
- Rate limiting operability improvements in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}41977[41977]
- Rate limiting fault tolerance improvements in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}42094[42094]
- Introduce ignore older and start timestamp filters for AWS S3 input. {pull}41804[41804]
- Journald input now can report its status to Elastic-Agent. {issue}39791[39791] {pull}42462[42462]
- Publish events progressively in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}42567[42567]
- Journald `include_matches.match` now accepts `+` to represent a logical disjunction (OR). {issue}40185[40185] {pull}42517[42517]
- The journald input is now generally available. {pull}42107[42107]

*Heartbeat*

- Add support for RFC7231 methods to HTTP monitors. {pull}41975[41975]

*Metricbeat*

- Add `use_kubeadm` config option in kubernetes module in order to toggle kubeadm-config API requests. {pull}40086[40086]
- Preserve queries for debugging when `merge_results: true` in SQL module. {pull}42271[42271]
- Collect more fields from ES node/stats metrics and only those that are necessary. {pull}42421[42421]

*Metricbeat*
- Add benchmark module. {pull}41801[41801]

*Osquerybeat*

- Increase maximum query timeout to 24 hours. {pull}42356[42356]

*Winlogbeat*

- Properly set events `UserData` when experimental API is used. {pull}41525[41525]
- Include XML is respected for experimental API. {pull}41525[41525]
- Forwarded events use renderedtext info for experimental API. {pull}41525[41525]
- Language setting is respected for experimental API. {pull}41525[41525]
- Language setting also added to decode XML wineventlog processor. {pull}41525[41525]
- Format embedded messages in the experimental API. {pull}41525[41525]
- Make the experimental API GA and rename it to winlogbeat-raw. {issue}39580[39580] {pull}41770[41770]
- Remove 22 clause limitation. {issue}35047[35047] {pull}42187[42187]
- Add handling for recoverable publisher disabled errors. {issue}35316[35316] {pull}42187[42187]

*Functionbeat*

- Remove Functionbeat binaries from CI pipelines. {issue}40745[40745] {pull}41506[41506]


[[release-notes-8.17.3]]
=== Beats version 8.17.3
https://github.com/elastic/beats/compare/v8.17.2\...v8.17.3[View commits]

==== Bugfixes

*Affecting all Beats*

- Restored event Meta fields in the Elasticsearch output's error logs. {pull}42559[42559]

*Filebeat*

- [Journald] Fixes handling of `journalctl` restart. A known symptom was broken multiline messages when there was a restart of journalctl while aggregating the lines. {issue}41331[41331] {pull}42595[42595]
- Fix entityanalytics activedirectory provider full sync use before initialization bug. {pull}42682[42682]
- In the `http_endpoint` input, fix the check for a missing HMAC HTTP header. {pull}42756[42756]

*Metricbeat*

- Fixed panic caused by uninitialized meraki device wifi0 and wifi1 struct pointers in the device WiFi data fetching. {issue}42745[42745] {pull}42746[42746]
- Only fetch cluster-level index stats summary. {issue}36019[36019] {pull}42901[42901]
- Fixed an issue in Metricbeat's Windows module where data collection would fail if the data was unavailable. {issue}42802[42802] {pull}42803[42803]

*Winlogbeat*

- Sync missing changes in modules pipelines. {pull}42619[42619]

==== Added

*Affecting all Beats*

- Update Go version to 1.22.12. {pull}42681[42681]

*Filebeat*

- Introduce ignore older and start timestamp filters for AWS S3 input. {pull}41804[41804]
- Publish events progressively in the Okta provider of the Entity Analytics input. {issue}40106[40106] {pull}42567[42567]

*Metricbeat*

- Log every 401 response from Kubernetes API Server. {pull}42714[42714]
- Collect more fields from ES node/stats metrics and only those that are necessary. {pull}42421[42421]


[[release-notes-8.17.2]]
=== Beats version 8.17.2
https://github.com/elastic/beats/compare/v8.17.1\...v8.17.2[View commits]
Expand Down
16 changes: 1 addition & 15 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -127,21 +127,6 @@ otherwise no tag is added. {issue}42208[42208] {pull}42403[42403]
- Support Elastic Agent control protocol chunking support {pull}37343[37343]
- Lower logging level to debug when attempting to configure beats with unknown fields from autodiscovered events/environments {pull}[37816][37816]
- Set timeout of 1 minute for FQDN requests {pull}37756[37756]
- Fix issue where old data could be saved in the memory queue after acknowledgment, increasing memory use {pull}41356[41356]
- Ensure Elasticsearch output can always recover from network errors {pull}40794[40794]
- Add `translate_ldap_attribute` processor. {pull}41472[41472]
- Remove unnecessary debug logs during idle connection teardown {issue}40824[40824]
- Remove unnecessary reload for Elastic Agent managed beats when apm tracing config changes from nil to nil {pull}41794[41794]
- Fix incorrect cloud provider identification in add_cloud_metadata processor using provider priority mechanism {pull}41636[41636]
- Prevent panic if libbeat processors are loaded more than once. {issue}41475[41475] {pull}41857[51857]
- Allow network condition to handle field values that are arrays of IP addresses. {pull}41918[41918]
- Fix a bug where log files are rotated on startup when interval is configured and rotateonstartup is disabled {issue}41894[41894] {pull}41895[41895]
- Fix setting unique registry for non beat receivers {issue}42288[42288] {pull}42292[42292]
- The Kafka output now drops events when there is an authorisation error {issue}42343[42343] {pull}42401[42401]
- Fix autodiscovery memory leak related to metadata of start events {pull}41748[41748]
- All standard queue metrics are now included in metrics monitoring, including: `added.{events, bytes}`, `consumed.{events, bytes}`, `removed.{events, bytes}`, and `filled.{events, bytes, pct}`. {pull}42439[42439]
- The following output latency metrics are now included in metrics monitoring: `output.latency.{count, max, median, p99}`. {pull}42439[42439]
- Restored event Meta fields in the Elasticsearch output's error logs. {pull}42559[42559]

*Auditbeat*

Expand Down Expand Up @@ -283,6 +268,7 @@ otherwise no tag is added. {issue}42208[42208] {pull}42403[42403]
- Sync missing changes in modules pipelines. {pull}42619[42619]
- Reset EventLog if error EOF is encountered. {pull}42826[42826]
- Implement backoff on error retrial. {pull}42826[42826]
- Fix boolean key in security pipelines and sync pipelines with integration. {pull}43027[43027]


*Elastic Logging Plugin*
Expand Down
44 changes: 43 additions & 1 deletion dev-tools/mage/build.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import (
"log"
"os"
"path/filepath"
"regexp"
"strings"

"github.com/josephspurrier/goversioninfo"
Expand All @@ -46,6 +47,39 @@ type BuildArgs struct {
WinMetadata bool // Add resource metadata to Windows binaries (like add the version number to the .exe properties).
}

// buildTagRE is a regexp to match strings like "-tags=abcd"
// but does not match "-tags= "
var buildTagRE = regexp.MustCompile(`-tags=([\S]+)?`)

// ParseBuildTags returns the ExtraFlags param where all flags that are go build tags are joined by a comma.
//
// For example if given -someflag=val1 -tags=buildtag1 -tags=buildtag2
// It will return -someflag=val1 -tags=buildtag1,buildtag2
func (b BuildArgs) ParseBuildTags() []string {
flags := make([]string, 0)
if len(b.ExtraFlags) == 0 {
return flags
}

buildTags := make([]string, 0)
for _, flag := range b.ExtraFlags {
if buildTagRE.MatchString(flag) {
arr := buildTagRE.FindStringSubmatch(flag)
if len(arr) != 2 || arr[1] == "" {
log.Printf("Parsing buildargs.ExtraFlags found strange flag %q ignoring value", flag)
continue
}
buildTags = append(buildTags, arr[1])
} else {
flags = append(flags, flag)
}
}
if len(buildTags) > 0 {
flags = append(flags, "-tags="+strings.Join(buildTags, ","))
}
return flags
}

// DefaultBuildArgs returns the default BuildArgs for use in builds.
func DefaultBuildArgs() BuildArgs {
args := BuildArgs{
Expand Down Expand Up @@ -74,6 +108,10 @@ func DefaultBuildArgs() BuildArgs {
// Remove all file system paths from the compiled executable, to improve build reproducibility
args.ExtraFlags = append(args.ExtraFlags, "-trimpath")
}
if FIPSBuild {
args.ExtraFlags = append(args.ExtraFlags, "-tags=requirefips")
args.CGO = true
}

return args
}
Expand Down Expand Up @@ -175,6 +213,10 @@ func Build(params BuildArgs) error {
if params.CGO {
cgoEnabled = "1"
}
if FIPSBuild {
cgoEnabled = "1"
env["GOEXPERIMENT"] = "systemcrypto"
}
env["CGO_ENABLED"] = cgoEnabled

// Spec
Expand All @@ -186,7 +228,7 @@ func Build(params BuildArgs) error {
if params.BuildMode != "" {
args = append(args, "-buildmode", params.BuildMode)
}
args = append(args, params.ExtraFlags...)
args = append(args, params.ParseBuildTags()...)

// ldflags
ldflags := params.LDFlags
Expand Down
72 changes: 72 additions & 0 deletions dev-tools/mage/build_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
// Licensed to Elasticsearch B.V. under one or more contributor
// license agreements. See the NOTICE file distributed with
// this work for additional information regarding copyright
// ownership. Elasticsearch B.V. licenses this file to you under
// the Apache License, Version 2.0 (the "License"); you may
// not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

package mage

import (
"testing"

"github.com/stretchr/testify/assert"
)

func Test_BuildArgs_ParseBuildTags(t *testing.T) {
tests := []struct {
name string
input []string
expect []string
}{{
name: "no flags",
input: nil,
expect: []string{},
}, {
name: "multiple flags with no tags",
input: []string{"-a", "-b", "-key=value"},
expect: []string{"-a", "-b", "-key=value"},
}, {
name: "one build tag",
input: []string{"-tags=example"},
expect: []string{"-tags=example"},
}, {
name: "multiple build tags",
input: []string{"-tags=example", "-tags=test"},
expect: []string{"-tags=example,test"},
}, {
name: "joined build tags",
input: []string{"-tags=example,test"},
expect: []string{"-tags=example,test"},
}, {
name: "multiple build tags with other flags",
input: []string{"-tags=example", "-tags=test", "-key=value", "-a"},
expect: []string{"-key=value", "-a", "-tags=example,test"},
}, {
name: "incorrectly formatted tag",
input: []string{"-tags= example"},
expect: []string{},
}, {
name: "incorrectly formatted tag with valid tag",
input: []string{"-tags= example", "-tags=test"},
expect: []string{"-tags=test"},
}}

for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
args := BuildArgs{ExtraFlags: tc.input}
flags := args.ParseBuildTags()
assert.EqualValues(t, tc.expect, flags)
})
}
}
6 changes: 6 additions & 0 deletions dev-tools/mage/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,12 @@ func untar(sourceFile, destinationDir string) error {
return err
}
case tar.TypeReg:
// create containing folder if it doesn't exist yet
targetContainingDir := filepath.Dir(filepath.FromSlash(path))
if mkDirErr := os.MkdirAll(targetContainingDir, 0755); mkDirErr != nil {
return fmt.Errorf("creating container directory for file %s: %w", header.Name, mkDirErr)
}

writer, err := os.Create(path)
if err != nil {
return err
Expand Down
4 changes: 4 additions & 0 deletions dev-tools/mage/crossbuild.go
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,9 @@ func CrossBuildImage(platform string) (string, error) {
if err != nil {
return "", err
}
if FIPSBuild {
tagSuffix += "-fips"
}

return BeatsCrossBuildImage + ":" + goVersion + "-" + tagSuffix, nil
}
Expand Down Expand Up @@ -331,6 +334,7 @@ func (b GolangCrossBuilder) Build() error {
"--env", "MAGEFILE_VERBOSE="+verbose,
"--env", "MAGEFILE_TIMEOUT="+EnvOr("MAGEFILE_TIMEOUT", ""),
"--env", fmt.Sprintf("SNAPSHOT=%v", Snapshot),
"--env", fmt.Sprintf("FIPS=%v", FIPSBuild),
"-v", repoInfo.RootDir+":"+mountPoint,
"-w", workDir,
)
Expand Down
3 changes: 3 additions & 0 deletions dev-tools/mage/dockerbuilder.go
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,9 @@ func (b *dockerBuilder) dockerBuild() (string, error) {
if b.Snapshot {
tag = tag + "-SNAPSHOT"
}
if b.FIPS {
tag = tag + "-fips"
}
if repository, _ := b.ExtraVars["repository"]; repository != "" {
tag = fmt.Sprintf("%s/%s", repository, tag)
}
Expand Down
4 changes: 2 additions & 2 deletions dev-tools/mage/gotest.go
Original file line number Diff line number Diff line change
Expand Up @@ -292,9 +292,9 @@ func GoTest(ctx context.Context, params GoTestArgs) error {
}
}
if len(params.Tags) > 0 {
params := strings.Join(params.Tags, " ")
params := strings.Join(params.Tags, ",")
if params != "" {
testArgs = append(testArgs, "-tags", params)
testArgs = append(testArgs, "-tags="+params)
}
}
if params.CoverageProfileFile != "" {
Expand Down
Loading

0 comments on commit c677f73

Please sign in to comment.