WIP: deduplicate and filter logs

[europaul]

This PR introduces log deduplication and filtering components to newlogd. The operations are performed as the logs are being compressed before sending them to the controller. The controller can configure which logs to filter out and which to count or deduplicate.

THIS IMPLEMENTATION IS STILL UNDER DEVELOPMENT & TESTING, PLEASE DON'T REVIEW OR MERGE!

[Copilot]

Pull Request Overview

This PR implements log deduplication and filtering for newlogd, aiming to reduce redundant log entries before they reach the controller. Key changes include:

• New modules for counting and deduplicating log entries (counter.go, dedup.go, filter.go).
• Integration of deduplication and filtering logic into the log compression process (newlogd.go) together with corresponding tests (dedup_test.go).
• Global configuration updates and minor logging improvements across related modules.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/newlog/cmd/counter.go	Adds functions to count log occurrences with a minor spelling issue
pkg/newlog/cmd/dedup.go	Implements deduplication logic using a ring buffer
pkg/newlog/cmd/dedup_test.go	Adds tests for deduplication; one loop construct appears incorrect
pkg/newlog/cmd/filter.go	Introduces filtering logic based on filename and function
pkg/newlog/cmd/newlogd.go	Integrates deduplication and filtering into log file processing
pkg/pillar/types/global.go	Adds new global settings for deduplication and filtering
Other pkg/pillar/* files	Adjust logging and error formatting in cloud connectivity functions

[Copilot]

The function name 'countLogOccurances' contains a misspelling. It should be 'countLogOccurrences'.

Suggested change

	// countLogOccurances checks if the log entries should be merged into a single entry with a counter of occurances.
	// If so it increments the counter for that log entry. It returns true if the log entry
	// should be kept in the logs file (log's first occurance) and false otherwise.
	func countLogOccurances(line []byte, filterMap map[string]int) error {
	// countLogOccurrences checks if the log entries should be merged into a single entry with a counter of occurrences.
	// If so it increments the counter for that log entry. It returns true if the log entry
	// should be kept in the logs file (log's first occurrence) and false otherwise.
	func countLogOccurrences(line []byte, filterMap map[string]int) error {

[Copilot]

The inline comment mentions 'occurance' instead of 'occurrence'. Update the spelling in the comment.

Suggested change

	// no need to add additional count field if there is only one occurance
	// no need to add additional count field if there is only one occurrence

[Copilot]

Using 'for i := range totalLogs' is invalid as totalLogs is an integer. Consider replacing it with a conventional loop, e.g., 'for i := 0; i < totalLogs; i++ {'.

Suggested change

	for i := range totalLogs {
	for i := 0; i < totalLogs; i++ {

[europaul]

This is a feature that was added in go1.22 https://go.dev/ref/spec#For_range.
But you are right, this should be fixed since we are still using go1.18.