Merge pull request #371 from alltilla/4.9.0-news-entries

news: add missing 4.9.0 news entries

Author: MrAnno

news/README.md

@@ -4,6 +4,8 @@ How to create a newsfile entry
  1. Create a file in the `news/` directory called `<type>-<pr-id>.md`, where `<type>` is either:
      * `feature`: New functionality.
      * `bugfix`: Fix to a reported bug.
+     * `fx-feature`: FilterX related feature.
+     * `fx-bugfix`: FilterX related bugfix.
      * `packaging`: Packaging related change.
      * `developer-note`: Changes, that are only interesting to developers. (internal API change, etc...)
      * `other`: Other important, but not categorized change.

news/bugfix-296.md

@@ -0,0 +1,2 @@
+`stats`: Fixed `free_window` counters.
+

news/create-newsfile.py

@@ -1,5 +1,7 @@
 #!/usr/bin/env python3
 #############################################################################
+# Copyright (c) 2024 Axoflow
+# Copyright (c) 2024 Attila Szakacs <attila.szakacs@axoflow.com>
 # Copyright (c) 2020 Balabit
 #
 # This program is free software; you can redistribute it and/or modify it
@@ -41,6 +43,8 @@
 blocks = [
     ('Features', 'feature-*.md'),
     ('Bugfixes', 'bugfix-*.md'),
+    ('FilterX features', 'fx-feature-*.md'),
+    ('FilterX bugfixes', 'fx-bugfix-*.md'),
     ('Packaging', 'packaging-*.md'),
     ('Notes to developers', 'developer-note-*.md'),
     ('Other changes', 'other-*.md'),

news/feature-326.md

@@ -0,0 +1,14 @@
+`syslog-ng-ctl`: Added `attach` subcommand.
+
+With `attach`, it is possible to attach to the
+standard IO of the `syslog-ng` proccess.
+
+Example usage:
+```
+# takes the stdio fds for 10 seconds and displays syslog-ng output in that time period
+$ syslog-ng-ctl attach stdio --seconds 10
+```
+```
+# steal trace level log messages for 10 seconds
+$ syslog-ng-ctl attach logs --seconds 10 --log-level trace
+```

news/feature-351.md

@@ -0,0 +1,4 @@
+socked based sources: Added new flag, called `exit-on-eof`
+
+Setting this flag to a source makes AxoSyslog stop,
+when EOF is received.

news/feature-355.md

@@ -0,0 +1,4 @@
+socked based sources: Added a new option called `idle-timeout()`.
+
+Setting this option makes AxoSyslog close the client connection
+if no data is received for the set amount of seconds.

news/fx-bugfix-218.md

@@ -0,0 +1,2 @@
+`format_csv()`: Fixed delimiter formatting.
+

news/fx-bugfix-230.md

@@ -0,0 +1,2 @@
+`json`: Fixed a crash that occured when doubles were stored and accessed.
+

news/fx-bugfix-249.md

@@ -0,0 +1,2 @@
+`parse_csv()`: Fixed a race condition.
+

news/fx-bugfix-257.md

@@ -0,0 +1,2 @@
+Fixed race conditions in several functions.
+

news/fx-bugfix-258.md

@@ -0,0 +1,2 @@
+`json`: Fixed race condition in marshalling.
+

news/fx-bugfix-273.md

@@ -0,0 +1,2 @@
+`json_array`: Fixed failing to return `null` values.
+

news/fx-bugfix-287-2.md

@@ -0,0 +1,2 @@
+`parse_csv()`: Fixed an invalid read.
+

news/fx-bugfix-333.md

@@ -0,0 +1,2 @@
+Fixed LogMessage -> FilterX variable synchronization.
+

news/fx-feature-203.md

@@ -0,0 +1,19 @@
+`regexp_subst()`: Added various pcre flags.
+
+* `jit`:
+  * enables or disables JIT compliling
+  * default: `true`
+* `global`:
+  * sets whether all found matches should be replaced
+  * default: `false`
+* `utf8`:
+  * enables or disables UTF-8 validation
+  * default: `false`
+* `ignorecase`
+  * sets case sensitivity
+  * default: `false` (case-sensitive)
+* `newline`
+  * configures the behavior of end of line finding
+  * `false` returns end of line when CR, LF and CRLF characters are found
+  * `true` makes the matcher process CR, LF, CRLF characters
+  * default: `false`

news/fx-feature-215.md

@@ -0,0 +1,2 @@
+`unset()`: Now accepts any number of variables to unset.
+

news/fx-feature-217.md

@@ -0,0 +1,2 @@
+Added `+` operator.
+

news/fx-feature-219-1.md

@@ -0,0 +1,2 @@
+`parse_csv()`: Changed strip whitespace default to `false`.
+

news/fx-feature-219-2.md

@@ -0,0 +1,2 @@
+`parse_csv()`: Renamed `strip_whitespaces` argument to `strip_whitespace`.
+

news/fx-feature-220.md

@@ -0,0 +1,6 @@
+`update_metric()`: Added a new function similar to `metrics-probe` parser.
+
+Example usage:
+```
+update_metric("filterx_metric", labels={"msg": $MSG, "foo": "foovalue"}, level=1, increment=$INCREMENT);
+```

news/fx-feature-221.md

@@ -0,0 +1,10 @@
+`flatten()`: Added new function to flatten dicts and lists.
+
+The function modifies the object in-place.
+The separator can be set with the `separator` argument,
+which is `.` by default.
+
+Example usage:
+```
+flatten(my_dict_or_list, separator="->");
+```

news/fx-feature-238.md

@@ -0,0 +1,2 @@
+Added `!~` operator as the negated `=~` operator.
+

news/fx-feature-242.md

@@ -0,0 +1,15 @@
+Added new RFC5424 SDATA related functions.
+
+All of the functions require traditional syslog parsing beforehand.
+
+* `has_sdata()`
+  * Returns whether the current log has SDATA information.
+  * Example: `sdata_avail = has_sdata(;)`
+* `is_sdata_from_enterprise()`
+  * Checks if there is SDATA that corresponds to the given enterprise ID.
+  * Example: `sdata_from_6876 = is_sdata_from_enterprise("6876");`
+* `get_sdata()`
+  * Returns a 2 level dict of the available SDATAs.
+  * Example: `sdata = get_sdata();`
+  * Returns: `{"Originator@6876": {"sub": "Vimsvc.ha-eventmgr", "opID": "esxui-13c6-6b16"}}`
+

news/fx-feature-251.md

@@ -0,0 +1,52 @@
+`parse_xml()`: Added new function to parse XMLs.
+
+Example usage:
+```
+my_structured_data = parse_xml(raw_xml);
+```
+
+Converting XML to a dict is not standardized.
+
+Our intention is to create the most compact dict as possible,
+which means certain nodes will have different types and
+structures based on a number of different qualities of the
+input XML element.
+
+The following points will demonstrate the choices we made in our parser.
+In the examples we will use the JSON dict implementation.
+
+1. Empty XML elements become empty strings.
+```
+  XML:  <foo></foo>
+  JSON: {"foo": ""}
+```
+
+2. Attributions are stored in `@attr` key-value pairs,
+   similarly to some other converters (e.g.: python xmltodict).
+```
+  XML:  <foo bar="123" baz="bad"/>
+  JSON: {"foo": {"@bar": "123", "@baz": "bad"}}
+```
+
+3. If an XML element has both attributes and a value, 
+   we need to store them in a dict, and the value needs a key.
+   We store the text value under the #text key.
+```
+  XML:  <foo bar="123">baz</foo>
+  JSON: {"foo": {"@bar": "123", "#text": "baz"}}
+```
+
+4. An XML element can have both a value and inner elements.
+   We use the `#text` key here, too.
+```
+  XML:  <foo>bar<baz>123</baz></foo>
+  JSON: {"foo": {"#text": "bar", "baz": "123"}}
+```
+
+5. An XML element can have multiple values separated by inner elements.
+   In that case we concatenate the values.
+```
+  XML:  <foo>bar<a></a>baz</foo>
+  JSON: {"foo": {"#text": "barbaz", "a": ""}}
+```
+

news/fx-feature-255.md

@@ -0,0 +1,7 @@
+Use `json` and `json_array` as default types for dict and list literals.
+
+This is now a valid config and creates `json` and `json_array` objects:
+```
+my_json_object = {"foo": "bar"};
+my_json_array = ["foo", "bar"];
+```

news/fx-feature-269.md

@@ -0,0 +1,5 @@
+Added new filterx control flow controls.
+
+* `drop`: Drops the currently processed message and returns success.
+* `done`: Stops the processing and returns success.
+

news/fx-feature-275.md

@@ -0,0 +1,21 @@
+`unset_empties()`: Added advanced options.
+
+`unset_empties` removes elements from the given dictionary or list that match
+the empties set. If the `recursive` argument is provided, the function will
+process nested dictionaries as well. The `replacement` argument allows
+replacing target elements with a specified object, and the targets
+argument customizes which elements are removed or replaced, overriding
+the default empties set.
+
+* Optional named arguments:
+  * recursive: Enables recursive processing of nested dictionaries. default: `true`
+  * ignorecase: Enables case-insensitive matching. default: `true`
+  * replacement: Specifies an object to replace target elements instead of removing them.
+    default: nothing (remove)
+  * targets: A list of elements to identify for removal or replacement, clearing the default empty set.
+    default: `["", null, [], {}]`
+
+Example usage:
+```
+unset_empties(js1, targets=["foo", "bar", null, "", [], {}], ignorecase=false, replacement="N/A", recursive=false);
+```

news/fx-feature-282.md

@@ -0,0 +1,8 @@
+`parse_windows_eventlog_xml()`: Added a new function to parse Windows EventLog XMLs.
+
+This parser is really similar to `parse_xml()` with
+a couple of small differences:
+
+1. There is a quick schema validation.
+2. The `Event`->`EventData` field automatically handles named `Data` elements.
+

news/fx-feature-283.md

@@ -0,0 +1,2 @@
+`datetime`: 0 valued `datetime` objects are now falsy.
+

news/fx-feature-284.md

@@ -0,0 +1,2 @@
+`datetime`: `datetime` objects can now be cased to `integer` and `double`.
+

news/fx-feature-287-1.md

@@ -0,0 +1,2 @@
+Declared variables now can be set with dict and list literals.
+

news/fx-feature-297.md

@@ -0,0 +1,18 @@
+`startswith()`, `endswith()`, `includes()`: Added string matching functions.
+
+* First argument is the string that is being matched.
+* Second argument is either a single substring or a list of substrings.
+* Optionally the `ignorecase` argument can be set to configure case sensitivity
+  * default: `false`
+
+Example usage:
+```
+startswith(string, prefix, ignorecase=false);
+startswith(string, [prefix_1, prefix_2], ignorecase=true);
+
+endswith(string, suffix, ignorecase=false);
+endswith(string, [suffix_1, suffix_2], ignorecase=true);
+
+includes(string, substring, ignorecase=false);
+includes(string, [substring_1, substring_2], ignorecase=true);
+```

news/fx-feature-324.md

@@ -0,0 +1,11 @@
+`parse_cef()`, `parse_leef()`: Added CEF and LEEF parsers.
+
+* The first argument is the raw message.
+* Optionally `pair_separator` and `value_separator` arguments
+  can be set to override the respective extension parsing behavior.
+
+Example usage:
+```
+my_structured_leef = parse_leef(leef_message);
+my_structured_cef = parse_cef(cef_message);
+```