axosyslog-4.3.1

4.3.1

This is the combination of the news entries of 4.3.0 and 4.3.1. 4.3.1 hotfixed
a python-parser() related crash and a metrics related memory leak. It also
added Ubuntu 23.04 and Debian 12 support for APT packages and the opensearch()
destination.

Read Axoflow's blog post for more details.

Highlights

parallelize() support for pipelines

syslog-ng has traditionally performed processing of log messages arriving
from a single connection sequentially. This was done to ensure message ordering
as well as most efficient use of CPU on a per message basis. This mode of
operation is performing well as long as we have a relatively large number
of parallel connections, in which case syslog-ng would use all the CPU cores
available in the system.

In case only a small number of connections deliver a large number of
messages, this behaviour may become a bottleneck.

With the new parallelization feature, syslog-ng gained the ability to
re-partition a stream of incoming messages into a set of partitions, each of
which is to be processed by multiple threads in parallel. This does away
with ordering guarantees and adds an extra per-message overhead. In exchange
it will be able to scale the incoming load to all CPUs in the system, even
if coming from a single, chatty sender.

To enable this mode of execution, use the new parallelize() element in your
log path:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4));

  # from this part on, messages are processed in parallel even if
  # messages are originally coming from a single connection

  parser { ... };
  destination { ... };
};

The config above will take all messages emitted by the tcp() source and push
the work to 4 parallel threads of execution, regardless of how many
connections were in use to deliver the stream of messages to the tcp()
driver.

parallelize() uses round-robin to allocate messages to partitions by default.
You can however retain ordering for a subset of messages with the
partition-key() option.

You can use partition-key() to specify a message template. Messages that
expand to the same value are guaranteed to be mapped to the same partition.

For example:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4) partition-key("$HOST"));

  # from this part on, messages are processed in parallel if their
  # $HOST value differs. Messages with the same $HOST will be mapped
  # to the same partition and are processed sequentially.

  parser { ... };
  destination { ... };
};

NOTE: parallelize() requires a patched version of libivykis that contains
this PR buytenh/ivykis#25. syslog-ng source
releases bundle this version of ivykis in their source trees, so if you are
building from source, be sure to use the internal version
(--with-ivykis=internal). You can also use Axoflow's cloud native container
image for syslog-ng, named AxoSyslog
(https://github.com/axoflow/axosyslog-docker) which also incorporates this
change.

(#3966)

Receiving and sending OpenTelemetry (OTLP) messages

The opentelemetry() source, parser and destination are now available to receive, parse and send OTLP/gRPC
messages.

syslog-ng accepts logs, metrics and traces.

The incoming fields are not available through syslog-ng log message name-value pairs for the user by default.
This is useful for forwarding functionality (the opentelemetry() destination can access and format them).
If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields
with some limitations.

The behavior of the opentelemetry() parser is the following:

The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type
(possible values: log, metric and span). The resource info is mapped to .otel.resource.<...>
(e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info
is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...).

The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...).

The fields of metrics are mapped to .otel.metric.<...> (e.g. .otel.metric.name, .otel.metric.unit, ...),
the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram,
exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...>
(e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...).

The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...).

repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano).

The mapping of AnyValue type fields is limited.
string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type
(e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types
are stored serialized with protobuf type. protobuf and bytes types are not directly available for the
user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed
to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64
encode the bytes content).

Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts().
tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values:
required-trusted, required-untrusted, optional-trusted and optional-untrusted) options.
ALTS is a simple to use authentication, only available within Google's infrastructure.

The same methods are available in the destination auth() block, with two differences: tls(peer-verify())
is not available, and there is a fourth method, called ADC, which accepts the target-service-account()
option, where a list of service accounts can be configured to match against when authenticating the server.

Example configs:

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(12345)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

log otel_to_non_otel_insecure {
  source {
    opentelemetry(
      port(12345)
    );
  };

  parser {
    opentelemetry();
  };

  destination {
    network(
      "my-network-server"
      port(12345)
      template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n")
    );
  };
};

log non_otel_to_otel_tls {
  source {
    network(
      port(12346)
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
};

(#4523)
(#4510)

Sending messages to CrowdStrike Falcon LogScale (Humio)

The logscale() destination feeds LogScale via the Ingest API.

Minimal config:

destination d_logscale {
  logscale(
    token("my-token")
  );
};

Additional options include:

• url()
• rawstring()
• timestamp()
• timezone()
• attributes()
• extra-headers()
• content-type()

(#4472)

Features

• afmongodb: Bulk MongoDB insert is added via the following options

  • bulk (yes/no) turns on/off bulk insert usage, no forces the old behavior (each log is inserted one by one into the MongoDB)
  • bulk_unordered (yes/no) turns on/off unordered MongoDB bulk operations
  • bulk_bypass_validation (yes/no) turns on/off MongoDB bulk operations validation
  • write_concern (unacked/acked/majority/n > 0) sets write concern mode of the MongoDB operations, both bulk and single

  NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range.
  (#4483)

• sql: Added 2 new options

  • quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers)
    **N...

axosyslog-4.3.0

4.3.0

Read Axoflow's blog post for more details.

Highlights

parallelize() support for pipelines

syslog-ng has traditionally performed processing of log messages arriving
from a single connection sequentially. This was done to ensure message ordering
as well as most efficient use of CPU on a per message basis. This mode of
operation is performing well as long as we have a relatively large number
of parallel connections, in which case syslog-ng would use all the CPU cores
available in the system.

In case only a small number of connections deliver a large number of
messages, this behaviour may become a bottleneck.

With the new parallelization feature, syslog-ng gained the ability to
re-partition a stream of incoming messages into a set of partitions, each of
which is to be processed by multiple threads in parallel. This does away
with ordering guarantees and adds an extra per-message overhead. In exchange
it will be able to scale the incoming load to all CPUs in the system, even
if coming from a single, chatty sender.

To enable this mode of execution, use the new parallelize() element in your
log path:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4));

  # from this part on, messages are processed in parallel even if
  # messages are originally coming from a single connection

  parser { ... };
  destination { ... };
};

The config above will take all messages emitted by the tcp() source and push
the work to 4 parallel threads of execution, regardless of how many
connections were in use to deliver the stream of messages to the tcp()
driver.

parallelize() uses round-robin to allocate messages to partitions by default.
You can however retain ordering for a subset of messages with the
partition-key() option.

You can use partition-key() to specify a message template. Messages that
expand to the same value are guaranteed to be mapped to the same partition.

For example:

log {
  source {
    tcp(
      port(2000)
      log-iw-size(10M) max-connections(10) log-fetch-limit(100000)
    );
  };
  parallelize(partitions(4) partition-key("$HOST"));

  # from this part on, messages are processed in parallel if their
  # $HOST value differs. Messages with the same $HOST will be mapped
  # to the same partition and are processed sequentially.

  parser { ... };
  destination { ... };
};

NOTE: parallelize() requires a patched version of libivykis that contains
this PR buytenh/ivykis#25. syslog-ng source
releases bundle this version of ivykis in their source trees, so if you are
building from source, be sure to use the internal version
(--with-ivykis=internal). You can also use Axoflow's cloud native container
image for syslog-ng, named AxoSyslog
(https://github.com/axoflow/axosyslog-docker) which also incorporates this
change.

(#3966)

Receiving and sending OpenTelemetry (OTLP) messages

The opentelemetry() source, parser and destination are now available to receive, parse and send OTLP/gRPC
messages.

syslog-ng accepts logs, metrics and traces.

The incoming fields are not available through syslog-ng log message name-value pairs for the user by default.
This is useful for forwarding functionality (the opentelemetry() destination can access and format them).
If such functionality is required, you can configure the opentelemetry() parser, which maps all the fields
with some limitations.

The behavior of the opentelemetry() parser is the following:

The name-value pairs always start with .otel. prefix. The type of the message is stored in .otel.type
(possible values: log, metric and span). The resource info is mapped to .otel.resource.<...>
(e.g.: .otel.resource.dropped_attributes_count, .otel.resource.schema_url ...), the scope info
is mapped to .otel.scope.<...> (e.g.: .otel.scope.name, .otel.scope.schema_url, ...).

The fields of log records are mapped to .otel.log.<...> (e.g. .otel.log.body, .otel.log.severity_text, ...).

The fields of metrics are mapped to .otel.metric.<...> (e.g. .otel.metric.name, .otel.metric.unit, ...),
the type of the metric is mapped to .otel.metric.data.type (possible values: gauge, sum, histogram,
exponential_histogram, summary) with the actual data mapped to .otel.metric.data.<type>.<...>
(e.g.: .otel.metric.data.gauge.data_points.0.time_unix_nano, ...).

The fields of traces are mapped to .otel.span.<...> (e.g. .otel.span.name, .otel.span.trace_state, ...).

repeated fields are given an index (e.g. .otel.span.events.5.time_unix_nano).

The mapping of AnyValue type fields is limited.
string, bool, int64, double and bytes values are mapped with the respective syslog-ng name-value type
(e.g. .otel.resource.attributes.string_key => string_value), however ArrayValue and KeyValueList types
are stored serialized with protobuf type. protobuf and bytes types are not directly available for the
user, unless an explicit type cast is added (e.g. "bytes(${.otel.log.span_id})") or --include-bytes is passed
to name-value iterating template functions (e.g. $(format-json .otel.* --include-bytes), which will base64
encode the bytes content).

Three authentication methods are available in the source auth() block: insecure() (default), tls() and alts().
tls() accepts the key-file(), cert-file(), ca-file() and peer-verify() (possible values:
required-trusted, required-untrusted, optional-trusted and optional-untrusted) options.
ALTS is a simple to use authentication, only available within Google's infrastructure.

The same methods are available in the destination auth() block, with two differences: tls(peer-verify())
is not available, and there is a fourth method, called ADC, which accepts the target-service-account()
option, where a list of service accounts can be configured to match against when authenticating the server.

Example configs:

log otel_forward_mode_alts {
  source {
    opentelemetry(
      port(12345)
      auth(alts())
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12345")
      auth(alts())
    );
  };
};

log otel_to_non_otel_insecure {
  source {
    opentelemetry(
      port(12345)
    );
  };

  parser {
    opentelemetry();
  };

  destination {
    network(
      "my-network-server"
      port(12345)
      template("$(format-json .otel.* --shift-levels 1 --include-bytes)\n")
    );
  };
};

log non_otel_to_otel_tls {
  source {
    network(
      port(12346)
    );
  };

  destination {
    opentelemetry(
      url("my-otel-server:12346")
      auth(
        tls(
          ca-file("/path/to/ca.pem")
          key-file("/path/to/key.pem")
          cert-file("/path/to/cert.pem")
        )
      )
    );
  };
};

(#4523)
(#4510)

Sending messages to CrowdStrike Falcon LogScale (Humio)

The logscale() destination feeds LogScale via the Ingest API.

Minimal config:

destination d_logscale {
  logscale(
    token("my-token")
  );
};

Additional options include:

• url()
• rawstring()
• timestamp()
• timezone()
• attributes()
• extra-headers()
• content-type()

(#4472)

Features

• afmongodb: Bulk MongoDB insert is added via the following options

  • bulk (yes/no) turns on/off bulk insert usage, no forces the old behavior (each log is inserted one by one into the MongoDB)
  • bulk_unordered (yes/no) turns on/off unordered MongoDB bulk operations
  • bulk_bypass_validation (yes/no) turns on/off MongoDB bulk operations validation
  • write_concern (unacked/acked/majority/n > 0) sets write concern mode of the MongoDB operations, both bulk and single

  NOTE: Bulk sending is only efficient if the used collection is constant (e.g. not using templates) or the used template does not lead to too many collections switching within a reasonable time range.
  (#4483)

• sql: Added 2 new options

  • quote_char to aid custom quoting for table and index names (e.g. MySQL needs sometimes this for certain identifiers)
    NOTE: Using a back-tick character needs a special formatting as syslog-ng uses it for configuration parameter names, so for that use: quote_char("``") (double back-tick)
  • dbi_driver_dir to define an optional DBI driver location for DBD initializati...

axosyslog-4.2.0

4.2.0

Read Axoflow's blog post for more details.

Highlights

Sending messages to Splunk HEC

The splunk-hec-event() destination feeds Splunk via the HEC events API.

Minimal config:

destination d_splunk_hec_event {
  splunk-hec-event(
    url("https://localhost:8088")
    token("70b6ae71-76b3-4c38-9597-0c5b37ad9630")
  );
};

Additional options include:

• event()
• index()
• source()
• sourcetype()
• host()
• time()
• default-index()
• default-source()
• default-sourcetype()
• fields()
• extra-headers()
• extra-queries()
• content-type()

The splunk-hec-raw() destination feeds Splunk via the HEC raw API.

Minimal config:

destination d_splunk_hec_raw {
  splunk-hec-raw(
    url("https://localhost:8088")
    token("70b6ae71-76b3-4c38-9597-0c5b37ad9630")
    channel("05ed4617-f186-4ccd-b4e7-08847094c8fd")
  );
};

(#4462)

Smart multi-line for recognizing backtraces

multi-line-mode(smart):
With this multi-line mode, the inherently multi-line data backtrace format is
recognized even if they span multiple lines in the input and are converted
to a single log message for easier analysis. Backtraces for the following
programming languages are recognized : Python, Java, JavaScript, PHP, Go,
Ruby and Dart.

The regular expressions to recognize these programming languages are
specified by an external file called
/usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on
configure arguments), in a format that is described in that file.

group-lines() parser: this new parser correlates multi-line messages
received as separate, but subsequent lines into a single log message.
Received messages are first collected into streams related messages (using
key()), then collected into correlation contexts up to timeout() seconds.
The identification of multi-line messages are then performed on these
message contexts within the time period.

  group-lines(key("$FILE_NAME")
              multi-line-mode("smart")
        template("$MESSAGE")
        timeout(10)
        line-separator("\n")
  );

(#4225)

HYPR Audit Trail source

hypr-audit-trail() & hypr-app-audit-trail() source drivers are now
available to monitor the audit trails for HYPR applications.

See the README.md file in the driver's directory to see usage information.

(#4175)

ebpf() plugin and reuseport packet randomizer

A new ebpf() plugin was added as a framework to leverage the kernel's eBPF
infrastructure to improve performance and scalability of syslog-ng.

Example:

source s_udp {
        udp(so-reuseport(yes) port(2000) persist-name("udp1")
                ebpf(reuseport(sockets(4)))
        );
        udp(so-reuseport(yes) port(2000) persist-name("udp2"));
        udp(so-reuseport(yes) port(2000) persist-name("udp3"));
        udp(so-reuseport(yes) port(2000) persist-name("udp4"));
};

NOTE: The ebpf() plugin is considered advanced usage so its compilation is
disabled by default. Please don't use it unless all other avenues of
configuration solutions are already tried. You will need a special
toolchain and a recent kernel version to compile and run eBPF programs.

(#4365)

Features

• network source: During a TLS handshake, syslog-ng now automatically sets the
  certificate_authorities field of the certificate request based on the ca-file()
  and ca-dir() options. The pkcs12-file() option already had this feature.
  (#4412)

• metrics-probe(): Added level() option to set the stats level of the generated metrics.
  (#4453)

• metrics-probe(): Added increment() option.

  Users can now set a template, which resolves to a number that modifies
  the increment of the counter. If not set, the increment is 1.
  (#4447)

• python: Added support for typed custom options.

  This applies for python source, python-fetcher source, python destination,
  python parser and python-http-header inner destination.

  Example config:

  python(
    class("TestClass")
    options(
      "string_option" => "example_string"
      "bool_option" => True  # supported values are: True, False, yes, no
      "integer_option" => 123456789
      "double_option" => 123.456789
      "string_list_option" => ["string1", "string2", "string3"]
      "template_option" => LogTemplate("${example_template}")
    )
  );
  

  Breaking change! Previously values were converted to strings if possible, now they are passed
  to the python class with their real type. Make sure to follow up these changes
  in your python code!
  (#4354)

• mongodb destination: Added support for list, JSON and null types.
  (#4437)

• add-contextual-data(): significantly reduce memory usage for large CSV
  files.
  (#4444)

• python(): new LogMessage methods for querying as string and with default values

  • get(key[, default])
    Return the value for key if key exists, else default. If default is
    not given, it defaults to None, so that this method never raises a
    KeyError.

  • get_as_str(key, default=None, encoding='utf-8', errors='strict', repr='internal'):
    Return the string value for key if key exists, else default.
    If default is not given, it defaults to None, so that this method never
    raises a KeyError.

    The string value is decoded using the codec registered for encoding.
    errors may be given to set the desired error handling scheme.

    Note that currently repr='internal' is the only available representation.
    We may implement another more Pythonic representation in the future, so please
    specify the repr argument explicitly if you want to avoid future
    representation changes in your code.
    (#4410)

• kubernetes() source: Added support for json-file logging driver format.
  (#4419)

• The new $RAWMSG_SIZE hard macro can be used to query the original size of the
  incoming message in bytes.

  This information may not be available for all source drivers.
  (#4440)

• syslog-ng configuration identifier

  A new syslog-ng configuration keyword has been added, which allows specifying a config identifier. For example:

  @config-id: cfg-20230404-13-g02b0850fc
  

  This keyword can be used for config identification in managed environments, where syslog-ng instances and their
  configuration are deployed/generated automatically.

  syslog-ng-ctl config --id can be used to query the active configuration ID and the SHA256 hash of the full
  "preprocessed" syslog-ng configuration. For example:

  $ syslog-ng-ctl config --id
  cfg-20230404-13-g02b0850fc (08ddecfa52a3443b29d5d5aa3e5114e48dd465e195598062da9f5fc5a45d8a83)
  

  (#4420)

• syslog-ng: add --config-id command line option

  Similarly to --syntax-only, this command line option parses the configuration
  and then prints its ID before exiting.

  It can be used to query the ID of the current configuration persisted on
  disk.
  (#4435)

• Health metrics and syslog-ng-ctl healthcheck

  A new syslog-ng-ctl command has been introduced, which can be used to query a healthcheck status from syslog-ng.
  Currently, only 2 basic health values are reported.

  syslog-ng-ctl healthcheck --timeout <seconds> can be specified to use it as a boolean healthy/unhealthy check.

  Health checks are also published as periodically updated metrics.
  The frequency of these checks can be configured with the stats(healthcheck-freq()) option.
  The default is 5 minutes.
  (#4362)

• $(format-json) and template functions which support value-pairs
  expressions: new key transformations upper() and lower() have been added to
  translate the caps of keys while formatting the output template. For
  example:

  template("$(format-json test.* --upper)\n")
  

  Would convert all keys to uppercase. Only supports US ASCII.
  (#4452)

• python(), python-fetcher() sources: Added a mapping for the flags() option.

  The state of the flags() option is mapped to the self.flags variable, which is
  a Dict[str, bool], for example:

  {
      'parse': True,
      'check-hostname': False,
      'syslog-protocol': True,
      'assume-utf8': False,
      'validate-utf8': False,
      'sanitize-utf8': False,
      'multi-line': True,
      'store-legacy-msghdr': True,
      'store...

axosyslog-4.1.1

4.1.1

This is the combination of the news entries of 4.1.0 and 4.1.1.
4.1.1 hotfixed a grouping-by() and db-parser() related crash.

Highlights

PROXY protocol v2 support (#4211)

We've added support for PROXY protocol v2 (transport(proxied-tcp)), a protocol
used by network load balancers, such as Amazon Elastic Load Balancer and
HAProxy, to carry original source/destination address information, as described
in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Metrics revised

Prometheus metric format (#4325)

A new metric system has been introduced to syslog-ng, where metrics are
identified by names and partitioned by labels, which is similar to the
Prometheus data model.

The syslog-ng-ctl stats prometheus command can be used to query syslog-ng
metrics in a format that conforms to the Prometheus text-based exposition
format.

syslog-ng-ctl stats prometheus --with-legacy-metrics displays legacy metrics
as well. Legacy metrics do not follow Prometheus' metric and label conventions.

Classification (metadata-based metrics) (#4318)

metrics-probe(), a new parser has also been added, which counts messages
passing through based on the metadata of each message. The parser creates
labeled metrics based on the fields of the message.

Both the key and labels can be set in the config, the values of the labels can
be templated. E.g.:

parser p_metrics_probe {
  metrics-probe(
    key("custom_key")  # adds "syslogng_" prefix => "syslogng_custom_key"
    labels(
      "custom_label_name_1" => "foobar"
      "custom_label_name_2" => "${.custom.field}"
    )
  );
};

With this config, it creates counters like these:

syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3

The minimal config creates counters with the key
syslogng_classified_events_total and labels app, host, program and
source. E.g.:

parser p_metrics_probe {
  metrics-probe();
};

With this config, it creates counters like these:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

Named log paths (path ingress/egress metrics) (#4344)

It is also possible to create named log paths, for example:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Each named log path counts its ingress and egress messages:

syslogng_log_path_ingress{id="top-level"} 114
syslogng_log_path_ingress{id="inner-1"} 114
syslogng_log_path_ingress{id="inner-2"} 114
syslogng_log_path_egress{id="top-level"} 103
syslogng_log_path_egress{id="inner-1"} 62
syslogng_log_path_egress{id="inner-2"} 41

Note that the egress statistics only count the messages which have been have not
been filtered out from the related log path, it does care about whether there
are any destinations in it or that any destination delivers or drops the
message.

The above three features are experimental; the output of stats prometheus
(names, labels, etc.) and the metrics created by metrics-probe() and named log
paths may change in the next 2-3 releases.

Features

• $(format-date): add a new template function to format time and date values

  $(format-date [options] format-string [timestamp])

  $(format-date) takes a timestamp in the DATETIME representation and
  formats it according to an strftime() format string. The DATETIME
  representation in syslog-ng is a UNIX timestamp formatted as a decimal
  number, with an optional fractional part, where the seconds and the
  fraction of seconds are separated by a dot.

  If the timestamp argument is missing, the timestamp of the message is
  used.

  Options:
  --time-zone <TZstring> -- override timezone of the original timestamp
  (#4202)

• syslog-parser() and all syslog related sources: accept unquoted RFC5424
  SD-PARAM-VALUEs instead of rejecting them with a parse error.

  sdata-parser(): this new parser allows you to parse an RFC5424 style
  structured data string. It can be used to parse this relatively complex
  format separately.
  (#4281)

• system() source: the system() source was changed on systemd platforms to
  fetch journal messages that relate to the current boot only (e.g. similar
  to journalctl -fb) and to ignore messages generated in previous boots,
  even if those messages were succesfully stored in the journal and were not
  picked up by syslog-ng. This change was implemented as the journald access
  APIs work incorrectly if time goes backwards across reboots, which is an
  increasingly frequent event in virtualized environments and on systems that
  lack an RTC. If you want to retain the old behaviour, please bypass the
  system() source and use systemd-journal() directly, where this option
  can be customized. The change is not tied to @version as we deemed the new
  behaviour fixing an actual bug. For more information consult #2836.

  systemd-journald() source: add match-boot() and matches() options to
  allow you to constrain the collection of journal records to a subset of what
  is in the journal. match-boot() is a yes/no value that allows you to fetch
  messages that only relate to the current boot. matches() allows you to
  specify one or more filters on journal fields.

  Examples:

  source s_journal_current_boot_only {
    systemd-source(match-boot(yes));
  };
  
  source s_journal_systemd_only {
    systemd-source(matches(
      "_COMM" => "systemd"
      )
    );
  };
  

  (#4245)

• date-parser(): add value() parameter to instruct date-parser() to store
  the resulting timestamp in a name-value pair, instead of changing the
  timestamp value of the LogMessage.

  datetime type representation: typed values in syslog-ng are represented as
  strings when stored as a part of a log message. syslog-ng simply remembers
  the type it was stored as. Whenever the value is used as a specific type in
  a type-aware context where we need the value of the specific type, an
  automatic string parsing takes place. This parsing happens for instance
  whenever syslog-ng stores a datetime value in MongoDB or when
  $(format-date) template function takes a name-value pair as parameter.
  The datetime() type has stored its value as the number of milliseconds since
  the epoch (1970-01-01 00:00:00 GMT). This has now been enhanced by making
  it possible to store timestamps up to nanosecond resolutions along with an
  optional timezone offset.

  $(format-date): when applied to name-value pairs with the datetime type,
  use the timezone offset if one is available.
  (#4319)

• stats: Added syslog-stats() global stats() group option.

  E.g.:

  options {
    stats(
      syslog-stats(no);
    );
  };
  

  It changes the behavior of counting messages based on different syslog-proto fields,
  like SEVERITY, FACILITY, HOST, etc...

  Possible values are:

  • yes => force enable
  • no => force disable
  • auto => let stats(level()) decide (old behavior)
    (#4337)

• kubernetes source: Added key-delimiter() option.

  Some metadata fields can contain .-s in their name. This does not work
  with syslog-ng-s macros, which by default use . as a delimiter. The added
  key-delimiter() option changes this behavior by storing the parsed
  metadata fields with a custom delimiter. In order to reach the fields, the
  accessor side has to use the new delimiter format, e.g. --key-delimiter
  option in $(format-json).
  (#4213)

Bugfixes

• Fix conditional evaluation with a dangling filter

  We've fixed a bug that caused conditional evaluation (if/else/elif) and certain logpath flags (final, fallback)
  to occasionally malfunction. The issue only happened in certain logpath constructs; examples can be found in the
  PR description.
  (#4058)

• python: Fixed a bug, where PYTHONPATH was ignored with python3.11.
  (#4298)

• disk-buffer: Fixed disk-queue file becoming corrupt when changing disk-buf-size().

  syslog-ng now continues with the originally set disk-buf-size().
  Note that changing the disk-buf-size() of an existing disk-queue was never supported,
  but could cause errors, which are fixed now.
  (#4308)

• dqtool: fix dqtool assign
  ([#4355](https://github.com/sys...

axosyslog-4.1.0

4.1.0

Highlights

PROXY protocol v2 support (#4211)

We've added support for PROXY protocol v2 (transport(proxied-tcp)), a protocol
used by network load balancers, such as Amazon Elastic Load Balancer and
HAProxy, to carry original source/destination address information, as described
in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

Metrics revised

Prometheus metric format (#4325)

A new metric system has been introduced to syslog-ng, where metrics are
identified by names and partitioned by labels, which is similar to the
Prometheus data model.

The syslog-ng-ctl stats prometheus command can be used to query syslog-ng
metrics in a format that conforms to the Prometheus text-based exposition
format.

syslog-ng-ctl stats prometheus --with-legacy-metrics displays legacy metrics
as well. Legacy metrics do not follow Prometheus' metric and label conventions.

Classification (metadata-based metrics) (#4318)

metrics-probe(), a new parser has also been added, which counts messages
passing through based on the metadata of each message. The parser creates
labeled metrics based on the fields of the message.

Both the key and labels can be set in the config, the values of the labels can
be templated. E.g.:

parser p_metrics_probe {
  metrics-probe(
    key("custom_key")  # adds "syslogng_" prefix => "syslogng_custom_key"
    labels(
      "custom_label_name_1" => "foobar"
      "custom_label_name_2" => "${.custom.field}"
    )
  );
};

With this config, it creates counters like these:

syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3

The minimal config creates counters with the key
syslogng_classified_events_total and labels app, host, program and
source. E.g.:

parser p_metrics_probe {
  metrics-probe();
};

With this config, it creates counters like these:

syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1

Named log paths (path ingress/egress metrics) (#4344)

It is also possible to create named log paths, for example:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Each named log path counts its ingress and egress messages:

syslogng_log_path_ingress{id="top-level"} 114
syslogng_log_path_ingress{id="inner-1"} 114
syslogng_log_path_ingress{id="inner-2"} 114
syslogng_log_path_egress{id="top-level"} 103
syslogng_log_path_egress{id="inner-1"} 62
syslogng_log_path_egress{id="inner-2"} 41

Note that the egress statistics only count the messages which have been have not
been filtered out from the related log path, it does care about whether there
are any destinations in it or that any destination delivers or drops the
message.

The above three features are experimental; the output of stats prometheus
(names, labels, etc.) and the metrics created by metrics-probe() and named log
paths may change in the next 2-3 releases.

Features

• $(format-date): add a new template function to format time and date values

  $(format-date [options] format-string [timestamp])

  $(format-date) takes a timestamp in the DATETIME representation and
  formats it according to an strftime() format string. The DATETIME
  representation in syslog-ng is a UNIX timestamp formatted as a decimal
  number, with an optional fractional part, where the seconds and the
  fraction of seconds are separated by a dot.

  If the timestamp argument is missing, the timestamp of the message is
  used.

  Options:
  --time-zone <TZstring> -- override timezone of the original timestamp
  (#4202)

• syslog-parser() and all syslog related sources: accept unquoted RFC5424
  SD-PARAM-VALUEs instead of rejecting them with a parse error.

  sdata-parser(): this new parser allows you to parse an RFC5424 style
  structured data string. It can be used to parse this relatively complex
  format separately.
  (#4281)

• system() source: the system() source was changed on systemd platforms to
  fetch journal messages that relate to the current boot only (e.g. similar
  to journalctl -fb) and to ignore messages generated in previous boots,
  even if those messages were succesfully stored in the journal and were not
  picked up by syslog-ng. This change was implemented as the journald access
  APIs work incorrectly if time goes backwards across reboots, which is an
  increasingly frequent event in virtualized environments and on systems that
  lack an RTC. If you want to retain the old behaviour, please bypass the
  system() source and use systemd-journal() directly, where this option
  can be customized. The change is not tied to @version as we deemed the new
  behaviour fixing an actual bug. For more information consult #2836.

  systemd-journald() source: add match-boot() and matches() options to
  allow you to constrain the collection of journal records to a subset of what
  is in the journal. match-boot() is a yes/no value that allows you to fetch
  messages that only relate to the current boot. matches() allows you to
  specify one or more filters on journal fields.

  Examples:

  source s_journal_current_boot_only {
    systemd-source(match-boot(yes));
  };
  
  source s_journal_systemd_only {
    systemd-source(matches(
      "_COMM" => "systemd"
      )
    );
  };
  

  (#4245)

• date-parser(): add value() parameter to instruct date-parser() to store
  the resulting timestamp in a name-value pair, instead of changing the
  timestamp value of the LogMessage.

  datetime type representation: typed values in syslog-ng are represented as
  strings when stored as a part of a log message. syslog-ng simply remembers
  the type it was stored as. Whenever the value is used as a specific type in
  a type-aware context where we need the value of the specific type, an
  automatic string parsing takes place. This parsing happens for instance
  whenever syslog-ng stores a datetime value in MongoDB or when
  $(format-date) template function takes a name-value pair as parameter.
  The datetime() type has stored its value as the number of milliseconds since
  the epoch (1970-01-01 00:00:00 GMT). This has now been enhanced by making
  it possible to store timestamps up to nanosecond resolutions along with an
  optional timezone offset.

  $(format-date): when applied to name-value pairs with the datetime type,
  use the timezone offset if one is available.
  (#4319)

• stats: Added syslog-stats() global stats() group option.

  E.g.:

  options {
    stats(
      syslog-stats(no);
    );
  };
  

  It changes the behavior of counting messages based on different syslog-proto fields,
  like SEVERITY, FACILITY, HOST, etc...

  Possible values are:

  • yes => force enable
  • no => force disable
  • auto => let stats(level()) decide (old behavior)
    (#4337)

• kubernetes source: Added key-delimiter() option.

  Some metadata fields can contain .-s in their name. This does not work
  with syslog-ng-s macros, which by default use . as a delimiter. The added
  key-delimiter() option changes this behavior by storing the parsed
  metadata fields with a custom delimiter. In order to reach the fields, the
  accessor side has to use the new delimiter format, e.g. --key-delimiter
  option in $(format-json).
  (#4213)

Bugfixes

• Fix conditional evaluation with a dangling filter

  We've fixed a bug that caused conditional evaluation (if/else/elif) and certain logpath flags (final, fallback)
  to occasionally malfunction. The issue only happened in certain logpath constructs; examples can be found in the
  PR description.
  (#4058)

• python: Fixed a bug, where PYTHONPATH was ignored with python3.11.
  (#4298)

• disk-buffer: Fixed disk-queue file becoming corrupt when changing disk-buf-size().

  syslog-ng now continues with the originally set disk-buf-size().
  Note that changing the disk-buf-size() of an existing disk-queue was never supported,
  but could cause errors, which are fixed now.
  (#4308)

• dqtool: fix dqtool assign
  (#4355)

• example-diskq-source: Fixed failing to read the disk-queue content in some cases.
  ([#4308](ht...

axosyslog-4.0.1

ci: fix multiline output issue with tags

Signed-off-by: László Várady <laszlo.varady@axoflow.com>