v4.46.0

For installation instructions check out the getting started guide.

Added

• New mysql_cdc input supporting change data capture (CDC) from MySQL. (@rockwotj, @le-vlad)
• Field instance_id added to kafka, kafka_franz, ockam_kafka, redpanda, redpanda_common, and redpanda_migrator inputs. (@rockwotj)
• Fields rebalance_timeout, session_timeout and heartbeat_interval added to the kafka_franz, redpanda, redpanda_common, redpanda_migrator and ockam_kafka inputs. (@rockwotj)
• Field avro.preserve_logical_types for processor schema_registry_decode was added to preserve logical types instead of decoding them as their primitive representation. (@rockwotj)
• Processor schema_registry_decode now adds metadata schema_id for the schema's ID in the schema registry. (@rockwotj)
• Field schema_evolution.processors added to snowpipe_streaming to support side effects or enrichment during schema evolution. (@rockwotj)
• Field unchanged_toast_value added to postgres_cdc to control the value substituted for unchanged toast values when a table does not have full replica identity. (@rockwotj)

Fixed

• Fix a snapshot stream consistency issue with postgres_cdc where data could be missed if writes were happening during the snapshot phase. (@rockwotj)
• Fix an issue where @table metadata was quoted for the snapshot phase in postgres_cdc. (@rockwotj)

Changed

• Field avro_raw_json was deprecated in favor of avro.raw_unions for processor schema_registry_decode. (@rockwotj)
• The snowpipe_streaming output now has better error handling for authentication failures when uploading to cloud storage. (@rockwotj)
• Field schema_evolution.new_column_type_mapping for snowpipe_streaming is deprecated and can be replaced with schema_evolution.processors. (@rockwotj)
• Increased the default values for max_message_bytes and broker_write_max_bytes by using IEC units instead of SI units. This better matches defaults in Redpanda and Kafka. (@rockwotj)
• Dropped support for postgres 10 and 11 in postgres_cdc. (@rockwotj)

The full change log can be found here.

v4.46.0-rc3

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Added

• New mysql_cdc input supporting change data capture (CDC) from MySQL. (@rockwotj, @le-vlad)
• Field instance_id added to kafka, kafka_franz, ockam_kafka, redpanda, redpanda_common, and redpanda_migrator inputs. (@rockwotj)
• Fields rebalance_timeout, session_timeout and heartbeat_interval added to the kafka_franz, redpanda, redpanda_common, redpanda_migrator and ockam_kafka inputs. (@rockwotj)
• Field avro.preserve_logical_types for processor schema_registry_decode was added to preserve logical types instead of decoding them as their primitive representation. (@rockwotj)
• Processor schema_registry_decode now adds metadata schema_id for the schema's ID in the schema registry. (@rockwotj)
• Field schema_evolution.processors added to snowpipe_streaming to support side effects or enrichment during schema evolution. (@rockwotj)
• Field unchanged_toast_value added to postgres_cdc to control the value substituted for unchanged toast values when a table does not have full replica identity. (@rockwotj)

Fixed

• Fix a snapshot stream consistency issue with postgres_cdc where data could be missed if writes were happening during the snapshot phase. (@rockwotj)
• Fix an issue where @table metadata was quoted for the snapshot phase in postgres_cdc. (@rockwotj)

Changed

• Field avro_raw_json was deprecated in favor of avro.raw_unions for processor schema_registry_decode. (@rockwotj)
• The snowpipe_streaming output now has better error handling for authentication failures when uploading to cloud storage. (@rockwotj)
• Field schema_evolution.new_column_type_mapping for snowpipe_streaming is deprecated and can be replaced with schema_evolution.processors. (@rockwotj)
• Increased the default values for max_message_bytes and broker_write_max_bytes by using IEC units instead of SI units. This better matches defaults in Redpanda and Kafka. (@rockwotj)
• Dropped support for postgres 10 and 11 in postgres_cdc. (@rockwotj)

The full change log can be found here.

v4.46.0-rc2

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Added

• New mysql_cdc input supporting change data capture (CDC) from MySQL. (@rockwotj, @le-vlad)
• Field instance_id added to kafka, kafka_franz, ockam_kafka, redpanda, redpanda_common, and redpanda_migrator inputs. (@rockwotj)
• Fields rebalance_timeout, session_timeout and heartbeat_interval added to the kafka_franz, redpanda, redpanda_common, redpanda_migrator and ockam_kafka inputs. (@rockwotj)
• Field avro.preserve_logical_types for processor schema_registry_decode was added to preserve logical types instead of decoding them as their primitive representation. (@rockwotj)
• Processor schema_registry_decode now adds metadata schema_id for the schema's ID in the schema registry. (@rockwotj)
• Field schema_evolution.processors added to snowpipe_streaming to support side effects or enrichment during schema evolution. (@rockwotj)

Changed

• Field avro_raw_json was deprecated in favor of avro.raw_unions for processor schema_registry_decode. (@rockwotj)
• The snowpipe_streaming output now has better error handling for authentication failures when uploading to cloud storage. (@rockwotj)
• Field schema_evolution.new_column_type_mapping for snowpipe_streaming is deprecated and can be replaced with schema_evolution.processors. (@rockwotj)
• Increased the default values for max_message_bytes and broker_write_max_bytes by using IEC units instead of SI units. This better matches defaults in Redpanda and Kafka. (@rockwotj)

The full change log can be found here.

v4.46.0-rc1

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Added

• New mysql_cdc input supporting change data capture (CDC) from MySQL. (@rockwotj, @le-vlad)
• Field instance_id added to kafka, kafka_franz, ockam_kafka, redpanda, redpanda_common, and redpanda_migrator inputs. (@rockwotj)
• Fields rebalance_timeout, session_timeout and heartbeat_interval added to the kafka_franz, redpanda, redpanda_common, redpanda_migrator and ockam_kafka inputs. (@rockwotj)
• Field avro.preserve_logical_types for processor schema_registry_decode was added to preserve logical types instead of decoding them as their primitive representation. (@rockwotj)

Changed

• Field avro_raw_json was deprecated in favor of avro.raw_unions for processor schema_registry_decode. (@rockwotj)

Changed

• The snowpipe_streaming output now has better error handling for authentication failures when uploading to cloud storage. (@rockwotj)

The full change log can be found here.

v4.45.1

For installation instructions check out the getting started guide.

Fixed

• Empty files read by input aws_s3 no longer cause spurious errors. (@rockwotj)
• Fixes a SIGSEGV in postgres_cdc when using TOAST values with tables that don't have FULL replica identity. (@rockwotj)

The full change log can be found here.

v4.45.0

For installation instructions check out the getting started guide.

Fixed

• The code and file fields on the javascript processor docs no longer erroneously mention interpolation support. (@mihaitodor)
• The postgres_cdc now correctly handles null values. (@rockwotj)
• The redpanda_migrator output no longer rejects messages if it can't perform schema ID translation. (@mihaitodor)
• The redpanda_migrator input no longer converts the kafka key to string. (@mihaitodor)

Added

• aws_sqs input now has a max_outstanding field to prevent unbounded memory usage. (@rockwotj)
• avro scanner now emits metadata for the Avro schema it used along with the schema fingerprint. (@rockwotj)
• Field content_type added to the amqp_1 output. (@timo102)
• Field fetch_max_wait added to the kafka_franz, ockam_kafka, redpanda, redpanda_common and redpanda_migrator inputs. (@birdayz)
• snowpipe_streaming output now supports interpolating table names. (@rockwotj)
• snowpipe_streaming output now supports interpolating channel names. (@rockwotj)
• snowpipe_streaming output now supports exactly once delivery using offset_token. (@rockwotj)
• ollama_chat processor now supports tool calling. (@rockwotj)
• New ollama_moderation processor which allows using LlamaGuard or ShieldGemma to check if LLM responses are safe. (@rockwotj)
• Field queries added to sql_raw processor and output to support rummong multiple SQL statements transactionally. (@rockwotj)
• New redpanda_migrator_offsets input. (@mihaitodor)
• Fields offset_topic, offset_group, offset_partition, offset_commit_timestamp and offset_metadata added to the redpanda_migrator_offsets output. (@mihaitodor)
• Field topic_lag_refresh_period added to the redpanda and redpanda_common inputs. (@mihaitodor)
• Metric redpanda_lag now emitted by the redpanda and redpanda_common inputs. (@mihaitodor)
• Metadata kafka_lag now emitted by the redpanda and redpanda_common inputs. (@mihaitodor)
• The redpanda_migrator_bundle input and output now set labels for their subcomponents. (@mihaitodor)
• (Benthos) Field label added to the template tests definitions. (@mihaitodor)
• (Benthos) Metadata field label can now be utilized within a template's mapping field to access the label that is associated with the template instantiation in a config. (@mihaitodor)
• (Benthos) bloblang scalar type added to template fields. (@mihaitodor)
• (Benthos) Go API: Method SetOutputBrokerPattern added to the StreamBuilder type. (@mihaitodor)
• (Benthos) New error_source_name, error_source_label and error_source_path bloblang functions. (@mihaitodor)
• (Benthos) Flag --verbose added to the benthos lint and benthos template lint commands. (@mihaitodor)

Changed

• Fix an issue in aws_sqs with refreshing in-flight message leases which could prevent acks from processed. (@rockwotj)
• Fix an issue with postgres_cdc with TOAST values not being propagated with REPLICA IDENTITY FULL. (@rockwotj)
• Fix a initial snapshot streaming consistency issue with postgres_cdc. (@rockwotj)
• Fix bug in sftp input where the last file was not deleted when watcher and delete_on_finish were enabled. (@ooesili)
• Fields batch_size, multi_header, replication_factor, replication_factor_override and output_resource for the redpanda_migrator input are now deprecated. (@mihaitodor)
• Fields kafka_key and max_in_flight for the redpanda_migrator_offsets output are now deprecated. (@mihaitodor)
• Field batching for the redpanda_migrator output is now deprecated. (@mihaitodor)
• The redpanda_migrator input no longer emits tombstone messages. (@mihaitodor)
• (Benthos) The branch processor no longer emits an entry in the log at error level when the child processors throw errors. (@mihaitodor)
• (Benthos) Streams and the StreamBuilder API now use reject by default when no output is specified in the config and stdout isn't registered (for example when the io components are not imported). (@mihaitodor)

The full change log can be found here.

v4.45.0-rc1

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Added

• aws_sqs now has a max_outstanding field to prevent unbounded memory usage. (@rockwotj)
• avro scanner now emits metadata for the Avro schema it used along with the schema fingerprint. (@rockwotj)
• Field content_type added to the amqp_1 output. (@timo102)
• kafka_franz, ockam_kafka, redpanda, redpanda_common, redpanda_migrator now support fetch_max_wait configuration field.
• snowpipe_streaming now supports interpolating table names (@rockwotj)
• snowpipe_streaming now supports interpolating channel names (@rockwotj)
• snowpipe_streaming now supports exactly once delivery using offset_token (@rockwotj)

Fixed

• The code and file fields on the javascript processor docs no longer erroneously mention interpolation support. (@mihaitodor)
• The postgres_cdc now correctly handles null values. (@rockwotj)
• Fix an issue in aws_sqs with refreshing in-flight message leases which could prevent acks from processed. (@rockwotj)

The full change log can be found here.

v4.44.0

For installation instructions check out the getting started guide.

Added

• Go API: New public/license package added to allow custom programmatic instantiations of Redpanda Connect to run enterprise license components. (@Jeffail)

Fixed

• gcp_bigquery output with parquet format no longer returns errors incorrectly. (@rockwotj)
• postgres_cdc input now allows quoted identifiers for the table names. (@mihaitodor, @rockwotj)

The full change log can be found here.

v4.43.1

For installation instructions check out the getting started guide.

Fixed

• Trial Redpanda Enterprise licenses are now considered valid. (@Jeffail)
• The redpanda_migrator_bundle output now skips schema ID translation when translate_schema_ids: false and schema_registry is configured. (@mihaitodor)

The full change log can be found here.

v4.43.0

For installation instructions check out the getting started guide.

Changed

• The pg_stream input has been renamed to postgres_cdc. The old name will continue to function as an alias. (@rockwotj)
• The postgres_cdc input no longer emits mode metadata and instead snapshot reads set operation metadata to be read instead of insert. (@rockwotj)

Fixed

• The redpanda_migrator_bundle output no longer attempts to translate schema IDs when a schema registry is not configured. (@mihaitodor)

The full change log can be found here.