fix(socket sink): gracefully shutdown on reload when stream is terminated #21455

neuronull · 2024-10-08T19:30:27Z

It looks like a PR introduced a regression while making changes to the tcp socket sink to emit internal telemetry in accordance with the component specification.

This introduced a bug where the TcpSink crashes Vector during config reloads, while events are flowing through the topology.

The fix is to restore the behavior of peeking onto the stream before attempting to access the next event.

To test this, just use the socket sink in tcp mode, and watch the vector config. Send events continuously. Change the name of the input to the sink. In the before case, it crashes. After the fix, it does not.

reproducing it (before fix)

RUST_BACKTRACE=1 VECTOR_LOG=info ./target/debug/vector --config ../vector_configs/http_server_socket.yaml --watch-config
2024-10-08T19:17:41.160536Z  INFO vector::app: Log level is enabled. level="info"
2024-10-08T19:17:41.163068Z  INFO vector::config::watcher: Creating configuration file watcher.
2024-10-08T19:17:41.164893Z  INFO vector::config::watcher: Watching configuration files.
2024-10-08T19:17:41.165237Z  INFO vector::app: Loading configs. paths=["../vector_configs/http_server_socket.yaml"]
2024-10-08T19:17:41.177815Z  INFO vector::topology::running: Running healthchecks.
2024-10-08T19:17:41.178421Z  INFO vector: Vector has started. debug="true" version="0.42.0" arch="aarch64" revision=""
2024-10-08T19:17:41.178566Z  INFO source{component_kind="source" component_id=http2 component_type=http}: vector::sources::util::http::prelude: Building HTTP server. address=0.0.0.0:4242
2024-10-08T19:17:41.179324Z  INFO vector::topology::builder: Healthcheck passed.
2024-10-08T19:17:58.679911Z  INFO vector::config::watcher: Configuration file changed.
2024-10-08T19:17:58.683478Z  INFO vector::topology::running: Reloading running topology with new configuration.
thread 'vector-worker' panicked at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/stream-cancel-0.8.2/src/combinator.rs:165:40:
`async fn` resumed after completion
stack backtrace:
   0: rust_begin_unwind
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:652:5
   1: core::panicking::panic_fmt
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panicking.rs:72:14
   2: core::panicking::panic_const::panic_const_async_fn_resumed
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panicking.rs:180:21
   3: <stream_cancel::combinator::Tripwire as core::future::future::Future>::poll::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/stream-cancel-0.8.2/src/combinator.rs:165:40
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/future/future.rs:123:9
   5: <stream_cancel::combinator::Tripwire as core::future::future::Future>::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/stream-cancel-0.8.2/src/combinator.rs:180:9
   6: <stream_cancel::combinator::TakeUntilIf<S,F> as futures_core::stream::Stream>::poll_next
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/stream-cancel-0.8.2/src/combinator.rs:93:45
   7: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
   8: <futures_util::stream::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/map.rs:58:26
   9: <futures_util::stream::stream::flatten::Flatten<St,<St as futures_core::stream::Stream>::Item> as futures_core::stream::Stream>::poll_next
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/flatten.rs:55:44
  10: <futures_util::stream::stream::FlatMap<St,U,F> as futures_core::stream::Stream>::poll_next
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:102:13
  11: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/stream.rs:120:9
  12: <futures_util::stream::stream::map::Map<St,F> as futures_core::stream::Stream>::poll_next
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/map.rs:58:26
  13: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/mod.rs:1638:9
  14: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/next.rs:32:9
  15: <vector::sinks::util::tcp::TcpSink<E> as vector_core::sink::StreamSink<vector_core::event::Event>>::run::{{closure}}
             at ./src/sinks/util/tcp.rs:305:45
  16: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/future/future.rs:123:9
  17: <vector_core::sink::EventStream<T> as vector_core::sink::StreamSink<vector_core::event::array::EventArray>>::run::{{closure}}
             at ./lib/vector-core/src/sink.rs:178:30
  18: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/future/future.rs:123:9
  19: vector_core::sink::VectorSink::run::{{closure}}
             at ./lib/vector-core/src/sink.rs:21:55
  20: vector::topology::builder::Builder::build_sinks::{{closure}}::{{closure}}
             at ./src/topology/builder.rs:619:18
  21: <vector::topology::task::Task as core::future::future::Future>::poll
             at ./src/topology/task.rs:92:9
  22: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::future::future::Future>::poll
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panic/unwind_safe.rs:297:9
  23: <futures_util::future::future::catch_unwind::CatchUnwind<Fut> as core::future::future::Future>::poll::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/catch_unwind.rs:36:42
  24: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panic/unwind_safe.rs:272:9
  25: std::panicking::try::do_call
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:559:40
  26: ___rust_try
  27: std::panicking::try
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:523:19
  28: std::panic::catch_unwind
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panic.rs:149:14
  29: <futures_util::future::future::catch_unwind::CatchUnwind<Fut> as core::future::future::Future>::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/catch_unwind.rs:36:9
  30: vector::topology::handle_errors::{{closure}}
             at ./src/topology/mod.rs:66:10
  31: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  32: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:331:17
  33: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9
  34: tokio::runtime::task::core::Core<T,S>::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:320:13
  35: tokio::runtime::task::harness::poll_future::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:500:19
  36: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panic/unwind_safe.rs:272:9
  37: std::panicking::try::do_call
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:559:40
  38: ___rust_try
  39: std::panicking::try
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:523:19
  40: std::panic::catch_unwind
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panic.rs:149:14
  41: tokio::runtime::task::harness::poll_future
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:488:18
  42: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:209:27
  43: tokio::runtime::task::harness::Harness<T,S>::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:154:15
  44: tokio::runtime::task::raw::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:271:5
  45: tokio::runtime::task::raw::RawTask::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:201:18
  46: tokio::runtime::task::LocalNotified<S>::run
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/mod.rs:436:9
  47: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:598:13
  48: tokio::runtime::coop::with_budget
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  49: tokio::runtime::coop::budget
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  50: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:597:9
  51: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:548:24
  52: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:513:21
  53: tokio::runtime::context::scoped::Scoped<T>::set
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/scoped.rs:40:9
  54: tokio::runtime::context::set_scheduler::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:26
  55: std::thread::local::LocalKey<T>::try_with
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/thread/local.rs:286:12
  56: std::thread::local::LocalKey<T>::with
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/thread/local.rs:262:9
  57: tokio::runtime::context::set_scheduler
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:9
  58: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:508:9
  59: tokio::runtime::context::runtime::enter_runtime
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  60: tokio::runtime::scheduler::multi_thread::worker::run
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:500:5
  61: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:466:45
  62: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/task.rs:42:21
  63: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:331:17
  64: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9
  65: tokio::runtime::task::core::Core<T,S>::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/core.rs:320:13
  66: tokio::runtime::task::harness::poll_future::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:500:19
  67: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panic/unwind_safe.rs:272:9
  68: std::panicking::try::do_call
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:559:40
  69: ___rust_try
  70: std::panicking::try
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:523:19
  71: std::panic::catch_unwind
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panic.rs:149:14
  72: tokio::runtime::task::harness::poll_future
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:488:18
  73: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:209:27
  74: tokio::runtime::task::harness::Harness<T,S>::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/harness.rs:154:15
  75: tokio::runtime::task::raw::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:271:5
  76: tokio::runtime::task::raw::RawTask::poll
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/raw.rs:201:18
  77: tokio::runtime::task::UnownedTask<S>::run
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/task/mod.rs:473:9
  78: tokio::runtime::blocking::pool::Task::run
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:160:9
  79: tokio::runtime::blocking::pool::Inner::run
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:518:17
  80: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/blocking/pool.rs:476:13
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2024-10-08T19:17:58.818471Z ERROR sink{component_kind="sink" component_id=socket2 component_type=socket}: vector::topology: An error occurred that Vector couldn't handle: the task panicked and was aborted.
thread 'main' panicked at src/topology/running.rs:520:54:
called `Result::unwrap()` on an `Err` value: Panicked
stack backtrace:
   0: rust_begin_unwind
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/std/src/panicking.rs:652:5
   1: core::panicking::panic_fmt
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/result.rs:1654:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/result.rs:1077:23
   4: vector::topology::running::RunningTopology::shutdown_diff::{{closure}}
             at ./src/topology/running.rs:520:30
   5: vector::topology::running::RunningTopology::reload_config_and_respawn::{{closure}}
             at ./src/topology/running.rs:244:62
   6: vector::topology::controller::TopologyController::reload::{{closure}}
             at ./src/topology/controller.rs:107:14
   7: vector::app::reload_config_from_result::{{closure}}
             at ./src/app.rs:357:72
   8: vector::app::handle_signal::{{closure}}
             at ./src/app.rs:341:72
   9: vector::app::StartedApplication::main::{{closure}}
             at ./src/app.rs:291:19
  10: vector::app::StartedApplication::run::{{closure}}
             at ./src/app.rs:264:21
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/future/future.rs:123:9
  12: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  13: tokio::runtime::coop::with_budget
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  14: tokio::runtime::coop::budget
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  15: tokio::runtime::park::CachedParkThread::block_on
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  16: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  17: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  18: tokio::runtime::context::runtime::enter_runtime
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  19: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  20: tokio::runtime::runtime::Runtime::block_on_inner
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:363:45
  21: tokio::runtime::runtime::Runtime::block_on
             at /Users/neuronull/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:333:13
  22: vector::app::Application::run
             at ./src/app.rs:154:9
  23: vector::main
             at ./src/main.rs:40:21
  24: core::ops::function::FnOnce::call_once
             at /rustc/129f3b9964af4d4a709d1383930ade12dfe7c081/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

after the fix:

VECTOR_LOG=info ./target/debug/vector --config ../vector_configs/http_server_socket.yaml --watch-config
2024-10-08T19:16:09.774602Z  INFO vector::app: Log level is enabled. level="info"
2024-10-08T19:16:09.777461Z  INFO vector::config::watcher: Creating configuration file watcher.
2024-10-08T19:16:09.779871Z  INFO vector::config::watcher: Watching configuration files.
2024-10-08T19:16:09.780024Z  INFO vector::app: Loading configs. paths=["../vector_configs/http_server_socket.yaml"]
2024-10-08T19:16:09.792364Z  INFO vector::topology::running: Running healthchecks.
2024-10-08T19:16:09.792828Z  INFO vector: Vector has started. debug="true" version="0.42.0" arch="aarch64" revision=""
2024-10-08T19:16:09.793156Z  INFO source{component_kind="source" component_id=http3 component_type=http}: vector::sources::util::http::prelude: Building HTTP server. address=0.0.0.0:4242
2024-10-08T19:16:09.793841Z  INFO vector::topology::builder: Healthcheck passed.
2024-10-08T19:16:27.711462Z  INFO vector::config::watcher: Configuration file changed.
2024-10-08T19:16:27.716493Z  INFO vector::topology::running: Reloading running topology with new configuration.
2024-10-08T19:16:27.722131Z  INFO vector::topology::running: Running healthchecks.
2024-10-08T19:16:27.722741Z  INFO vector::topology::running: New configuration loaded successfully.
2024-10-08T19:16:27.722882Z  INFO source{component_kind="source" component_id=http2 component_type=http}: vector::sources::util::http::prelude: Building HTTP server. address=0.0.0.0:4242
2024-10-08T19:16:27.722941Z  INFO vector: Vector has reloaded. path=[File("../vector_configs/http_server_socket.yaml", None)]
2024-10-08T19:16:27.725322Z  INFO vector::topology::builder: Healthcheck passed.

…ated

src/sinks/util/tcp.rs

pront

Great fix 👍

github-actions · 2024-10-08T23:51:00Z

Regression Detector Results

Run ID: a2138e42-06b3-41c8-b7eb-52311561cb98 Metrics dashboard

Baseline: 5581b24
Comparison: 597ebd1

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	links
➖	file_to_blackhole	egress throughput	-4.25	[-11.15, +2.66]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	splunk_hec_route_s3	ingress throughput	+3.67	[+3.35, +3.99]
➖	http_to_http_acks	ingress throughput	+2.97	[+1.74, +4.20]
➖	syslog_log2metric_humio_metrics	ingress throughput	+2.06	[+1.95, +2.16]
➖	syslog_log2metric_splunk_hec_metrics	ingress throughput	+1.14	[+1.02, +1.26]
➖	datadog_agent_remap_datadog_logs_acks	ingress throughput	+0.95	[+0.76, +1.13]
➖	datadog_agent_remap_blackhole	ingress throughput	+0.77	[+0.67, +0.87]
➖	syslog_regex_logs2metric_ddmetrics	ingress throughput	+0.57	[+0.44, +0.69]
➖	syslog_log2metric_tag_cardinality_limit_blackhole	ingress throughput	+0.54	[+0.45, +0.63]
➖	syslog_splunk_hec_logs	ingress throughput	+0.50	[+0.40, +0.60]
➖	datadog_agent_remap_blackhole_acks	ingress throughput	+0.17	[+0.07, +0.27]
➖	http_to_http_noack	ingress throughput	+0.12	[+0.05, +0.19]
➖	http_to_http_json	ingress throughput	+0.08	[+0.01, +0.14]
➖	splunk_hec_indexer_ack_blackhole	ingress throughput	+0.03	[-0.04, +0.10]
➖	splunk_hec_to_splunk_hec_logs_acks	ingress throughput	+0.01	[-0.09, +0.11]
➖	fluent_elasticsearch	ingress throughput	+0.00	[-0.49, +0.50]
➖	syslog_loki	ingress throughput	-0.01	[-0.08, +0.06]
➖	splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.05	[-0.14, +0.04]
➖	http_to_s3	ingress throughput	-0.19	[-0.46, +0.08]
➖	datadog_agent_remap_datadog_logs	ingress throughput	-0.80	[-1.00, -0.59]
➖	http_elasticsearch	ingress throughput	-1.03	[-1.21, -0.86]
➖	syslog_humio_logs	ingress throughput	-1.27	[-1.40, -1.14]
➖	otlp_grpc_to_blackhole	ingress throughput	-1.58	[-1.69, -1.47]
➖	http_text_to_http_json	ingress throughput	-2.24	[-2.34, -2.14]
➖	socket_to_socket_blackhole	ingress throughput	-2.38	[-2.44, -2.31]
➖	otlp_http_to_blackhole	ingress throughput	-3.03	[-3.16, -2.91]
➖	file_to_blackhole	egress throughput	-4.25	[-11.15, +2.66]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

github-actions · 2024-10-09T16:21:28Z

Regression Detector Results

Run ID: 32419d32-3e15-4f6d-8ba9-0c0341b2926a Metrics dashboard

Baseline: 290b9dd
Comparison: dd30186

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	links
➖	file_to_blackhole	egress throughput	-1.50	[-8.65, +5.66]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	http_to_http_acks	ingress throughput	+2.45	[+1.21, +3.69]
➖	syslog_log2metric_tag_cardinality_limit_blackhole	ingress throughput	+1.98	[+1.89, +2.06]
➖	datadog_agent_remap_datadog_logs_acks	ingress throughput	+1.34	[+1.14, +1.55]
➖	syslog_log2metric_humio_metrics	ingress throughput	+0.97	[+0.84, +1.10]
➖	syslog_regex_logs2metric_ddmetrics	ingress throughput	+0.76	[+0.65, +0.88]
➖	fluent_elasticsearch	ingress throughput	+0.70	[+0.20, +1.20]
➖	syslog_loki	ingress throughput	+0.66	[+0.59, +0.73]
➖	otlp_http_to_blackhole	ingress throughput	+0.53	[+0.41, +0.65]
➖	datadog_agent_remap_blackhole_acks	ingress throughput	+0.29	[+0.17, +0.41]
➖	socket_to_socket_blackhole	ingress throughput	+0.21	[+0.15, +0.27]
➖	http_to_http_noack	ingress throughput	+0.18	[+0.09, +0.27]
➖	http_to_http_json	ingress throughput	+0.13	[+0.04, +0.22]
➖	syslog_splunk_hec_logs	ingress throughput	+0.08	[-0.02, +0.19]
➖	datadog_agent_remap_blackhole	ingress throughput	+0.02	[-0.08, +0.12]
➖	splunk_hec_to_splunk_hec_logs_noack	ingress throughput	+0.01	[-0.08, +0.10]
➖	splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-0.00	[-0.10, +0.10]
➖	splunk_hec_indexer_ack_blackhole	ingress throughput	-0.01	[-0.09, +0.08]
➖	http_text_to_http_json	ingress throughput	-0.18	[-0.31, -0.06]
➖	http_to_s3	ingress throughput	-0.26	[-0.53, +0.02]
➖	syslog_humio_logs	ingress throughput	-0.59	[-0.71, -0.47]
➖	otlp_grpc_to_blackhole	ingress throughput	-0.79	[-0.90, -0.67]
➖	datadog_agent_remap_datadog_logs	ingress throughput	-1.31	[-1.51, -1.12]
➖	file_to_blackhole	egress throughput	-1.50	[-8.65, +5.66]
➖	syslog_log2metric_splunk_hec_metrics	ingress throughput	-1.74	[-1.84, -1.63]
➖	splunk_hec_route_s3	ingress throughput	-1.79	[-2.10, -1.48]
➖	http_elasticsearch	ingress throughput	-2.81	[-2.96, -2.67]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

github-actions · 2024-10-09T17:31:53Z

Regression Detector Results

Run ID: 55fb6e0a-d9d5-483d-8d5b-02a3d848167e Metrics dashboard

Baseline: 775d9de
Comparison: 59b7c9a

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	links
✅	file_to_blackhole	egress throughput	+23.30	[+15.53, +31.06]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
✅	file_to_blackhole	egress throughput	+23.30	[+15.53, +31.06]
➖	syslog_humio_logs	ingress throughput	+2.88	[+2.73, +3.03]
➖	syslog_splunk_hec_logs	ingress throughput	+2.58	[+2.51, +2.65]
➖	otlp_http_to_blackhole	ingress throughput	+2.40	[+2.26, +2.53]
➖	syslog_log2metric_humio_metrics	ingress throughput	+2.35	[+2.20, +2.50]
➖	syslog_log2metric_tag_cardinality_limit_blackhole	ingress throughput	+1.85	[+1.75, +1.94]
➖	splunk_hec_route_s3	ingress throughput	+1.49	[+1.19, +1.80]
➖	http_to_http_acks	ingress throughput	+1.11	[-0.13, +2.34]
➖	http_text_to_http_json	ingress throughput	+0.83	[+0.71, +0.96]
➖	syslog_loki	ingress throughput	+0.65	[+0.55, +0.75]
➖	http_to_s3	ingress throughput	+0.04	[-0.24, +0.33]
➖	http_to_http_json	ingress throughput	+0.02	[-0.01, +0.05]
➖	splunk_hec_to_splunk_hec_logs_acks	ingress throughput	+0.00	[-0.10, +0.11]
➖	http_to_http_noack	ingress throughput	+0.00	[-0.02, +0.02]
➖	splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.00	[-0.10, +0.10]
➖	splunk_hec_indexer_ack_blackhole	ingress throughput	-0.00	[-0.08, +0.07]
➖	socket_to_socket_blackhole	ingress throughput	-0.19	[-0.26, -0.11]
➖	datadog_agent_remap_blackhole	ingress throughput	-0.63	[-0.74, -0.51]
➖	syslog_log2metric_splunk_hec_metrics	ingress throughput	-0.67	[-0.80, -0.53]
➖	otlp_grpc_to_blackhole	ingress throughput	-0.67	[-0.79, -0.55]
➖	datadog_agent_remap_blackhole_acks	ingress throughput	-1.00	[-1.11, -0.90]
➖	syslog_regex_logs2metric_ddmetrics	ingress throughput	-1.29	[-1.41, -1.16]
➖	fluent_elasticsearch	ingress throughput	-1.38	[-1.87, -0.89]
➖	datadog_agent_remap_datadog_logs_acks	ingress throughput	-2.13	[-2.31, -1.95]
➖	datadog_agent_remap_datadog_logs	ingress throughput	-2.47	[-2.69, -2.25]
➖	http_elasticsearch	ingress throughput	-3.09	[-3.31, -2.88]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

github-actions bot added the domain: sinks Anything related to the Vector's sinks label Oct 8, 2024

neuronull force-pushed the neuronull/fix_socket_sink_tcp_graceful_shutdown branch from 5ba54b2 to 76070ce Compare October 8, 2024 19:38

fix(socket sink): gracefully shutdown on reload when stream is termin…

a449052

…ated

neuronull force-pushed the neuronull/fix_socket_sink_tcp_graceful_shutdown branch from 76070ce to a449052 Compare October 8, 2024 19:44

neuronull self-assigned this Oct 8, 2024

neuronull marked this pull request as ready for review October 8, 2024 19:52

neuronull requested a review from a team as a code owner October 8, 2024 19:52

bruceg reviewed Oct 8, 2024

View reviewed changes

src/sinks/util/tcp.rs Show resolved Hide resolved

pront approved these changes Oct 8, 2024

View reviewed changes

fix configured out dep

0ccd06b

bruceg approved these changes Oct 8, 2024

View reviewed changes

neuronull enabled auto-merge October 8, 2024 22:15

neuronull added this pull request to the merge queue Oct 8, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 8, 2024

fix scoping fr

b5b59df

neuronull added this pull request to the merge queue Oct 9, 2024

Merged via the queue into master with commit 59b7c9a Oct 9, 2024
50 checks passed

neuronull deleted the neuronull/fix_socket_sink_tcp_graceful_shutdown branch October 9, 2024 17:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(socket sink): gracefully shutdown on reload when stream is terminated #21455

fix(socket sink): gracefully shutdown on reload when stream is terminated #21455

neuronull commented Oct 8, 2024 •

edited

Loading

pront left a comment

github-actions bot commented Oct 8, 2024

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

github-actions bot commented Oct 9, 2024

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

github-actions bot commented Oct 9, 2024

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

fix(socket sink): gracefully shutdown on reload when stream is terminated #21455

fix(socket sink): gracefully shutdown on reload when stream is terminated #21455

Conversation

neuronull commented Oct 8, 2024 • edited Loading

reproducing it (before fix)

pront left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 8, 2024

Regression Detector Results

No significant changes in experiment optimization goals

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

github-actions bot commented Oct 9, 2024

Regression Detector Results

No significant changes in experiment optimization goals

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

github-actions bot commented Oct 9, 2024

Regression Detector Results

No significant changes in experiment optimization goals

Experiments ignored for regressions

Fine details of change detection per experiment

Explanation

neuronull commented Oct 8, 2024 •

edited

Loading