-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(codecs): Implement chunked GELF decoding #20859
Merged
jszwedko
merged 72 commits into
vectordotdev:master
from
jorgehermo9:feature/chunked-gelf
Nov 7, 2024
Merged
Changes from 22 commits
Commits
Show all changes
72 commits
Select commit
Hold shift + click to select a range
de3e0c0
feat(gelf): Initial implementation of chunked gelf
jorgehermo9 c6e3204
feat(gelf): improve error handling when decoding
jorgehermo9 5a1b49f
feat(gelf): improve error handling when decoding
jorgehermo9 b85235e
test(gelf): Add more tests for chunked gelf
jorgehermo9 baccaba
feat(gelf): derive clone for chunked gelf decoder
jorgehermo9 148e09b
docs(gelf): Improve doc comments
jorgehermo9 530f243
feat(gelf): improve mutex poisioning handling
jorgehermo9 2bc10a7
feat(gelf): improve mutex poisioning handling
jorgehermo9 94b1cf6
style: fix Cargo.toml stlye
jorgehermo9 d405e6c
style: fix Cargo.toml stlye
jorgehermo9 9b69264
chore: remove comment
jorgehermo9 bfc332e
fix(gelf): Fix partial read of buffer
jorgehermo9 0bc3376
test(gelf): Refactor some tests
jorgehermo9 7f633b0
style: fix log style
jorgehermo9 103b933
style: fix log style
jorgehermo9 6d4f8c7
test(socket): Add chunked gelf tests for socket source
jorgehermo9 2fda018
docs(gelf): Improve doc comments
jorgehermo9 0862c10
docs(gelf): Improve doc comments
jorgehermo9 a5b1121
test(socket): Add chunked gelf tests for unix socket source
jorgehermo9 a5757c6
chore: remove comments
jorgehermo9 33bda36
docs: add changelog
jorgehermo9 aaa9dc6
Merge branch 'master' into feature/chunked-gelf
jorgehermo9 7cff6a7
chore: Remove comments
jorgehermo9 119e122
chore: Remove comments
jorgehermo9 1f2751b
docs(framing): Add chunked gelf framing documentation
jorgehermo9 6f8a214
fix: clippy lints
jorgehermo9 d5b753f
docs: Fix documentation grammar
jorgehermo9 1b7bfe1
fix: clippy lints
jorgehermo9 770f1e3
fix: Add 'unchunked' to spellcheck list
jorgehermo9 7a03832
Update lib/codecs/src/decoding/framing/chunked_gelf.rs
jorgehermo9 49540d5
Merge branch 'master' into feature/chunked-gelf
jorgehermo9 8c66ea2
docs(chunked gelf): Remove default value for chunked gelf options in …
jorgehermo9 09fb7bc
feat(decoding): Return concrete error types in chunked gelf
jorgehermo9 67d9992
test: add assert to matches! macros
jorgehermo9 e9e6563
Merge with master
jorgehermo9 2adaeb5
feat: change timeout_millis u64 to timeout_secs f64
jorgehermo9 049c8b6
style: use tracing macros as the style guides dictates
jorgehermo9 0b41a8a
test: fix broken test
jorgehermo9 d050b3e
style: modify tracing logs usage according to style guide
jorgehermo9 8b74992
test: fix broken udp socket test
jorgehermo9 8331964
test: fix broken chunked gelf framing test
jorgehermo9 c70a645
fix: clippy lints
jorgehermo9 ccdc4c0
Merge with master
jorgehermo9 12fb1e2
refactor: cleanup some code
jorgehermo9 e76a750
refactor: cleanup
jorgehermo9 5c8d38c
docs: update components docs
jorgehermo9 81b1079
Merge remote-tracking branch 'origin/master' into feature/chunked-gelf
jszwedko 6c66ff0
fix: remove malformed comment
jorgehermo9 72414d5
fix: default chunked gelf options
jorgehermo9 b97de92
Merge branch 'master' into feature/chunked-gelf
jorgehermo9 85edb00
feat: default pending_messages_limit to None
jorgehermo9 b214fdb
feat: remove skip_serialization for timeout_secs
jorgehermo9 f1e5996
Update lib/codecs/src/decoding/framing/chunked_gelf.rs
jorgehermo9 77ec8f5
feat: change duplicated chunk log error to debug
jorgehermo9 2ab9bf3
test: add shuffle messages test
jorgehermo9 ef3f0a7
fix: clippy lint
jorgehermo9 308146b
docs: update components docs
jorgehermo9 16f8228
docs: improve chunked gelf decoder option documentation
jorgehermo9 e3f8245
docs: improve chunked gelf decoder option documentation
jorgehermo9 cecfa8c
docs: update components docs
jorgehermo9 2139ae4
feat: Implement max_chunk_length and max_message_length
jorgehermo9 f0ea000
fix: clippy lint
jorgehermo9 f4630e9
test: add test for max_message_length
jorgehermo9 ee78d8c
Merge with master
jorgehermo9 eeaaf31
feat: add serde(default) into new configs
jorgehermo9 ecc61da
docs: update component docs
jorgehermo9 dc07697
test: fix broken max chunk length test
jorgehermo9 8fbfeeb
feat: remove max_chunk_length option
jorgehermo9 4896744
Merge with master
jorgehermo9 d438fb2
chore: update Cargo.lock
jorgehermo9 901978c
docs: generate component docs
jorgehermo9 38a2af0
fix: windows compilation
jorgehermo9 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
Allows for chunked gelf decoding in message-based sources, such as UDP sockets or unix datagram sockets. Implementation is based on [Graylog's documentation](https://go2docs.graylog.org/5-0/getting_in_log_data/gelf.html#GELFviaUDP). | ||
|
||
This framing method can be configured via the `framing.method = "chunked_gelf"` option in the source configuration. | ||
|
||
authors: jorgehermo9 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needed to use tokio inside the
lib/codecs
crate, in order to implement gelf decoding timeouts with tokio tasks, so I added the dependency as a workspace one. If there is any problem with this, we may find another solution