Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using timestamps in nanos since epoch instead of seconds #5

Closed
wants to merge 1 commit into from

Conversation

JoseCVM
Copy link

@JoseCVM JoseCVM commented Aug 12, 2024

Hey, first of all thanks for this crate! 'm using it, but I would like for it to have better precision on the timestamps.
Fluentd has nanosecond support, so we can send it from here as nanos since epoch and have better granularity in logs. Optionally, we could also add a with_timer option to the Builder and have as_secs as the default if this change has consequences I don't see. Does this sound reasonable?

@DoumanAsh
Copy link
Owner

It is been a while since I ever used fluentd, but isn't it supporting only unix timestamp?

Can you share details how you use nanoseconds?

@JoseCVM
Copy link
Author

JoseCVM commented Aug 12, 2024

I'm not doing anything particular - if I run a fluentd server with docker run -v ~/fluent-conf/:/fluentd/etc fluent/fluentd:edge-debian -c /fluentd/etc/fluentd.conf and a standard conf just forwarding to stdout, I get this for the timestamps:
image

If we send the timestamps as seconds instead, we get this as expected
image

So it seems fluentd server is ok with parsing timestamps in the nanoseconds since epoch format. I wish I could find a better reference for this being explicitly allowed beyond just providing a screenshot that it works, but the best I could find was fluent/fluentd#653 which shows support for nanoseconds, but not necessarily this format

@DoumanAsh
Copy link
Owner

What I mean is that it is not clear to me how fluentd would be able to automatically distinguish type of timestamp since both would be deserialized from integers...

Anyway I will take a look at issue and maybe source code to figure out if it is possible to explicitly specify type of timestamp

@DoumanAsh
Copy link
Owner

Ok so messagepack has timestmap type and we should just encode as in examples from rmp crate
3Hren/msgpack-rust#298

I will take a look at it tomorrow

@DoumanAsh
Copy link
Owner

DoumanAsh commented Aug 12, 2024

@JoseCVM I made PR to implement serialization using special msgpack Timestamp extension
This should be a more reliable way to encode timestamp and guarantee seconds and nanoseconds are interpreted by fluent correctly

Feel free to give it a go and let me know if fluent interprets it correctly
#6

@DoumanAsh
Copy link
Owner

I will close this PR as it is generally not correct to send nanoseconds straight instead of EventTime

@DoumanAsh DoumanAsh closed this Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants