Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix or remove noisy ddtrace info log for after_in_child hook #885

Open
2 of 3 tasks
robrap opened this issue Jan 6, 2025 · 6 comments
Open
2 of 3 tasks

Fix or remove noisy ddtrace info log for after_in_child hook #885

robrap opened this issue Jan 6, 2025 · 6 comments
Assignees

Comments

@robrap
Copy link
Contributor

robrap commented Jan 6, 2025

Here is an example message:

2025-01-06 16:06:52,552 INFO 1 [request_id None] [ddtrace.internal.forksafe] forksafe.py:69 - after_in_child hook reset_at_fork was unregistered without first being registered

On 2024-01-06, I added a temp log retention filter of 99% exclusion for:

  • "after_in_child hook reset_at_fork was unregistered without first being registered"

Note: Although we aren't indexing this, it will still appear in Live Tail view.

Acceptance Criteria

  • Create or update DD support ticket to learn if DD will fix or if this should be ignored.
  • Confirm fix (log messages no longer appearing), or bump filter to 100% and remove "[TEMP]" designation from name if we need to permanently ignore. (bumped to 100% on Feb 19)
  • Remove "[TEMP]" log retention rule once DD removal of this log message has made it to Production. (see comments for details)
@robrap robrap added this to Arch-BOM Jan 6, 2025
@robrap robrap converted this from a draft issue Jan 6, 2025
@jristau1984
Copy link

Arch-BOM (Tim) will inform DataDog about this, and support Aurora (Rodrigo) while that team looks into it for their specific service.

@jristau1984 jristau1984 moved this from Backlog to Ready For Development in Arch-BOM Jan 7, 2025
@robrap
Copy link
Contributor Author

robrap commented Jan 13, 2025

@timmc-edx: I unassigned you because you are no longer on-call. Either I will get to this as part of on-call, or anyone who frees up can get this. Also, I updated the AC to clarify that the only think we are doing for not is creating the DD support ticket.

@robrap robrap assigned robrap and unassigned timmc-edx Jan 13, 2025
@robrap robrap moved this from Ready For Development to In Progress in Arch-BOM Jan 15, 2025
@robrap
Copy link
Contributor Author

robrap commented Jan 16, 2025

The support ticket was created: https://help.datadoghq.com/hc/en-us/requests/1995270. I'm going to move this to the backlog, although I may follow-up on the DD Support ticket.

@robrap robrap removed their assignment Jan 16, 2025
@robrap robrap moved this from In Progress to Backlog in Arch-BOM Jan 16, 2025
@robrap robrap self-assigned this Jan 28, 2025
@robrap robrap moved this from Backlog to Blocked in Arch-BOM Jan 28, 2025
@robrap
Copy link
Contributor Author

robrap commented Jan 28, 2025

I added some comments to the DD ticket. I also rehydrated logs so I could discover when this problem first started. The problem started for us around 12pm on Oct 30, 2024.

@robrap
Copy link
Contributor Author

robrap commented Jan 29, 2025

[inform] We may need to see if this can be reproduced locally, and not sure if it requires gunicorn on devstack.

@robrap
Copy link
Contributor Author

robrap commented Feb 19, 2025

From DD support ticket:

Additionally, a fix for this issue is included in the release candidate for 3.x and will be available when version 3.x is officially released. You can find more details here: Release 3.0.0rc2 · DataDog/dd-trace-py.

Once this lands, we should be able to remove the rule. We could change the rule to 0% first and ensure the messages don't reappear.

I don't know how long it will take to get to a 3.x release, so I'm going to have us look at this next quarter for a review date.

I also change this to P3, because I think clean-up is important.

@robrap robrap moved this from Blocked to Backlog in Arch-BOM Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Backlog
Development

No branches or pull requests

3 participants