Enable python and django monitoring in kafka consumers and celery workers

[robrap]

This probably should be two separate tasks (kafka consumers and celery workers), but adding one to start to capture the ideas.

Implementation thoughts:

• In Datadog, I'm not sure if the Python version can be seen manually. This might need to be asked in a DD support ticket.
• These changes may highlight where DatadogBackend is missing in other services, and thus they might be missing all custom span tags. So this dashboard might serve as a useful verification that all services are set up correctly.
  • Of course we'd need to expand the operation_name on this dashboard to include celery and kafka consumer related operations.
• Take the part of DeploymentMonitoringMiddleware that actually sets the custom attributes, and make it into a utility method that could be used from outside the middleware as well
• In event-bus-kafka, we could add a call to this new method here. Reminder: add an edx-django-utils minimum constraint in base.in to ensure it gets published as part of the package.
• The Celery change is a little trickier, only because we don't have a mechanism set up to plug in to all services yet (although this is very close).
  • It would probably need to be implemented in a plugin. It would either use a celery signal or DD hook as mentioned in Automatically set code owner for celery tasks #729.
  • If the plugin were defined in edx-django-utils, it would get added to all services (once the new plugin endpoint were defined and added to each service), because this library is already installed in each service.
    • The trouble with doing this in edx-django-utils is that it probably won't work for New Relic. That said, the feature benefits the entire community for upgrades, so hopefully this would be acceptable.
    • The alternative would be to create a 2u-django-utils repo that privately gets deployed to all our services. This is a nice-to-have capability in the long run, but would take additional work, so we'd need to consider if it is worth it.

[robrap]

@robrap: Note to self: Determine if any of this is still valid, or if it is all complete, and then clean-up this ticket.

[UsamaSadiq]

@robrap I noticed you had assigned this issue to yourself. Would you prefer to work on this yourself. I'll remove this from arbi-bom board in that case.

[robrap]

@UsamaSadiq: You are welcome to take this. I will only note that this ticket is from early in the DD migration, so some of it is no longer relevant. As a first step, please help determine what is actually still needed, if anything. Thank you!

[UsamaSadiq]

DatadogBackend was verified to be present in all services and the Datadog migration tracking sheet was updated in the issue.