faet(sentry apps): Add context manager for sentry apps and impl for event webhooks

[Christinarlong]

Per https://www.notion.so/sentry/Sentry-App-SLOs-19f8b10e4b5d805dacb9f80beccd1c65?pvs=4#1a68b10e4b5d807a80fbde13daf0ff7b

We're separating out the preparation and sending of webhooks.

Preparation

• process_resource_change_bound

Sending

• send_webhooks
• send_resource_change_webhook
• send_and_save_webhook_request

With the current implementation we will receive the following metrics:

• sentry_app.prepare_webhook.{outcome} w/ event_type: "process_resource_change.{sender}_{action}"

  • The outcome of this metric is independent of the sending tasks
  • sender would be Error, Group
  • action would be "created"

• sentry_app.send_webhook.{outcome} w/ event_type : issue.created or error.created - potentially recorded 2 times (is this bad?)

  • We declare the context manager in send_resource_change_webhook and send_and_save_webhook_request .
  • Having the context manager be declared twice could lead to some ambiguity on where something failed (?) But I don't think it's that major since the logic in send_resource_change_webhook & send_webhooks is p. minimal (also we have the logs to pinpoint failure place).

[Christinarlong]

We dont have an 'event' string at this point so making something up

[Christinarlong]

ApiHostError & ApiTimeoutError are raised from send_and_save_webhook_request when the response is 503 or 504.

RequestExceptions occur when we get a Timeout or ConnectionError, these I also woudn't consider our server being at fault.

ClientErrors are anything <500 and also wouldn't be our fault, because it's the 3p responsibility to properly consume our requests.

We re raise since the TASK_OPTIONS already have the retry logic specified (i.e which errors to retry or ignore)

[codecov]

Codecov Report

Attention: Patch coverage is 91.91919% with 16 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/sentry/sentry_apps/tasks/sentry_apps.py	82.14%	10 Missing ⚠️
src/sentry/utils/sentry_apps/webhooks.py	82.35%	6 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #86136       +/-   ##
===========================================
+ Coverage   68.06%   87.90%   +19.83%     
===========================================
  Files        9700     9713       +13     
  Lines      550060   550918      +858     
  Branches    21419    21419               
===========================================
+ Hits       374406   484282   +109876     
+ Misses     175270    66252   -109018     
  Partials      384      384