Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(replays): Separate I/O and compute portions of the runtime into different Arroyo tasks #86360

Merged
merged 12 commits into from
Mar 5, 2025

Conversation

cmanallen
Copy link
Member

@cmanallen cmanallen commented Mar 5, 2025

I/O and compute was previously separated into different stages of the same task. With this PR we add a two-step recording consumer which separates compute and I/O between two different Arroyo tasks. This allows us to continue processing messages while we upload files to GCS.

This does not overwrite our current production consumer. We deprecate the old recording_buffered consumer and replace it with this new "two-step" consumer. We'll test it in S4S before transitioning our production consumer.

@cmanallen cmanallen requested review from a team as code owners March 5, 2025 01:17
@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Mar 5, 2025
Copy link

codecov bot commented Mar 5, 2025

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
23831 1 23830 289
View the top 1 failed test(s) by shortest run time
tests.acceptance.test_organization_global_selection_header.OrganizationGlobalHeaderTest test_issues_list_to_details_and_back_with_initial_project
Stack Traces | 58.3s run time
#x1B[1m#x1B[31mtests/acceptance/test_organization_global_selection_header.py#x1B[0m:340: in test_issues_list_to_details_and_back_with_initial_project
    assert f"project={self.project_3.id}" in self.browser.current_url
#x1B[1m#x1B[31mE   AssertionError: assert 'project=4555674631208961' in 'http://172.17.0.1:.../organizations/rowdy-tiger/issues/?project=4555674631208960&query=is%3Aunresolved%20issue.priority%3A%5Bhigh%2C%20medium%5D&referrer=issue-stream&stream_index=0'#x1B[0m
#x1B[1m#x1B[31mE    +  where 'http://172.17.0.1:.../organizations/rowdy-tiger/issues/?project=4555674631208960&query=is%3Aunresolved%20issue.priority%3A%5Bhigh%2C%20medium%5D&referrer=issue-stream&stream_index=0' = <sentry.testutils.pytest.selenium.Browser object at 0x7fe12a0143d0>.current_url#x1B[0m
#x1B[1m#x1B[31mE    +    where <sentry.testutils.pytest.selenium.Browser object at 0x7fe12a0143d0> = <sentry.testutils.silo.OrganizationGlobalHeaderTest testMethod=test_issues_list_to_details_and_back_with_initial_project>.browser#x1B[0m

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

},
):
try:
return process_recording_message(parse_recording_message(message.payload.value))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can split to 2 lines

@cmanallen cmanallen merged commit ee75481 into master Mar 5, 2025
49 checks passed
@cmanallen cmanallen deleted the cmanallen/separate-compute-and-io-runners branch March 5, 2025 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Scope: Backend Automatically applied to PRs that change backend components
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants