Addon-Test: Fix config and watch mode inconsistencies #30491

JReinhold · 2025-02-06T20:55:47Z

What I did

Changed the flow for Testing Providers, to move config and watch handling to be an internal thing, completely opaque to Storybook core. This means that addon-test now handles all of that internally, using the new UniversalStore API instead.

User facing changes:

watch mode, coverage and a11y is now controlled by the server. No more inconsistent checkboxes when restarting the server and the UI still says they are enabled. Nothing is persisted to session storage.
This also means that multiple open tabs will sync their state immediately.
No more "Settings updated" message and green glow.

Breaking API changes to the experimental Test provider API:

watching, watchable and changing config is now not part of the API anymore. None of these were used by VTA, the only other known user of the API.
Updated telemetry to include data on coverage and a11y state.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

stories
unit tests
integration tests
end-to-end tests

Manual testing

Start any Storybook, the UI, a sandbox or the Portable Stories kitchen sink we use to E2E test Testing Module
Enable watch mode, you should see Vitest start up in the terminal
Make changes to stories and ensure it triggers Vitest
Disable watch mode
Enable coverage and a11y
Run a full test, you should get coverage and a11y results
Open a second tab and expand the testing module.
Change coverage and a11y, you should see the changes immediately in the other tab
Restart the dev sever and reload the browser, you should see that coverage, a11y and watch has been disabled again in the UI

Documentation

Add or update documentation reflecting your changes
If you are deprecating/removing a feature, make sure to update
MIGRATION.MD

Checklist for Maintainers

When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli-storybook/src/sandbox-templates.ts
Make sure this PR contains one of the labels below:
Available labels
- bug: Internal changes that fixes incorrect behavior.
- maintenance: User-facing maintenance tasks.
- dependencies: Upgrading (sometimes downgrading) dependencies.
- build: Internal-facing build tooling & test updates. Will not show up in release changelog.
- cleanup: Minor cleanup style change. Will not show up in release changelog.
- documentation: Documentation only changes. Will not show up in release changelog.
- feature request: Introducing a new feature.
- BREAKING CHANGE: Changes that break compatibility in some way with current major version.
- other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

name	before	after	diff	z	%
createSize	0 B	0 B	0 B	-	-
generateSize	80.5 MB	80.5 MB	0 B	0.68	0%
initSize	80.5 MB	80.5 MB	0 B	-0.3	0%
diffSize	97 B	97 B	0 B	-0.33	0%
buildSize	7.24 MB	7.28 MB	32.8 kB	3.32	0.5%
buildSbAddonsSize	1.87 MB	1.89 MB	22.1 kB	20.63	1.2%
buildSbCommonSize	195 kB	195 kB	0 B	-	0%
buildSbManagerSize	1.88 MB	1.88 MB	355 B	-0.66	0%
buildSbPreviewSize	0 B	0 B	0 B	-	-
buildStaticSize	0 B	0 B	0 B	-	-
buildPrebuildSize	3.94 MB	3.96 MB	22.5 kB	8.3	0.6%
buildPreviewSize	3.3 MB	3.31 MB	10.3 kB	1.4	0.3%
testBuildSize	0 B	0 B	0 B	-	-
testBuildSbAddonsSize	0 B	0 B	0 B	-	-
testBuildSbCommonSize	0 B	0 B	0 B	-	-
testBuildSbManagerSize	0 B	0 B	0 B	-	-
testBuildSbPreviewSize	0 B	0 B	0 B	-	-
testBuildStaticSize	0 B	0 B	0 B	-	-
testBuildPrebuildSize	0 B	0 B	0 B	-	-
testBuildPreviewSize	0 B	0 B	0 B	-	-

name	before	after	diff	z	%
createTime	14.3s	27.2s	12.8s	0.7	47.1%
generateTime	20.8s	21.5s	700ms	2.12	3.2%
initTime	4.9s	4.5s	-428ms	-0.38	-9.4%
buildTime	9.9s	8.2s	-1s -655ms	-1.02	-20.1%
testBuildTime	0ms	0ms	0ms	-	-
devPreviewResponsive	5.4s	4.8s	-624ms	-1.08	-12.9%
devManagerResponsive	4.3s	3.8s	-499ms	-0.65	-13.1%
devManagerHeaderVisible	1s	651ms	-423ms	-1.22	-65%
devManagerIndexVisible	1.1s	704ms	-452ms	-1.06	-64.2%
devStoryVisibleUncached	2.9s	2.3s	-599ms	-1.93	🔰-25%
devStoryVisible	1.1s	705ms	-481ms	-1.13	-68.2%
devAutodocsVisible	868ms	611ms	-257ms	-1.31	🔰-42.1%
devMDXVisible	831ms	670ms	-161ms	-0.89	-24%
buildManagerHeaderVisible	796ms	771ms	-25ms	-0.34	-3.2%
buildManagerIndexVisible	895ms	914ms	19ms	-0.28	2.1%
buildStoryVisible	710ms	751ms	41ms	-0.25	5.5%
buildAutodocsVisible	720ms	709ms	-11ms	-0.21	-1.6%
buildMDXVisible	575ms	600ms	25ms	-0.45	4.2%

Greptile Summary

This PR refactors the addon-test module to internalize configuration and watch mode handling using the new UniversalStore API, improving state synchronization across tabs and removing session storage persistence.

Removed watching, watchable, and config properties from the experimental Test Provider API, making these features internal to the addon
Added UniversalStore implementation for state management with proper environment handling and type safety
Added telemetry reporting for coverage and accessibility states with error sanitization
Improved test reliability by adding proper setup/cleanup in e2e tests and removing redundant checks
Added MockUniversalStore for testing with proper state isolation and reset capabilities

nx-cloud · 2025-02-06T21:01:21Z

View your CI Pipeline Execution ↗ for commit 94f64dd.

Command	Status	Duration	Result
`nx run-many -t build --parallel=3`	✅ Succeeded	2m 2s	View ↗

☁️ Nx Cloud last updated this comment at 2025-02-12 13:37:57 UTC

…ybookjs/storybook into jeppe/universal-store-in-addon-test

storybook-pr-benchmarking · 2025-02-07T20:22:37Z

Package Benchmarks

^{Commit: 94f64dd, ran on 12 February 2025 at 13:43:23 UTC}

The following packages have significant changes to their size or dependencies:

`@storybook/core`

	Before	After	Difference
Dependency count	52	52	0
Self size	19.18 MB	19.20 MB	🚨 +19 KB 🚨
Dependency size	14.19 MB	14.19 MB	0 B
Bundle Size Analyzer	Link	Link

`storybook`

	Before	After	Difference
Dependency count	53	53	0
Self size	24 KB	23 KB	🎉 -710 B 🎉
Dependency size	33.37 MB	33.39 MB	🚨 +19 KB 🚨
Bundle Size Analyzer	Link	Link

`sb`

	Before	After	Difference
Dependency count	54	54	0
Self size	1 KB	1 KB	0 B
Dependency size	33.40 MB	33.41 MB	🚨 +19 KB 🚨
Bundle Size Analyzer	Link	Link

`@storybook/cli`

	Before	After	Difference
Dependency count	358	358	0
Self size	261 KB	261 KB	🚨 +145 B 🚨
Dependency size	83.83 MB	83.85 MB	🚨 +21 KB 🚨
Bundle Size Analyzer	Link	Link

`@storybook/codemod`

	Before	After	Difference
Dependency count	276	276	0
Self size	612 KB	612 KB	0 B
Dependency size	65.43 MB	65.45 MB	🚨 +19 KB 🚨
Bundle Size Analyzer	Link	Link

…iversal-store-in-addon-test

greptile-apps

_{38 file(s) reviewed, 17 comment(s)}
_{Edit PR Review Bot Settings | Greptile}

code/addons/test/src/components/TestProviderRender.stories.tsx

code/addons/test/src/components/TestProviderRender.tsx

code/addons/test/src/constants.ts

code/addons/test/src/manager.tsx

code/addons/test/src/node/boot-test-runner.ts

code/core/src/core-server/index.ts

code/core/src/core-server/presets/common-preset.ts

code/core/src/shared/universal-store/mock.ts

code/addons/test/src/components/TestProviderRender.tsx

valentinpalkovic · 2025-02-11T14:32:27Z

code/addons/test/src/preset.ts

+        const status = 'status' in payload ? payload.status : undefined;
+        const progress = 'progress' in payload ? payload.progress : undefined;
+        const error = 'error' in payload ? payload.error : undefined;


Oh TypeScript...

valentinpalkovic · 2025-02-11T14:34:33Z

code/addons/test/src/preset.ts

+    channel.on(
+      TESTING_MODULE_PROGRESS_REPORT,
+      async (payload: TestingModuleProgressReportPayload) => {
+        if (payload.providerId !== TEST_PROVIDER_ID) {


Can you elaborate why a separate telemetry handling is necessary for addon test?

Because we want telemetry to include the information of whether or not coverage/a11y was enabled for the run. And that information is now completely opaque to SB core, so the default core telemetry can't include that information.

code/addons/test/src/preset.ts

valentinpalkovic · 2025-02-11T19:50:49Z

code/addons/test/src/node/reporter.ts

@@ -182,7 +182,6 @@ export class StorybookReporter implements Reporter {
      } as TestingModuleProgressReportProgress,
      details: {
        testResults,
-        config: this.testManager.config,


This change will very likely cause behavioural changes in the frontend. Each run request sent information of how the run should be executed (e.g. a11y=true|false). When sending progress back to the frontend, we take the initial config and send it back. The reason is, that certain states in the sidebar require information of how and whether a particular StatusAPI status should be shown. When receiving the progress report and the user afterwards changes settings, like activating/deactivating a11y, certain statuses might be shown wrongly in the Sidebar. The changes made here are therefore very likely wrong, because they only take the current state into account and not the state from the former test run.

Context: https://github.com/storybookjs/storybook/pull/30077/files

Additionally, if you keep this setting, you don't have to move this telemetry section to addon-test, but rather it can remain as it was in core-presets, because the progress contains the configuration information with which the run was triggered.

Co-authored-by: Norbert de Langen <norbert@chromatic.com>

…ybookjs/storybook into jeppe/universal-store-in-addon-test

JReinhold added 2 commits February 6, 2025 20:41

replace config flow with universalStore

5e7030c

purge config from TestProvider API

59c3eab

purge watching/watchable from TestProvider API

2e5de9d

JReinhold self-assigned this Feb 7, 2025

JReinhold added core ci:normal addon: test labels Feb 7, 2025

Merge branch 'next' into jeppe/universal-store-in-addon-test

c45395e

JReinhold added the bug label Feb 7, 2025

JReinhold added 5 commits February 7, 2025 13:10

purge watching/watchable from TestProvider API

675e015

Merge branch 'jeppe/universal-store-in-addon-test' of github.com:stor…

98a2dbc

…ybookjs/storybook into jeppe/universal-store-in-addon-test

getStore() -> store

d4b0e41

rename universalStore -> store

710d623

introduce MockUniversalStore

80fbe3a

JReinhold added 7 commits February 10, 2025 22:03

add MockUniversalStore, use in TestProviderRender.stories

6e3cffc

Merge branch 'next' of github.com:storybookjs/storybook into jeppe/un…

592f6f3

…iversal-store-in-addon-test

fix mocks in tests

8a17f46

make Testing Module E2E tests more resilient

15cdc17

dependency inject store into test-manager

7551f2f

add readme

24d05c8

cleanup

2d76e12

JReinhold marked this pull request as ready for review February 11, 2025 12:20

Merge branch 'next' into jeppe/universal-store-in-addon-test

1f5205a

JReinhold requested a review from valentinpalkovic February 11, 2025 12:21

greptile-apps bot reviewed Feb 11, 2025

View reviewed changes

valentinpalkovic reviewed Feb 11, 2025

View reviewed changes

JReinhold added 2 commits February 11, 2025 16:32

fix type import

7637b3f

cleanup

3f65a93

Merge branch 'next' into jeppe/universal-store-in-addon-test

8b8e41c

JReinhold requested a review from valentinpalkovic February 11, 2025 19:02

valentinpalkovic reviewed Feb 11, 2025

View reviewed changes

code/addons/test/src/preset.ts Show resolved Hide resolved

valentinpalkovic reviewed Feb 11, 2025

View reviewed changes

JReinhold and others added 4 commits February 12, 2025 10:12

Fix presets getting a CJS module of core when core is running in ESM

6271cb6

Co-authored-by: Norbert de Langen <norbert@chromatic.com>

fix state listeners not getting called when store state was synced

5a9c67c

don't allow events to be handled when store is still syncing

147aeda

Merge branch 'next' into jeppe/universal-store-in-addon-test

013140e

valentinpalkovic approved these changes Feb 12, 2025

View reviewed changes

JReinhold added 2 commits February 12, 2025 14:30

loosen mock testutils types

002aef3

Merge branch 'jeppe/universal-store-in-addon-test' of github.com:stor…

94f64dd

…ybookjs/storybook into jeppe/universal-store-in-addon-test

JReinhold merged commit 29dc3e5 into next Feb 12, 2025
59 checks passed

JReinhold deleted the jeppe/universal-store-in-addon-test branch February 12, 2025 13:56

github-actions bot mentioned this pull request Feb 12, 2025

Release: Prerelease 8.6.0-alpha.6 #30512

Open

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addon-Test: Fix config and watch mode inconsistencies #30491

Addon-Test: Fix config and watch mode inconsistencies #30491

JReinhold commented Feb 6, 2025 •

edited by storybook-pr-benchmarking bot

Loading

nx-cloud bot commented Feb 6, 2025 •

edited

Loading

storybook-pr-benchmarking bot commented Feb 7, 2025 •

edited

Loading

greptile-apps bot left a comment

valentinpalkovic Feb 11, 2025

valentinpalkovic Feb 11, 2025

JReinhold Feb 11, 2025

valentinpalkovic Feb 11, 2025 •

edited

Loading

valentinpalkovic Feb 11, 2025 •

edited

Loading

Addon-Test: Fix config and watch mode inconsistencies #30491

Addon-Test: Fix config and watch mode inconsistencies #30491

Conversation

JReinhold commented Feb 6, 2025 • edited by storybook-pr-benchmarking bot Loading

What I did

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

Manual testing

Documentation

Checklist for Maintainers

🦋 Canary release

Greptile Summary

nx-cloud bot commented Feb 6, 2025 • edited Loading

storybook-pr-benchmarking bot commented Feb 7, 2025 • edited Loading

Package Benchmarks

@storybook/core

storybook

sb

@storybook/cli

@storybook/codemod

greptile-apps bot left a comment

Choose a reason for hiding this comment

valentinpalkovic Feb 11, 2025

Choose a reason for hiding this comment

valentinpalkovic Feb 11, 2025

Choose a reason for hiding this comment

JReinhold Feb 11, 2025

Choose a reason for hiding this comment

valentinpalkovic Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

valentinpalkovic Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

JReinhold commented Feb 6, 2025 •

edited by storybook-pr-benchmarking bot

Loading

nx-cloud bot commented Feb 6, 2025 •

edited

Loading

storybook-pr-benchmarking bot commented Feb 7, 2025 •

edited

Loading

`@storybook/core`

`storybook`

`sb`

`@storybook/cli`

`@storybook/codemod`

valentinpalkovic Feb 11, 2025 •

edited

Loading

valentinpalkovic Feb 11, 2025 •

edited

Loading