[tests] Fix bnb cpu error #5380

Job	Run time
run-tests (latest, test_prod)	1m 56s
run-tests (latest, test_core)	2m 52s
run-tests (latest, test_cli)	2m 33s
run-tests (latest, test_big_modeling)	2m 22s
run-tests (latest, test_deepspeed)	2m 18s
run-tests (latest, test_fsdp)	2m 23s
run-tests (latest, test_example_differences)	2m 15s
run-tests (latest, test_checkpoint_step)	4m 0s
run-tests (latest, test_checkpoint_epoch)	3m 44s
run-tests (latest, test_rest)	9m 41s
run-tests (minimum, test_prod)	2m 42s
run-tests (minimum, test_core)	4m 6s
run-tests (minimum, test_cli)	4m 23s
run-tests (minimum, test_big_modeling)	3m 17s
run-tests (minimum, test_deepspeed)	3m 22s
run-tests (minimum, test_fsdp)	3m 22s
run-tests (minimum, test_example_differences)	3m 16s
run-tests (minimum, test_checkpoint_step)	4m 50s
run-tests (minimum, test_checkpoint_epoch)	4m 37s
run-tests (minimum, test_rest)	10m 49s
	1h 18m 48s

Provide feedback