Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add custom load npz function #293

Merged
merged 1 commit into from
Dec 14, 2024
Merged

Add custom load npz function #293

merged 1 commit into from
Dec 14, 2024

Conversation

mthrok
Copy link
Collaborator

@mthrok mthrok commented Dec 13, 2024

When loading data from NPZ format downloaded from remote source, np.load function can be bottleneck. This PR adds spdl.io.load_npz function, which loads NPZ file faster than the official implementation.

TODO:

  • Bundle zlib into binary distribution.
  • Explore the replacement of np.lib.format.read_array function
    Loading using np.fromfile seems to be faster, but the current approach won't trigger this branch, as SPDL's custom ZipFile class does not detected as file-like object by isfileobj function. Re-implementing np.fromfile might be more straightforward, (but comes with more maintenance cost.)

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 13, 2024
@mthrok mthrok force-pushed the zip branch 5 times, most recently from b2fbdde to 6fb6997 Compare December 14, 2024 04:21
@mthrok mthrok changed the title Add custom zip function Add custom load npz function Dec 14, 2024
@mthrok mthrok marked this pull request as ready for review December 14, 2024 04:37
@mthrok mthrok closed this Dec 14, 2024
@mthrok mthrok reopened this Dec 14, 2024
@mthrok mthrok force-pushed the zip branch 2 times, most recently from 3307d51 to e021cc9 Compare December 14, 2024 04:58
@mthrok mthrok merged commit c6cc0f2 into main Dec 14, 2024
20 of 30 checks passed
@mthrok mthrok deleted the zip branch December 14, 2024 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants