New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement parallel data loading #21

Open

houstonhaynes opened this issue Feb 14, 2025 · 0 comments

Collaborator

houstonhaynes commented Feb 14, 2025

Implement parallel data loading

Issue #408 | Created by @dsyme | 2022-02-09 20:17:38 UTC |

PyTorch supports "num_workers" on data loaders - add support for similar parallel data loading

train_loader = DataLoader(dataset_train, batch_size=opt.batch_size, shuffle=True, num_workers=opt.num_workers)
valid_loader = DataLoader(dataset_valid, batch_size=opt.batch_size, shuffle=False, num_workers=opt.num_workers)

Comment by @gbaydin | 2022-02-19 02:05:20 UTC

This is definitely needed. Any suggestions on what F# setup to use to implement multiprocessing?

This is how things look like in PyTorch: https://github.com/pytorch/pytorch/blob/52175307e2ade72c41785b2b5724719b4ef578f9/torch/utils/data/dataloader.py#L576

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment