Fix oversubscription with CHPL_LOCALE_MODEL=gpu #26059
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Don't crash if the logical accessible CPU set for a locale isn't set. Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
If we are oversubscribed then the locales should share the devices, instead of treating them as co-locales and partitioning the devices Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
If the number of nodes does not evenly divide the number of locales there will be a "remainder node" that has fewer co-locales than the other nodes. Previously, there was some special-casing to deal with the remainder node which was clunky and error-prone. This commit introduces the "partition" abstraction in which the number of partitions on each node is the expected number of co-locales on the node. All nodes, including the remainder node, allocate resources based on partitions, then assign co-locales to partitions. On the remainder node this means that some partitions (and therefore resources) go unused, but this is what we want because all locales should have the same amount of resources. This greatly cleans up the code. In addition, oversubscription handling is cleaner. If there are locales on the node, but the expected number of co-locales is zero, the node is oversubscribed and all locales share all resources. Also added some remainder node and oversubsciption tests. Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
jabraham17
approved these changes
Oct 9, 2024
jabraham17
reviewed
Oct 9, 2024
Signed-off-by: John H. Hartman <jhh67@users.noreply.github.com>
|
@jhh67 : Does this resolve #25989? @e-kayrakli : And presumably that user's GPU/SMP case that you were helping me with last week as well? |
|
@bradcray: I realized last night I forgot to close the issues resolved by the PRs I merged yesterday. I will do it today. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The PRs to add GPU support to co-locales (PRs #25734 and #25846) broke oversubscription such that no locales had any GPUs. This PR fixes that problem, and cleans up resource allocation with co-locales in general. Oversubscription is handled more cleanly, as is the "remainder" node that occurs when the number of locales is not evenly divisible by the number of nodes.
Signed-off-by: John H. Hartman jhh67@users.noreply.github.com