fix(al2): add retries to cluster-active waiter #2096
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Change looks reasonable to me. Worth noting that you can avoid this category of issue if you pass your cluster details to /ci |
|
@cartermckinnon roger that! I've dispatched a workflow. 👍 |
|
@cartermckinnon the workflow that you requested has completed. 🎉
|
cartermckinnon
approved these changes
Dec 18, 2024
cartermckinnon
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available: None Active (closed: #803)
Description of changes:
this checks the return code for
aws eks wait cluster-activein the bootstrap script.we've noticed that if a lot of nodes are booting at the same time, this can get backed off via AWS:
since
errexitis set, this automatically fails the bootstrap script. With this change, I'm intending for this loop to check both returns fromwaitanddescribe clusterto avoid throttling/errors from both.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Testing Done
make k8s=1.29testing a failure scenario would take a little more time and the code change is minimal, but I can try to replicate it if desired (probably easiest by creating an AMI where I set
$rcto286manually on the first attempt between the wait / describe calls)See this guide for recommended testing for PRs. Some tests may not apply. Completing tests and providing additional validation steps are not required, but it is recommended and may reduce review time and time to merge.