DFBUGS-1696: mgr: Wait for builtin mgr pool to exist before enabling stretch

[travisn]

There is a race condition in ceph v19 at startup of a stretch cluster where if the .mgr pool is created by the mgr pod after stretch mode is entered, the devicehealth mgr module will crash and cause a ceph HEALTH_ERR. The error then causes rook reconcile of all CRs to be skipped because Rook is assuming admin intervention is needed to continue. The fix is to wait until the .mgr pool is created by the mgr pod before entering stretch mode.

At the same time, we suppress mgr module health errors during reconcile. Some ceph health errors should not block the reconcile of the cluster. Mgr modules do not have cause to block the reconcile, as the cluster can usually work even if a module is failing.

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[openshift-ci-robot]

@travisn: This pull request references [Jira Issue DFBUGS-1696](https://issues.redhat.com//browse/DFBUGS-1696), which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (odf-4.18) matches configured target version for branch (odf-4.18)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @nehaberry

In response to this:

There is a race condition in ceph v19 at startup of a stretch cluster where if the .mgr pool is created by the mgr pod after stretch mode is entered, the devicehealth mgr module will crash and cause a ceph HEALTH_ERR. The error then causes rook reconcile of all CRs to be skipped because Rook is assuming admin intervention is needed to continue. The fix is to wait until the .mgr pool is created by the mgr pod before entering stretch mode.

At the same time, we suppress mgr module health errors during reconcile. Some ceph health errors should not block the reconcile of the cluster. Mgr modules do not have cause to block the reconcile, as the cluster can usually work even if a module is failing.

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@openshift-ci-robot: GitHub didn't allow me to request PR reviews from the following users: nehaberry.

Note that only red-hat-storage members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@travisn: This pull request references [Jira Issue DFBUGS-1696](https://issues.redhat.com//browse/DFBUGS-1696), which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (odf-4.18) matches configured target version for branch (odf-4.18)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @nehaberry

In response to this:

There is a race condition in ceph v19 at startup of a stretch cluster where if the .mgr pool is created by the mgr pod after stretch mode is entered, the devicehealth mgr module will crash and cause a ceph HEALTH_ERR. The error then causes rook reconcile of all CRs to be skipped because Rook is assuming admin intervention is needed to continue. The fix is to wait until the .mgr pool is created by the mgr pod before entering stretch mode.

At the same time, we suppress mgr module health errors during reconcile. Some ceph health errors should not block the reconcile of the cluster. Mgr modules do not have cause to block the reconcile, as the cluster can usually work even if a module is failing.

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: BlaineEXE, travisn

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@travisn: [Jira Issue DFBUGS-1696](https://issues.redhat.com//browse/DFBUGS-1696): All pull requests linked via external trackers have merged:

• red-hat-storage/rook#839

[Jira Issue DFBUGS-1696](https://issues.redhat.com//browse/DFBUGS-1696) has been moved to the MODIFIED state.

In response to this:

There is a race condition in ceph v19 at startup of a stretch cluster where if the .mgr pool is created by the mgr pod after stretch mode is entered, the devicehealth mgr module will crash and cause a ceph HEALTH_ERR. The error then causes rook reconcile of all CRs to be skipped because Rook is assuming admin intervention is needed to continue. The fix is to wait until the .mgr pool is created by the mgr pod before entering stretch mode.

At the same time, we suppress mgr module health errors during reconcile. Some ceph health errors should not block the reconcile of the cluster. Mgr modules do not have cause to block the reconcile, as the cluster can usually work even if a module is failing.

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.