Skip to content

Commit b207e1b

Browse files
author
Ceph Jenkins
committed
Merge commit '700edb0b312ef7f2ed89afb3de03361f62a916d1' into sync_us--master
Signed-off-by: Ceph Jenkins <ceph-jenkins@redhat.com>
2 parents 062f59a + 700edb0 commit b207e1b

File tree

116 files changed

+2008
-4014
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

116 files changed

+2008
-4014
lines changed

‎.github/workflows/codegen.yml

+2-12
Original file line numberDiff line numberDiff line change
@@ -38,18 +38,8 @@ jobs:
3838
with:
3939
go-version: "1.22"
4040

41-
- name: post-process checkout - copy to github.com
42-
run: |
43-
# copy to the github.com path so codegen will work as expected
44-
mkdir -p $(go env GOPATH)/src/github.com/rook/rook/
45-
cp -R . $(go env GOPATH)/src/github.com/rook/rook/
46-
4741
- name: run codegen
48-
run: |
49-
cd $(go env GOPATH)/src/github.com/rook/rook
50-
GOPATH=$(go env GOPATH) make codegen
42+
run: GOPATH=$(go env GOPATH) make codegen
5143

5244
- name: validate codegen
53-
run: |
54-
cd $(go env GOPATH)/src/github.com/rook/rook
55-
tests/scripts/validate_modified_files.sh codegen
45+
run: tests/scripts/validate_modified_files.sh codegen

‎Documentation/CRDs/Cluster/ceph-cluster-crd.md

+1
Original file line numberDiff line numberDiff line change
@@ -357,6 +357,7 @@ The following storage selection settings are specific to Ceph and do not apply t
357357
* `encryptedDevice`**: Encrypt OSD volumes using dmcrypt ("true" or "false"). By default this option is disabled. See [encryption](http://docs.ceph.com/docs/master/ceph-volume/lvm/encryption/) for more information on encryption in Ceph. (Resizing is not supported for host-based clusters.)
358358
* `crushRoot`: The value of the `root` CRUSH map label. The default is `default`. Generally, you should not need to change this. However, if any of your topology labels may have the value `default`, you need to change `crushRoot` to avoid conflicts, since CRUSH map values need to be unique.
359359
* `enableCrushUpdates`: Enables rook to update the pool crush rule using Pool Spec. Can cause data remapping if crush rule changes, Defaults to false.
360+
* `migration`: Existing PVC based OSDs can be migrated to enable or disable encryption. Refer to the [osd management](../../Storage-Configuration/Advanced/ceph-osd-mgmt.md/#osd-encryption-as-day-2-operation) topic for details.
360361

361362
Allowed configurations are:
362363

‎Documentation/CRDs/specification.md

+85
Original file line numberDiff line numberDiff line change
@@ -8049,6 +8049,65 @@ bool
80498049
</tr>
80508050
</tbody>
80518051
</table>
8052+
<h3 id="ceph.rook.io/v1.Migration">Migration
8053+
</h3>
8054+
<p>
8055+
(<em>Appears on:</em><a href="#ceph.rook.io/v1.StorageScopeSpec">StorageScopeSpec</a>)
8056+
</p>
8057+
<div>
8058+
<p>Migration handles the OSD migration</p>
8059+
</div>
8060+
<table>
8061+
<thead>
8062+
<tr>
8063+
<th>Field</th>
8064+
<th>Description</th>
8065+
</tr>
8066+
</thead>
8067+
<tbody>
8068+
<tr>
8069+
<td>
8070+
<code>confirmation</code><br/>
8071+
<em>
8072+
string
8073+
</em>
8074+
</td>
8075+
<td>
8076+
<em>(Optional)</em>
8077+
<p>A user confirmation to migrate the OSDs. It destroys each OSD one at a time, cleans up the backing disk
8078+
and prepares OSD with same ID on that disk</p>
8079+
</td>
8080+
</tr>
8081+
</tbody>
8082+
</table>
8083+
<h3 id="ceph.rook.io/v1.MigrationStatus">MigrationStatus
8084+
</h3>
8085+
<p>
8086+
(<em>Appears on:</em><a href="#ceph.rook.io/v1.OSDStatus">OSDStatus</a>)
8087+
</p>
8088+
<div>
8089+
<p>MigrationStatus status represents the current status of any OSD migration.</p>
8090+
</div>
8091+
<table>
8092+
<thead>
8093+
<tr>
8094+
<th>Field</th>
8095+
<th>Description</th>
8096+
</tr>
8097+
</thead>
8098+
<tbody>
8099+
<tr>
8100+
<td>
8101+
<code>pending</code><br/>
8102+
<em>
8103+
int
8104+
</em>
8105+
</td>
8106+
<td>
8107+
</td>
8108+
</tr>
8109+
</tbody>
8110+
</table>
80528111
<h3 id="ceph.rook.io/v1.MirrorHealthCheckSpec">MirrorHealthCheckSpec
80538112
</h3>
80548113
<p>
@@ -9471,6 +9530,18 @@ map[string]int
94719530
<p>StoreType is a mapping between the OSD backend stores and number of OSDs using these stores</p>
94729531
</td>
94739532
</tr>
9533+
<tr>
9534+
<td>
9535+
<code>migrationStatus</code><br/>
9536+
<em>
9537+
<a href="#ceph.rook.io/v1.MigrationStatus">
9538+
MigrationStatus
9539+
</a>
9540+
</em>
9541+
</td>
9542+
<td>
9543+
</td>
9544+
</tr>
94749545
</tbody>
94759546
</table>
94769547
<h3 id="ceph.rook.io/v1.OSDStore">OSDStore
@@ -12935,6 +13006,20 @@ Selection
1293513006
</tr>
1293613007
<tr>
1293713008
<td>
13009+
<code>migration</code><br/>
13010+
<em>
13011+
<a href="#ceph.rook.io/v1.Migration">
13012+
Migration
13013+
</a>
13014+
</em>
13015+
</td>
13016+
<td>
13017+
<em>(Optional)</em>
13018+
<p>Migration handles the OSD migration</p>
13019+
</td>
13020+
</tr>
13021+
<tr>
13022+
<td>
1293813023
<code>store</code><br/>
1293913024
<em>
1294013025
<a href="#ceph.rook.io/v1.OSDStore">

‎Documentation/Storage-Configuration/Advanced/ceph-osd-mgmt.md

+26
Original file line numberDiff line numberDiff line change
@@ -190,3 +190,29 @@ If you don't see a new OSD automatically created, restart the operator (by delet
190190

191191
!!! note
192192
The OSD might have a different ID than the previous OSD that was replaced.
193+
194+
195+
## OSD Migration
196+
197+
Ceph does not support changing certain settings on existing OSDs. To support changing these settings on an OSD, the OSD must be destroyed and re-created with the new settings. Rook will automate this by migrating only one OSD at a time. The operator waits for the data to rebalance (PGs to become `active+clean`) before migrating the next OSD. This ensures that there is no data loss. Refer to the [OSD migration](https://github.com/rook/rook/blob/master/design/ceph/osd-migration.md) design doc for more information.
198+
199+
The following scenarios are supported for OSD migration:
200+
201+
- Enable or disable OSD encryption for existing PVC-based OSDs by changing the `encrypted` setting under the `storageClassDeviceSets`
202+
203+
For example:
204+
205+
```yaml
206+
storage:
207+
migration:
208+
confirmation: "yes-really-migrate-osds"
209+
storageClassDeviceSets:
210+
- name: set1
211+
count: 3
212+
encrypted: true # change to true or false based on whether encryption needs to enable or disabled.
213+
```
214+
215+
Details about the migration status can be found under the cephCluster `status.storage.osd.migrationStatus.pending` field which shows the total number of OSDs that are pending migration.
216+
217+
!!! note
218+
Performance of the cluster might be impacted during data rebalancing while OSDs are being migrated.

‎PendingReleaseNotes.md

+1
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@
88

99
- Enable mirroring for CephBlockPoolRadosNamespaces (see [#14701](https://github.com/rook/rook/pull/14701)).
1010
- Enable periodic monitoring for CephBlockPoolRadosNamespaces mirroring (see [#14896](https://github.com/rook/rook/pull/14896)).
11+
- Allow migration of PVC based OSDs to enable or disable encryption (see [#14776](https://github.com/rook/rook/pull/14776)).

‎build/codegen/codegen.sh

+27-23
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/usr/bin/env -S bash -e
1+
#!/usr/bin/env bash
22

33
# Copyright 2018 The Rook Authors. All rights reserved.
44
#
@@ -14,29 +14,33 @@
1414
# See the License for the specific language governing permissions and
1515
# limitations under the License.
1616

17-
GROUP_VERSIONS="ceph.rook.io:v1"
17+
set -o errexit
18+
set -o nounset
19+
set -o pipefail
1820

19-
scriptdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
21+
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
2022

21-
# CODE GENERATION
22-
# we run deepcopy and client,lister,informer generations separately so we can use the flag "--plural-exceptions"
23-
# which is only known by client,lister,informer binary and not the deepcopy binary
23+
base_dir="$(cd "${script_dir}/../.." && pwd)"
2424

25-
# run code deepcopy generation
26-
bash ${CODE_GENERATOR}/generate-groups.sh \
27-
deepcopy \
28-
github.com/rook/rook/pkg/client \
29-
github.com/rook/rook/pkg/apis \
30-
"${GROUP_VERSIONS}" \
31-
--output-base "$(dirname "${BASH_SOURCE[0]}")/../../../../.." \
32-
--go-header-file "${scriptdir}/boilerplate.go.txt"
25+
boilerplate="${base_dir}"/build/codegen/header.txt
3326

34-
# run code client,lister,informer generation
35-
bash ${CODE_GENERATOR}/generate-groups.sh \
36-
client,lister,informer \
37-
github.com/rook/rook/pkg/client \
38-
github.com/rook/rook/pkg/apis \
39-
"${GROUP_VERSIONS}" \
40-
--output-base "$(dirname "${BASH_SOURCE[0]}")/../../../../.." \
41-
--go-header-file "${scriptdir}/boilerplate.go.txt" \
42-
--plural-exceptions "CephNFS:CephNFSes" \
27+
source "${CODE_GENERATOR}/kube_codegen.sh"
28+
(
29+
# because apis is a separate go module, code generation
30+
# must be run from within the corresponding directory.
31+
cd ${base_dir}/pkg/apis
32+
33+
# run code deepcopy generation
34+
kube::codegen::gen_helpers \
35+
--boilerplate "${boilerplate}" \
36+
"${base_dir}/pkg/apis" \
37+
38+
# run code client,lister,informer generation
39+
kube::codegen::gen_client \
40+
--output-dir "${base_dir}/pkg/client" \
41+
--output-pkg "github.com/rook/rook/pkg/client" \
42+
--boilerplate "${boilerplate}" \
43+
--plural-exceptions "CephNFS:CephNFSes" \
44+
--with-watch \
45+
"${base_dir}/pkg/apis"
46+
)

‎build/makelib/golang.mk

+1-1
Original file line numberDiff line numberDiff line change
@@ -231,7 +231,7 @@ $(YQ):
231231
@curl -JL https://github.com/mikefarah/yq/releases/download/$(YQ_VERSION)/yq_$(REAL_HOST_PLATFORM) -o $(YQ)
232232
@chmod +x $(YQ)
233233

234-
export CODE_GENERATOR_VERSION=0.20.0
234+
export CODE_GENERATOR_VERSION=0.31.3
235235
export CODE_GENERATOR=$(TOOLS_HOST_DIR)/code-generator-$(CODE_GENERATOR_VERSION)
236236
$(CODE_GENERATOR):
237237
mkdir -p $(TOOLS_HOST_DIR)

‎cmd/rook/ceph/osd.go

+2-2
Original file line numberDiff line numberDiff line change
@@ -261,10 +261,10 @@ func prepareOSD(cmd *cobra.Command, args []string) error {
261261
}
262262

263263
// destroy the OSD using the OSD ID
264-
var replaceOSD *oposd.OSDReplaceInfo
264+
var replaceOSD *oposd.OSDInfo
265265
if replaceOSDID != -1 {
266266
logger.Infof("destroying osd.%d and cleaning its backing device", replaceOSDID)
267-
replaceOSD, err = osddaemon.DestroyOSD(context, &clusterInfo, replaceOSDID, cfg.pvcBacked, cfg.storeConfig.EncryptedDevice)
267+
replaceOSD, err = osddaemon.DestroyOSD(context, &clusterInfo, replaceOSDID, cfg.pvcBacked)
268268
if err != nil {
269269
rook.TerminateFatal(errors.Wrapf(err, "failed to destroy OSD %d.", replaceOSDID))
270270
}

‎deploy/charts/rook-ceph/templates/resources.yaml

+16
Original file line numberDiff line numberDiff line change
@@ -3445,6 +3445,16 @@ spec:
34453445
minimum: 0
34463446
nullable: true
34473447
type: number
3448+
migration:
3449+
description: Migration handles the OSD migration
3450+
properties:
3451+
confirmation:
3452+
description: |-
3453+
A user confirmation to migrate the OSDs. It destroys each OSD one at a time, cleans up the backing disk
3454+
and prepares OSD with same ID on that disk
3455+
pattern: ^$|^yes-really-migrate-osds$
3456+
type: string
3457+
type: object
34483458
nearFullRatio:
34493459
description: NearFullRatio is the ratio at which the cluster is considered nearly full and will raise a ceph health warning. Default is 0.85.
34503460
maximum: 1
@@ -5538,6 +5548,12 @@ spec:
55385548
osd:
55395549
description: OSDStatus represents OSD status of the ceph Cluster
55405550
properties:
5551+
migrationStatus:
5552+
description: MigrationStatus status represents the current status of any OSD migration.
5553+
properties:
5554+
pending:
5555+
type: integer
5556+
type: object
55415557
storeType:
55425558
additionalProperties:
55435559
type: integer

‎deploy/examples/crds.yaml

+16
Original file line numberDiff line numberDiff line change
@@ -3443,6 +3443,16 @@ spec:
34433443
minimum: 0
34443444
nullable: true
34453445
type: number
3446+
migration:
3447+
description: Migration handles the OSD migration
3448+
properties:
3449+
confirmation:
3450+
description: |-
3451+
A user confirmation to migrate the OSDs. It destroys each OSD one at a time, cleans up the backing disk
3452+
and prepares OSD with same ID on that disk
3453+
pattern: ^$|^yes-really-migrate-osds$
3454+
type: string
3455+
type: object
34463456
nearFullRatio:
34473457
description: NearFullRatio is the ratio at which the cluster is considered nearly full and will raise a ceph health warning. Default is 0.85.
34483458
maximum: 1
@@ -5536,6 +5546,12 @@ spec:
55365546
osd:
55375547
description: OSDStatus represents OSD status of the ceph Cluster
55385548
properties:
5549+
migrationStatus:
5550+
description: MigrationStatus status represents the current status of any OSD migration.
5551+
properties:
5552+
pending:
5553+
type: integer
5554+
type: object
55395555
storeType:
55405556
additionalProperties:
55415557
type: integer

‎design/ceph/osd-migration.md

+1
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ considered in the future:
2525
and application data on slower media.
2626
- Setups with multiple OSDs per drive, though with recent Ceph releases the
2727
motivation for deploying this way is mostly obviated.
28+
- OSDs where Persistent Volumes are using partitioned disks due to a [ceph issue](https://tracker.ceph.com/issues/68977).
2829

2930
## Proposal
3031
- Since migration requires destroying of the OSD and cleaning data from the disk,

‎pkg/apis/ceph.rook.io/v1/types.go

+19-1
Original file line numberDiff line numberDiff line change
@@ -500,7 +500,13 @@ type DeviceClasses struct {
500500
// OSDStatus represents OSD status of the ceph Cluster
501501
type OSDStatus struct {
502502
// StoreType is a mapping between the OSD backend stores and number of OSDs using these stores
503-
StoreType map[string]int `json:"storeType,omitempty"`
503+
StoreType map[string]int `json:"storeType,omitempty"`
504+
MigrationStatus MigrationStatus `json:"migrationStatus,omitempty"`
505+
}
506+
507+
// MigrationStatus status represents the current status of any OSD migration.
508+
type MigrationStatus struct {
509+
Pending int `json:"pending,omitempty"`
504510
}
505511

506512
// ClusterVersion represents the version of a Ceph Cluster
@@ -3051,6 +3057,9 @@ type StorageScopeSpec struct {
30513057
// +nullable
30523058
// +optional
30533059
StorageClassDeviceSets []StorageClassDeviceSet `json:"storageClassDeviceSets,omitempty"`
3060+
// Migration handles the OSD migration
3061+
// +optional
3062+
Migration Migration `json:"migration,omitempty"`
30543063
// +optional
30553064
Store OSDStore `json:"store,omitempty"`
30563065
// +optional
@@ -3089,6 +3098,15 @@ type StorageScopeSpec struct {
30893098
AllowOsdCrushWeightUpdate bool `json:"allowOsdCrushWeightUpdate,omitempty"`
30903099
}
30913100

3101+
// Migration handles the OSD migration
3102+
type Migration struct {
3103+
// A user confirmation to migrate the OSDs. It destroys each OSD one at a time, cleans up the backing disk
3104+
// and prepares OSD with same ID on that disk
3105+
// +optional
3106+
// +kubebuilder:validation:Pattern=`^$|^yes-really-migrate-osds$`
3107+
Confirmation string `json:"confirmation,omitempty"`
3108+
}
3109+
30923110
// OSDStore is the backend storage type used for creating the OSDs
30933111
type OSDStore struct {
30943112
// Type of backend storage to be used while creating OSDs. If empty, then bluestore will be used

0 commit comments

Comments
 (0)