api,ui: multi arch improvements #10289

shwstppr · 2025-01-28T11:59:45Z

Description

This PR adds the following changes concerning multi-arch support:

Allow downloading and seeing of aarch64 systemVM template for KVM
Improvements in deployments of system VMs including VRs to allow trying deployment with different architecture templates. Operator can now have their routing template with same name for different architectures and CloudStack will attempt deployment with all of them based on the below config. Similarly, now CloudStack will attempt to deploy SSVM and CPVM with different architecture templates available.
Configuring preferred system VM architecture for a zone using config - system.vm.preferred.architecture
UI changes to display, select and filter architecture fields for resources.
API and UI changes to allow filtering resources using arch

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2025-01-28T12:07:24Z

Codecov Report

Attention: Patch coverage is 5.52923% with 598 lines in your changes missing coverage. Please review.

Project coverage is 16.00%. Comparing base (c80b886) to head (c1750c8).
Report is 45 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...om/cloud/upgrade/SystemVmTemplateRegistration.java	0.00%	190 Missing ⚠️
.../java/com/cloud/storage/dao/VMTemplateDaoImpl.java	1.53%	64 Missing ⚠️
.../network/lb/InternalLoadBalancerVMManagerImpl.java	0.00%	62 Missing ⚠️
...om/cloud/consoleproxy/ConsoleProxyManagerImpl.java	0.00%	56 Missing ⚠️
...va/com/cloud/network/router/NetworkHelperImpl.java	0.00%	49 Missing ⚠️
utils/src/main/java/com/cloud/utils/HttpUtils.java	0.00%	36 Missing ⚠️
.../secondarystorage/SecondaryStorageManagerImpl.java	0.00%	35 Missing ⚠️
.../src/main/java/com/cloud/host/dao/HostDaoImpl.java	0.00%	14 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	0.00%	14 Missing ⚠️
...src/main/java/com/cloud/dc/dao/ClusterDaoImpl.java	0.00%	11 Missing ⚠️
... and 18 more

Additional details and impacted files

@@            Coverage Diff             @@
##               4.20   #10289    +/-   ##
==========================================
  Coverage     15.99%   16.00%            
- Complexity    13081    13105    +24     
==========================================
  Files          5649     5651     +2     
  Lines        495648   496179   +531     
  Branches      60006    60088    +82     
==========================================
+ Hits          79265    79399   +134     
- Misses       407537   407915   +378     
- Partials       8846     8865    +19

Flag	Coverage Δ
uitests	`4.00% <ø> (-0.01%)`	⬇️
unittests	`16.84% <5.52%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2025-03-10T12:48:04Z

@blueorangutan package

blueorangutan · 2025-03-10T12:50:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-10T14:21:15Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12726

New zone-scope config added system.vm.preferred.architecture to allow deployment using specific architecture for system VMs including VRs. Refactor and added checks for deployements Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2025-03-11T11:35:39Z

@blueorangutan package

blueorangutan · 2025-03-11T11:38:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

blueorangutan · 2025-03-11T12:40:14Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12736

shwstppr · 2025-03-11T12:44:43Z

@blueorangutan test

blueorangutan · 2025-03-11T12:46:04Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

nvazquez · 2025-03-11T12:30:03Z

engine/components-api/src/main/java/com/cloud/resource/ResourceManager.java

+    ConfigKey<String> SystemVmPreferredArchitecture = new ConfigKey<>("Advanced"
+            , String.class
+            , "system.vm.preferred.architecture"
+            , ""
+            , "Preferred architecture for the system VMs including virtual routers"
+            , true
+            , ConfigKey.Scope.Zone);


Can we have the commas at the end of each line instead of at the begginning?

Is it possible to use this constructor to define the configKey with kind: Select and specify options which probably would be helpful in the UI.

nvazquez · 2025-03-11T12:37:15Z

engine/schema/src/main/java/com/cloud/host/dao/HostDaoImpl.java

+    public List<Pair<HypervisorType, CPU.CPUArch>> listDistinctHypervisorArchTypes(final Long zoneId) {
+        List<Pair<HypervisorType, CPU.CPUArch>> hypervisorArchList = new ArrayList<>();
+        String selectSql = "SELECT DISTINCT hypervisor_type, arch FROM cloud.host WHERE removed IS NULL";
+        if (zoneId != null) {
+            selectSql += " AND data_center_id=" + zoneId;
+        }
+        TransactionLegacy txn = TransactionLegacy.currentTxn();
+        try {
+            PreparedStatement stmt = txn.prepareAutoCloseStatement(selectSql);
+            ResultSet rs = stmt.executeQuery();
+            while (rs.next()) {
+                HypervisorType hypervisorType = HypervisorType.valueOf(rs.getString("hypervisor_type"));
+                CPU.CPUArch arch = CPU.CPUArch.fromType(rs.getString("arch"));
+                hypervisorArchList.add(new Pair<>(hypervisorType, arch));
+            }
+        } catch (SQLException ex) {
+            logger.error("DB exception {}", ex.getMessage(), ex);
+            return Collections.emptyList();
+        }
+        return hypervisorArchList;
+    }
+


If possible can we include this method on GenericDaoBase accepting a table as parameter so then it can be invoked on ClusterDaoImpl and here just passing the different table to query from?

@nvazquez moving a method with specific columns in GenericDaoBase doesn't sound right to me.
I've refactored the code use SearchBuilder and SearchCriteria instead.

nvazquez · 2025-03-11T12:48:10Z

...lancer/src/main/java/org/apache/cloudstack/network/lb/InternalLoadBalancerVMManagerImpl.java

+        return templateName;
+    }
+
+    protected DomainRouterVO createOrUpdateInternalLb(DomainRouterVO internalLbVm, final long id,


Looks like this method is unused?

blueorangutan · 2025-03-11T13:49:36Z

[SF] Trillian Build Failed (tid-12650)

Pearl1594

Largely lgtm, left a small comment.

Pearl1594 · 2025-03-11T14:33:19Z

engine/components-api/src/main/java/com/cloud/resource/ResourceManager.java

+    ConfigKey<String> SystemVmPreferredArchitecture = new ConfigKey<>("Advanced"
+            , String.class
+            , "system.vm.preferred.architecture"
+            , ""
+            , "Preferred architecture for the system VMs including virtual routers"
+            , true
+            , ConfigKey.Scope.Zone);


Is it possible to use this constructor to define the configKey with kind: Select and specify options which probably would be helpful in the UI.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2025-03-12T06:40:04Z

@blueorangutan package

blueorangutan · 2025-03-12T06:42:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-12T07:33:58Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 12749

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Based on changes from apache#10289 s390x support was added paritally with apache#10038 Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

boring-cyborg bot added the component:UI label Jan 28, 2025

boring-cyborg bot added component:api component:management-server labels Jan 29, 2025

shwstppr changed the title ~~ui: multi arch improvements~~ api,ui: multi arch improvements Jan 29, 2025

boring-cyborg bot added the component:simulator label Jan 29, 2025

shwstppr added 12 commits February 21, 2025 13:57

wip

e7ff4ed

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

ui: multi-arch improvements

67042ce

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

api,ui: filter resources by arch

126dfa9

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

ui: show arch tag in infocard

8f48da3

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

ui: add arch display and filter for kubernetes iso

13f3f14

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

simulator: allow adding hosts with arch

b017025

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

api,ui: allow filtering vms by arch

ee59e61

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

ui: make arch filter a list

13ced5e

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

wip

16282dd

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

wip

bed9666

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

change for downloading templates if allowed

969660a

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

move download method to httputils

f74355f

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr force-pushed the multiarchzones-improvements branch from ca5dd68 to f74355f Compare February 21, 2025 08:58

boring-cyborg bot added component:build component:storage labels Feb 21, 2025

shwstppr added 6 commits February 21, 2025 14:36

remove unnecssary

9073ec1

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix build

48c9d29

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix

bc5f734

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

remove unnecessary change

60dc9d9

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fixes

0ac96d6

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

wip: try system vm deployment with different arch templates

33bb563

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

boring-cyborg bot added component:secondary-storage component:ssvm labels Mar 6, 2025

fix checkstyle

42598f7

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

preferred arch config

582187a

New zone-scope config added system.vm.preferred.architecture to allow deployment using specific architecture for system VMs including VRs. Refactor and added checks for deployements Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

boring-cyborg bot added component:compute component:orchestration labels Mar 11, 2025

ui: changes for arch auto selection in deploy vm

6c95e90

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr requested review from nvazquez and Pearl1594 March 11, 2025 12:08

nvazquez reviewed Mar 11, 2025

View reviewed changes

Pearl1594 reviewed Mar 11, 2025

View reviewed changes

shwstppr added 2 commits March 12, 2025 02:03

CPUArch as an enum, make config select

3069929

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix persiting ilbvm

c98ef69

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

boring-cyborg bot added the component:database label Mar 11, 2025

refactor to use searchbuilder/critera for distinct pair

ce43205

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr added 2 commits March 12, 2025 16:17

cpu test

5d696f0

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

refactor fetch arch types

a6bdd52

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr added a commit to shapeblue/cloudstack that referenced this pull request Mar 12, 2025

ui,server: add s390x architecture selection

87b792e

Based on changes from apache#10289 s390x support was added paritally with apache#10038 Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr mentioned this pull request Mar 12, 2025

ui,server: add s390x architecture selection #10556

Draft

14 tasks

shwstppr added 2 commits March 12, 2025 17:27

fix

499b861

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix

c1750c8

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

api,ui: multi arch improvements #10289

api,ui: multi arch improvements #10289

shwstppr commented Jan 28, 2025 •

edited

Loading

codecov bot commented Jan 28, 2025 •

edited

Loading

shwstppr commented Mar 10, 2025

blueorangutan commented Mar 10, 2025

blueorangutan commented Mar 10, 2025

shwstppr commented Mar 11, 2025

blueorangutan commented Mar 11, 2025

blueorangutan commented Mar 11, 2025

shwstppr commented Mar 11, 2025

blueorangutan commented Mar 11, 2025

nvazquez Mar 11, 2025

Pearl1594 Mar 11, 2025

shwstppr Mar 12, 2025

nvazquez Mar 11, 2025

shwstppr Mar 12, 2025

nvazquez Mar 11, 2025

shwstppr Mar 12, 2025

blueorangutan commented Mar 11, 2025

Pearl1594 left a comment

Pearl1594 Mar 11, 2025

shwstppr commented Mar 12, 2025

blueorangutan commented Mar 12, 2025

blueorangutan commented Mar 12, 2025

api,ui: multi arch improvements #10289

Are you sure you want to change the base?

api,ui: multi arch improvements #10289

Conversation

shwstppr commented Jan 28, 2025 • edited Loading

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov bot commented Jan 28, 2025 • edited Loading

Codecov Report

shwstppr commented Mar 10, 2025

blueorangutan commented Mar 10, 2025

blueorangutan commented Mar 10, 2025

shwstppr commented Mar 11, 2025

blueorangutan commented Mar 11, 2025

blueorangutan commented Mar 11, 2025

shwstppr commented Mar 11, 2025

blueorangutan commented Mar 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blueorangutan commented Mar 11, 2025

Pearl1594 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shwstppr commented Mar 12, 2025

blueorangutan commented Mar 12, 2025

blueorangutan commented Mar 12, 2025

shwstppr commented Jan 28, 2025 •

edited

Loading

codecov bot commented Jan 28, 2025 •

edited

Loading