Prevent network disruption on adding a VPC tier for redundant VRs #9251

vishesh92 · 2024-06-13T17:30:48Z

Description

This PR fixes #8108

As of now, on adding a new network tier to a VPC, results in a small network disruption because of keepalived's config reload on both the VRs. This patch fixes this by doing operations in this order:

Stop keepalived on all the backup VRs. This prevents the backup VR from becoming master when keepalived on the PRIMARY VR is getting reloaded.
Update the config and reload on all primary VRs.
Update the config and restart on all backup VRs.

Logs before patch on Primary VR

tail -f /var/log/messages | grep keepalived
Jun 13 11:53:31 systemvm Keepalived_vrrp[10528]: (inside_network) Entering BACKUP STATE
Jun 13 11:54:55 systemvm Keepalived_vrrp[10528]: (inside_network) Backup received priority 0 advertisement
Jun 13 11:54:56 systemvm Keepalived_vrrp[10528]: (inside_network) Entering MASTER STATE
Jun 13 12:14:00 systemvm Keepalived[10527]: Reloading ...
Jun 13 12:14:00 systemvm Keepalived[10527]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 13 12:14:00 systemvm Keepalived_vrrp[10528]: Reloading
Jun 13 12:14:00 systemvm Keepalived_vrrp[10528]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 13 12:14:00 systemvm Keepalived_vrrp[10528]: VRRP_Script(heartbeat) considered successful on reload
Jun 13 12:14:00 systemvm Keepalived_vrrp[10528]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jun 13 12:14:03 systemvm Keepalived_vrrp[10528]: (inside_network) received an unexpected ip number count 2, expected 3!
Jun 13 12:14:04 systemvm Keepalived_vrrp[10528]: (inside_network) received an unexpected ip number count 2, expected 3!
Jun 13 12:14:05 systemvm Keepalived_vrrp[10528]: (inside_network) IPSEC-AH : sequence number 4009 already processed. Packet dropped. Local(4009)
Jun 13 12:14:07 systemvm Keepalived_vrrp[10528]: (inside_network) received an unexpected ip number count 2, expected 3!
Jun 13 12:14:07 systemvm Keepalived_vrrp[10528]: (inside_network) Received advert from 172.31.1.88 with lower priority 100, ours 100, forcing new election
Jun 13 12:14:07 systemvm Keepalived_vrrp[10528]: (inside_network) IPSEC-AH : Syncing seq_num - Increment seq

Logs before patch on Backup VR

Jun 13 11:54:56 systemvm Keepalived_vrrp[10628]: (inside_network) Entering BACKUP STATE
Jun 13 12:05:18 systemvm Keepalived_vrrp[10628]: (inside_network) IPSEC-AH : sequence number 3479 already processed. Packet dropped. Local(3479)
Jun 13 12:14:00 systemvm Keepalived_vrrp[10628]: (inside_network) received an unexpected ip number count 3, expected 2!
Jun 13 12:14:01 systemvm Keepalived_vrrp[10628]: (inside_network) received an unexpected ip number count 3, expected 2!
Jun 13 12:14:02 systemvm Keepalived_vrrp[10628]: (inside_network) received an unexpected ip number count 3, expected 2!
Jun 13 12:14:03 systemvm Keepalived_vrrp[10628]: (inside_network) received an unexpected ip number count 3, expected 2!
Jun 13 12:14:03 systemvm Keepalived_vrrp[10628]: (inside_network) Entering MASTER STATE
Jun 13 12:14:05 systemvm Keepalived_vrrp[10628]: (inside_network) IPSEC-AH : sequence number 4009 already processed. Packet dropped. Local(4009)
Jun 13 12:14:06 systemvm Keepalived_vrrp[10628]: (inside_network) received an unexpected ip number count 3, expected 2!
Jun 13 12:14:07 systemvm Keepalived[10627]: Reloading ...
Jun 13 12:14:07 systemvm Keepalived[10627]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 13 12:14:07 systemvm Keepalived_vrrp[10628]: Reloading
Jun 13 12:14:07 systemvm Keepalived_vrrp[10628]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 13 12:14:07 systemvm Keepalived_vrrp[10628]: VRRP_Script(heartbeat) considered successful on reload
Jun 13 12:14:07 systemvm Keepalived_vrrp[10628]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jun 13 12:14:07 systemvm Keepalived_vrrp[10628]: (inside_network) Master received advert from 172.31.1.236 with same priority 100 but higher IP address than ours
Jun 13 12:14:07 systemvm Keepalived_vrrp[10628]: (inside_network) Entering BACKUP STATE

Logs after patch on Primary VR

Jun 14 10:23:04 systemvm Keepalived[116549]: Reloading ...
Jun 14 10:23:04 systemvm Keepalived[116549]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 14 10:23:04 systemvm Keepalived_vrrp[116550]: Reloading
Jun 14 10:23:04 systemvm Keepalived_vrrp[116550]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 14 10:23:04 systemvm Keepalived_vrrp[116550]: VRRP_Script(heartbeat) considered successful on reload
Jun 14 10:23:04 systemvm Keepalived_vrrp[116550]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.

Logs after patch on Backup VR

Jun 14 10:23:00 systemvm Keepalived[133456]: Stopping
Jun 14 10:23:01 systemvm Keepalived_vrrp[133462]: Stopped
Jun 14 10:23:01 systemvm Keepalived[133456]: Stopped Keepalived v2.1.5 (07/13,2020)
Jun 14 10:23:08 systemvm Keepalived[134884]: Starting Keepalived v2.1.5 (07/13,2020)
Jun 14 10:23:08 systemvm Keepalived[134884]: WARNING - keepalived was build for newer Linux 5.10.70, running on Linux 5.10.0-26-amd64 #1 SMP Debian 5.10.197-1 (2023-09-29)
Jun 14 10:23:08 systemvm Keepalived[134884]: Command line: '/usr/sbin/keepalived' '--dont-fork'
Jun 14 10:23:08 systemvm Keepalived[134884]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 14 10:23:08 systemvm Keepalived[134884]: NOTICE: setting config option max_auto_priority should result in better keepalived performance
Jun 14 10:23:08 systemvm Keepalived[134884]: Starting VRRP child process, pid=134885
Jun 14 10:23:08 systemvm Keepalived_vrrp[134885]: Registering Kernel netlink reflector
Jun 14 10:23:08 systemvm Keepalived_vrrp[134885]: Registering Kernel netlink command channel
Jun 14 10:23:08 systemvm Keepalived_vrrp[134885]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 14 10:23:08 systemvm Keepalived_vrrp[134885]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Jun 14 10:23:08 systemvm Keepalived_vrrp[134885]: SECURITY VIOLATION - scripts are being executed but script_security not enabled.
Jun 14 10:23:08 systemvm Keepalived_vrrp[134885]: Registering gratuitous ARP shared channel
Jun 14 10:23:08 systemvm Keepalived_vrrp[134885]: VRRP_Script(heartbeat) succeeded
Jun 14 10:23:08 systemvm Keepalived_vrrp[134885]: (inside_network) Entering BACKUP STATE

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

vishesh92 · 2024-06-13T17:30:57Z

@blueorangutan package

blueorangutan · 2024-06-13T17:32:04Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-13T17:39:27Z

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9935

codecov · 2024-06-13T17:40:00Z

Codecov Report

Attention: Patch coverage is 0% with 95 lines in your changes missing coverage. Please review.

Project coverage is 14.95%. Comparing base (034a5c8) to head (241d962).
Report is 11 commits behind head on 4.19.

Files	Patch %	Lines
...cloud/network/element/VpcVirtualRouterElement.java	0.00%	33 Missing ⚠️
.../router/VpcVirtualNetworkApplianceManagerImpl.java	0.00%	33 Missing ⚠️
...cloudstack/agent/routing/ManageServiceCommand.java	0.00%	16 Missing ⚠️
...esource/virtualnetwork/VirtualRoutingResource.java	0.00%	13 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##              4.19    #9251       +/-   ##
============================================
+ Coverage     4.29%   14.95%   +10.66%     
- Complexity       0    11012    +11012     
============================================
  Files          363     5379     +5016     
  Lines        29374   470006   +440632     
  Branches      5138    58140    +53002     
============================================
+ Hits          1261    70284    +69023     
- Misses       27970   391938   +363968     
- Partials       143     7784     +7641

Flag	Coverage Δ
uitests	`4.29% <ø> (-0.01%)`	⬇️
unittests	`15.66% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

vishesh92 · 2024-06-13T20:30:22Z

@blueorangutan package

blueorangutan · 2024-06-13T20:32:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-13T21:39:16Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9936

vishesh92 · 2024-06-14T09:33:25Z

@blueorangutan package

blueorangutan · 2024-06-14T09:34:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

core/src/main/java/com/cloud/agent/resource/virtualnetwork/VirtualRoutingResource.java

core/src/main/java/com/cloud/agent/api/ManageServiceCommand.java

blueorangutan · 2024-06-14T10:40:17Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9949

vishesh92 · 2024-06-14T12:21:33Z

server/src/main/java/com/cloud/network/element/VpcVirtualRouterElement.java

-        }
-
-        return success;
+        return destroy(network, context);


destroy method has the same implementation. The only difference is the name of the network parameter is config instead of network.

weizhouapache

code lgtm
not tested yet

weizhouapache · 2024-06-14T12:42:55Z

@blueorangutan package

blueorangutan · 2024-06-14T12:44:04Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-14T13:48:51Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9957

vishesh92 · 2024-06-15T05:50:42Z

@blueorangutan test

blueorangutan · 2024-06-15T05:52:03Z

@vishesh92 a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-06-15T18:18:49Z

[SF] Trillian test result (tid-10454)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43308 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9251-t10454-kvm-centos7.zip
Smoke tests completed. 130 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	341.63	test_safe_shutdown.py

vishesh92 · 2024-06-16T11:38:13Z

@blueorangutan package

blueorangutan · 2024-06-16T11:40:03Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-16T12:41:37Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9974

DaanHoogland · 2024-06-17T08:54:42Z

systemvm/debian/opt/cloud/bin/manage_service.sh

+# specific language governing permissions and limitations
+# under the License.
+
+systemctl $1 $2


DaanHoogland

clgtm

DaanHoogland · 2024-06-17T08:56:22Z

@blueorangutan test

blueorangutan · 2024-06-17T08:58:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-06-17T21:14:53Z

[SF] Trillian test result (tid-10466)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42653 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9251-t10466-kvm-centos7.zip
Smoke tests completed. 130 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	341.72	test_safe_shutdown.py

DaanHoogland · 2024-06-18T11:47:15Z

tested in a lab env:

started a redundant VPC
created a tier
deployed a vm
started a ping to the outside
added a tier
no packets being dropped.

…ache#9251)

boring-cyborg bot added component:api component:networking component:vpc labels Jun 13, 2024

vishesh92 added this to the 4.19.1.0 milestone Jun 13, 2024

vishesh92 mentioned this pull request Jun 13, 2024

BACKUP VR of VPC Enters MASTER state and then BACKUP state when add a new VPC tier #8108

Closed

vishesh92 force-pushed the prevent-network-disruption branch from de309fa to 4f9f76c Compare June 13, 2024 20:05

vishesh92 force-pushed the prevent-network-disruption branch from 4f9f76c to b5f7949 Compare June 14, 2024 08:50

boring-cyborg bot added the component:virtual-router label Jun 14, 2024

vishesh92 linked an issue Jun 14, 2024 that may be closed by this pull request

BACKUP VR of VPC Enters MASTER state and then BACKUP state when add a new VPC tier #8108

Closed

vishesh92 marked this pull request as ready for review June 14, 2024 08:51

vishesh92 force-pushed the prevent-network-disruption branch from b5f7949 to ca1d8e7 Compare June 14, 2024 09:04

vishesh92 changed the title ~~Prevent network disruption on adding a new VPC tier for redundant VRs~~ Prevent network disruption on adding/removing a VPC tier for redundant VRs Jun 14, 2024

vishesh92 force-pushed the prevent-network-disruption branch from ca1d8e7 to fd40cda Compare June 14, 2024 09:31

apache deleted a comment from blueorangutan Jun 14, 2024

Prevent network disruption on adding a new VPC tier

c9768c0

vishesh92 changed the title ~~Prevent network disruption on adding/removing a VPC tier for redundant VRs~~ Prevent network disruption on adding a VPC tier for redundant VRs Jun 14, 2024

vishesh92 force-pushed the prevent-network-disruption branch from fd40cda to c9768c0 Compare June 14, 2024 10:13

vishesh92 requested a review from weizhouapache June 14, 2024 10:24

weizhouapache reviewed Jun 14, 2024

View reviewed changes

vishesh92 commented Jun 14, 2024

View reviewed changes

weizhouapache approved these changes Jun 14, 2024

View reviewed changes

Address comments

241d962

vishesh92 force-pushed the prevent-network-disruption branch from 095f0f4 to 241d962 Compare June 16, 2024 11:38

DaanHoogland reviewed Jun 17, 2024

View reviewed changes

DaanHoogland approved these changes Jun 17, 2024

View reviewed changes

DaanHoogland added the status:needs-testing label Jun 17, 2024

DaanHoogland removed the status:needs-testing label Jun 18, 2024

DaanHoogland merged commit 6c7426e into apache:4.19 Jun 18, 2024
24 of 26 checks passed

DaanHoogland deleted the prevent-network-disruption branch June 18, 2024 11:47

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Jun 20, 2024

Prevent network disruption on adding a VPC tier for redundant VRs (ap…

7d40298

…ache#9251)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent network disruption on adding a VPC tier for redundant VRs #9251

Prevent network disruption on adding a VPC tier for redundant VRs #9251

vishesh92 commented Jun 13, 2024 •

edited

Loading

vishesh92 commented Jun 13, 2024

blueorangutan commented Jun 13, 2024

blueorangutan commented Jun 13, 2024

codecov bot commented Jun 13, 2024 •

edited

Loading

vishesh92 commented Jun 13, 2024

blueorangutan commented Jun 13, 2024

blueorangutan commented Jun 13, 2024

vishesh92 commented Jun 14, 2024

blueorangutan commented Jun 14, 2024

blueorangutan commented Jun 14, 2024

vishesh92 Jun 14, 2024

weizhouapache left a comment •

edited

Loading

weizhouapache commented Jun 14, 2024

blueorangutan commented Jun 14, 2024

blueorangutan commented Jun 14, 2024

vishesh92 commented Jun 15, 2024

blueorangutan commented Jun 15, 2024

blueorangutan commented Jun 15, 2024

vishesh92 commented Jun 16, 2024

blueorangutan commented Jun 16, 2024

blueorangutan commented Jun 16, 2024

DaanHoogland Jun 17, 2024

DaanHoogland left a comment

DaanHoogland commented Jun 17, 2024

blueorangutan commented Jun 17, 2024

blueorangutan commented Jun 17, 2024

DaanHoogland commented Jun 18, 2024

Prevent network disruption on adding a VPC tier for redundant VRs #9251

Prevent network disruption on adding a VPC tier for redundant VRs #9251

Conversation

vishesh92 commented Jun 13, 2024 • edited Loading

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

vishesh92 commented Jun 13, 2024

blueorangutan commented Jun 13, 2024

blueorangutan commented Jun 13, 2024

codecov bot commented Jun 13, 2024 • edited Loading

Codecov Report

vishesh92 commented Jun 13, 2024

blueorangutan commented Jun 13, 2024

blueorangutan commented Jun 13, 2024

vishesh92 commented Jun 14, 2024

blueorangutan commented Jun 14, 2024

blueorangutan commented Jun 14, 2024

vishesh92 Jun 14, 2024

Choose a reason for hiding this comment

weizhouapache left a comment • edited Loading

Choose a reason for hiding this comment

weizhouapache commented Jun 14, 2024

blueorangutan commented Jun 14, 2024

blueorangutan commented Jun 14, 2024

vishesh92 commented Jun 15, 2024

blueorangutan commented Jun 15, 2024

blueorangutan commented Jun 15, 2024

vishesh92 commented Jun 16, 2024

blueorangutan commented Jun 16, 2024

blueorangutan commented Jun 16, 2024

DaanHoogland Jun 17, 2024

Choose a reason for hiding this comment

DaanHoogland left a comment

Choose a reason for hiding this comment

DaanHoogland commented Jun 17, 2024

blueorangutan commented Jun 17, 2024

blueorangutan commented Jun 17, 2024

DaanHoogland commented Jun 18, 2024

vishesh92 commented Jun 13, 2024 •

edited

Loading

codecov bot commented Jun 13, 2024 •

edited

Loading

weizhouapache left a comment •

edited

Loading