Failed to apply site-to-site VPN #10519

tuanhoangth1603 · 2025-03-06T14:14:41Z

Hello,
I am experiencing an issue with S2S VPN after upgrading from version 4.19 to 4.20.

Context:

My VPC has a single-tier network.
Both RVRs are in a running state.
All S2S VPN connections were working fine before upgrading CloudStack, but after the upgrade, I encountered errors (as shown in the logs below) whenever I restart the VPN connection or even when I delete the VPN, which results in a similar error.
I have tried restarting the VPC with cleanup, deleting, and recreating the RVR, but none of these actions have resolved the issue.
Please help me with this problem. Thanks !!

2025-03-06 21:02:19,264 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:[]) (logid:) Seq 10-7107524636920774788: Processing:  { Ans: , MgmtId: 345052580983, via: 10, Ver: v1, Flags: 100, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.100.35, file: site_2_site_vpn.json.a8fa2457-69d5-4250-9d9e-498c0d22b73d","null - failed: java.io.IOException: Stream closed
        at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287)
        at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330)
        at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190)
        at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177)
        at java.base/java.io.BufferedReader.fill(BufferedReader.java:162)
        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329)
        at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396)
        at com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:41)
        at com.cloud.utils.script.Script.execute(Script.java:314)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:553)
        at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:303)
        at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:318)
        at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:165)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
        at com.cloud.agent.Agent.processRequest(Agent.java:686)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
"],"result":"false","wait":"0","bypassHostMaintenance":"false"}}] }
2025-03-06 21:02:19,264 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-5:[]) (logid:) Seq 1-7107524636920774788: No more commands found
2025-03-06 21:02:19,265 DEBUG [c.c.a.t.Request] (API-Job-Executor-22:[ctx-b5423d6a, job-29372, ctx-95a3513d]) (logid:9e09c55c) Seq 10-7107524636920774788: Received:  { Ans: , MgmtId: 345052580983, via: 10(host134.cs.cfox.local), Ver: v1, Flags: 100, { GroupAnswer } }
2025-03-06 21:02:19,298 WARN  [o.a.c.a.c.u.v.DeleteVpnConnectionCmd] (API-Job-Executor-22:[ctx-b5423d6a, job-29372, ctx-95a3513d]) (logid:9e09c55c) Exception: com.cloud.exception.ResourceUnavailableException: Resource [Site2SiteVpnConnection:20] is unreachable: Failed to apply site-to-site VPN
        at com.cloud.network.vpn.Site2SiteVpnManagerImpl.stopVpnConnection(Site2SiteVpnManagerImpl.java:613)
        at com.cloud.network.vpn.Site2SiteVpnManagerImpl.deleteVpnConnection(Site2SiteVpnManagerImpl.java:585)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:105)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy198.deleteVpnConnection(Unknown Source)
        at org.apache.cloudstack.api.command.user.vpn.DeleteVpnConnectionCmd.execute(DeleteVpnConnectionCmd.java:79)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:652)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:600)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2025-03-06 21:02:19,300 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-b5423d6a, job-29372]) (logid:9e09c55c) Complete async job-29372, jobStatus: FAILED, resultCode: 534, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"534","errortext":"Resource [Site2SiteVpnConnection:20] is unreachable: Failed to apply site-to-site VPN"}

The text was updated successfully, but these errors were encountered:

tuanhoangth1603 · 2025-03-06T14:23:56Z

And there are some logs in VR:

2025-03-06 14:21:10,953 INFO     Adding route table: 103 Table_eth3 to /etc/iproute2/rt_tables if not present
2025-03-06 14:21:10,953 INFO     Executing: ip rule show
2025-03-06 14:21:10,955 INFO     Executing: ip rule show
2025-03-06 14:21:10,957 INFO     Executing: ip link show eth3 | grep ' state '
2025-03-06 14:21:10,961 INFO     Executing2: arping -c 1 -I eth3 -A -U -s 10.1.1.13 10.1.1.1
2025-03-06 14:21:10,961 INFO     Adding route: dev eth3 table: Table_eth3 network: 10.1.1.0/24 if not present
2025-03-06 14:21:10,961 INFO     Executing: ip route show  10.1.1.0/24 table Table_eth3 proto static
2025-03-06 14:21:10,964 INFO     Executing: sudo ip route flush cache
2025-03-06 14:21:10,971 ERROR    Not able to setup source-nat for a regular router yet
2025-03-06 14:21:10,971 INFO     Making dns publicly available
2025-03-06 14:21:10,972 INFO     Executing: systemctl start cloud-password-server@10.1.1.1,10.1.1.13
2025-03-06 14:21:10,980 INFO     Service cloud-password-server@10.1.1.1,10.1.1.13 start
2025-03-06 14:21:10,980 INFO     Flush all IPv6 ACL rules
2025-03-06 14:21:10,980 INFO     Executing: nft list tables ip6 | grep ip6_acl
2025-03-06 14:21:10,989 ERROR    Command 'nft list tables ip6 | grep ip6_acl' returned non-zero exit status 1.
2025-03-06 14:21:10,989 INFO     Executing: iptables-save | grep '^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES
2025-03-06 14:21:10,994 INFO     Executing: iptables-save | grep '^-A FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' | bash
2025-03-06 14:21:11,001 INFO     Executing: iptables -F FW_EGRESS_RULES
2025-03-06 14:21:11,003 INFO     Executing: ipset -L | grep Name:  | awk {'print $2'} | ipset flush
2025-03-06 14:21:11,008 INFO     Executing: ipset -L | grep Name:  | awk {'print $2'} | ipset destroy
2025-03-06 14:21:11,013 INFO     Flush all IPv6 firewall rules
2025-03-06 14:21:11,013 INFO     Executing: nft list tables ip6 | grep ip6_firewall
2025-03-06 14:21:11,020 ERROR    Command 'nft list tables ip6 | grep ip6_firewall' returned non-zero exit status 1.

weizhouapache · 2025-03-06T14:35:29Z

@tuanhoangth1603
can you run the following commands inside the VR ?

cd /var/cache/cloud
cp processed/site_2_site_vpn.json.a8fa2457-69d5-4250-9d9e-498c0d22b73d.gz .
gzip -dk site_2_site_vpn.json.a8fa2457-69d5-4250-9d9e-498c0d22b73d.gz
/opt/cloud/bin/configure.py site_2_site_vpn.json.a8fa2457-69d5-4250-9d9e-498c0d22b73d

tuanhoangth1603 · 2025-03-06T14:39:15Z

Hi @weizhouapache,
I ran and see the output:

Invalid unit name "cloud-password-server@10.1.1.1,10.1.1.13" escaped as "cloud-password-server@10.1.1.1\x2c10.1.1.13" (maybe you should use systemd-escape?).
Traceback (most recent call last):
  File "/opt/cloud/bin/configure.py", line 1680, in <module>
    main(sys.argv)
  File "/opt/cloud/bin/configure.py", line 1670, in main
    execIptables(config)
  File "/opt/cloud/bin/configure.py", line 1660, in execIptables
    iptables_executor.process()
  File "/opt/cloud/bin/configure.py", line 1566, in process
    vpns.process()
  File "/opt/cloud/bin/configure.py", line 1030, in process
    self.configure_ipsec(self.dbag[vpn])
  File "/opt/cloud/bin/configure.py", line 1103, in configure_ipsec
    if splitconnections and peerlistarr.count > 1:
                            ^^^^^^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'builtin_function_or_method' and 'int'

weizhouapache · 2025-03-06T14:41:01Z

Hi @weizhouapache, I ran and see the output:

Invalid unit name "cloud-password-server@10.1.1.1,10.1.1.13" escaped as "cloud-password-server@10.1.1.1\x2c10.1.1.13" (maybe you should use systemd-escape?).
Traceback (most recent call last):
  File "/opt/cloud/bin/configure.py", line 1680, in <module>
    main(sys.argv)
  File "/opt/cloud/bin/configure.py", line 1670, in main
    execIptables(config)
  File "/opt/cloud/bin/configure.py", line 1660, in execIptables
    iptables_executor.process()
  File "/opt/cloud/bin/configure.py", line 1566, in process
    vpns.process()
  File "/opt/cloud/bin/configure.py", line 1030, in process
    self.configure_ipsec(self.dbag[vpn])
  File "/opt/cloud/bin/configure.py", line 1103, in configure_ipsec
    if splitconnections and peerlistarr.count > 1:
                            ^^^^^^^^^^^^^^^^^^^^^
TypeError: '>' not supported between instances of 'builtin_function_or_method' and 'int'

@tuanhoangth1603
this issue has been fixed by #10067

tuanhoangth1603 · 2025-03-06T14:45:02Z

I upgraded to 4.20 a few days ago, but the issue still occurs.
What should I do?

weizhouapache · 2025-03-06T14:59:58Z

4.20.0.0 does not have the fix.

you can refer to #10184 (reply in thread)

tuanhoangth1603 · 2025-03-06T15:10:38Z

Thanks @weizhouapache !

4.20.0.0 does not have the fix.

you can refer to #10184 (reply in thread)

tuanhoangth1603 closed this as completed Mar 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to apply site-to-site VPN #10519

Failed to apply site-to-site VPN #10519

tuanhoangth1603 commented Mar 6, 2025 •

edited

Loading

tuanhoangth1603 commented Mar 6, 2025

weizhouapache commented Mar 6, 2025

tuanhoangth1603 commented Mar 6, 2025

weizhouapache commented Mar 6, 2025

tuanhoangth1603 commented Mar 6, 2025

weizhouapache commented Mar 6, 2025

tuanhoangth1603 commented Mar 6, 2025

Failed to apply site-to-site VPN #10519

Failed to apply site-to-site VPN #10519

Comments

tuanhoangth1603 commented Mar 6, 2025 • edited Loading

tuanhoangth1603 commented Mar 6, 2025

weizhouapache commented Mar 6, 2025

tuanhoangth1603 commented Mar 6, 2025

weizhouapache commented Mar 6, 2025

tuanhoangth1603 commented Mar 6, 2025

weizhouapache commented Mar 6, 2025

tuanhoangth1603 commented Mar 6, 2025

tuanhoangth1603 commented Mar 6, 2025 •

edited

Loading