BYO Networking Oddities (or PEBKAC?)

[junkiebev]

/kind bug

What steps did you take and what happened:
It could be that I'm just not reading the documentation correctly, but I'm attempting to use BYO Networking (networks and subnets exist, they all have routing tables and NAT Gateways already) to roll out a CAPZ cluster. The virtual networks exist in a different resource group, however they are in the same subscription. As this is an enterprise environment, the network settings are not fully user-configurable.

I observe the following when using spec.networkSpec.apiServerLB.type: Internal and altering the KubeadmConfigTemplates to inject a record for the apiserver in /etc/hosts

• a node route table will be created, and not referenced (the vnet/subnet already has a route table)
  • If I reference the route table by name, it will be created in the resource group of the cluster, not that of AzureCluster.spec.networkSpec.vnet.resourceGroup
  • This does not appear to be the case on the control-plane subnet
  • This resource is not cleaned up when I delete the cluster
• A NAT gateway will be created, but never used (because it's not tied to a specific network)
  • It does not appear to be possible to opt out of the resource creation
• A private DNS zone will be created which is never used
  • The DNS is set and not configurable on the subnet's definition - VNET injection only works if you use the Azure Provided DNS
  • The configuration is not needed because I'm setting the host record in /etc/hosts in a preKubeAdm command for all config templates
  • It does not appear to be possible to opt out of the resource creation

What did you expect to happen:

• I believe the node route table should be able to be discovered if one brings their own network - I think this might just be a bug
• It should be possible to opt out of the NAT gateway creation if one brings their own network
• The Private Zone (as far as I can determine?) is only required to propagate the apiserver host record. In a world where that's done by writing a line to /etc/hosts, it's not needed and it should be possible to opt-out of the creation.

Anything else you would like to add:
This has been really cool to use!

Environment:

NAME                    NAMESPACE                           TYPE                     CURRENT VERSION   NEXT VERSION
addon-helm              caaph-system                        AddonProvider            v0.2.6            Already up to date
bootstrap-kubeadm       capi-kubeadm-bootstrap-system       BootstrapProvider        v1.8.5            Already up to date
control-plane-kubeadm   capi-kubeadm-control-plane-system   ControlPlaneProvider     v1.8.5            Already up to date
cluster-api             capi-system                         CoreProvider             v1.8.5            Already up to date
infrastructure-azure    capz-system                         InfrastructureProvider   v1.17.1           Already up to date

• cluster-api-provider-azure version: v1.17.1
• Kubernetes version: (use kubectl version): 1.30.3
• OS (e.g. from /etc/os-release): Ubuntu 22.04.5 LTS (assuming you mean on the bootstrap source?)