Terraform destroy of a VM is never finishing

[ckhordiasma]

Describe the bug
I have some VMs that get stuck on the "qmdestroy" task when I try to terraform destroy them. I have stop_on_destroy set to true (tried true/false with same results).

To Reproduce
Create a VM resource (I am using a module I made: https://github.com/kodama-labs/tf-proxmox/tree/master/vm) and terraform destroy it. Terraform gets stuck on "still destroying", and proxmox stays stuck on qmdestroy of that task forever.

Workaround
My workaround is to stop the qmdestroy task and then delete the VM from the proxmox web UI (or delete the vm conf file from the proxmox host)

Additional context
Proxmox output:

()
trying to acquire lock...
 OK
  Logical volume "vm-105-cloudinit" successfully removed.
trying to acquire lock...
 OK
  Logical volume "vm-105-disk-0" successfully removed.

• Single or clustered Proxmox: Single in clustered mode
• Proxmox version: 8.2.4
• Provider version (ideally it should be the latest version): 0.66.3
• Terraform/OpenTofu version: Terraform 1.9.8
• OS (where you run Terraform/OpenTofu from): Arch with Linux 6.6.59

[bpg]

Hey @ckhordiasma 👋🏼

Do you have the qemu agent enabled for your VM? If yes, do you have it installed and enabled in the template?

From the docs:

Setting agent.enabled = true informs Proxmox that the guest agent is expected
to be running inside the VM. Proxmox then uses qemu-guest-agent instead of
ACPI to control the VM. If the agent is not running, Proxmox operations
Shutdown and Reboot time out and fail. The failing operation gets a lock on
the VM, and until the operation times out, other operations like Stop and
Reboot cannot be used.

Do not run VM with agent.enabled = true, unless the VM is configured to
automatically start qemu-guest-agent at some point.

[ckhordiasma]

hi @bpg, thanks for the reply! I have agent.enabled set to false in my case. In particular I have (from https://github.com/kodama-labs/tf-proxmox/blob/master/vm/main.tf):

agent {
    # read 'Qemu guest agent' section, change to true only when ready
    enabled = var.has_qemu_agent
  }
  stop_on_destroy = !var.has_qemu_agent

var.has_qemu_agent is set to false when my issue is occurring.

[bpg]

I couldn't reproduce this issue in my environment.

Here's the minimal configuration I used for testing:

resource "proxmox_virtual_environment_vm" "ubuntu_vm" {
  name      = "test-ubuntu"
  node_name = "pve"

  # should be true if qemu agent is not installed / enabled on the VM
  stop_on_destroy = true

  initialization {
    user_account {
      # do not use this in production, configure your own ssh key instead!
      username = "user"
      password = "password"
    }
  }

  disk {
    datastore_id = "local-lvm"
    file_id      = proxmox_virtual_environment_download_file.ubuntu_cloud_image.id
    interface    = "virtio0"
    iothread     = true
    discard      = "on"
    size         = 20
  }
}

resource "proxmox_virtual_environment_download_file" "ubuntu_cloud_image" {
  content_type = "iso"
  datastore_id = "local"
  node_name    = "pve"
  url          = "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img"
}

Output:

❯ tofu apply -auto-approve
OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # proxmox_virtual_environment_download_file.ubuntu_cloud_image will be created
  + resource "proxmox_virtual_environment_download_file" "ubuntu_cloud_image" {
      + content_type        = "iso"
      + datastore_id        = "local"
      + file_name           = (known after apply)
      + id                  = (known after apply)
      + node_name           = "pve"
      + overwrite           = true
      + overwrite_unmanaged = false
      + size                = (known after apply)
      + upload_timeout      = 600
      + url                 = "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img"
      + verify              = true
    }

  # proxmox_virtual_environment_vm.ubuntu_vm will be created
  + resource "proxmox_virtual_environment_vm" "ubuntu_vm" {
      + acpi                    = true
      + bios                    = "seabios"
      + id                      = (known after apply)
      + ipv4_addresses          = (known after apply)
      + ipv6_addresses          = (known after apply)
      + keyboard_layout         = "en-us"
      + mac_addresses           = (known after apply)
      + migrate                 = false
      + name                    = "test-ubuntu"
      + network_interface_names = (known after apply)
      + node_name               = "pve"
      + on_boot                 = true
      + protection              = false
      + reboot                  = false
      + scsi_hardware           = "virtio-scsi-pci"
      + started                 = true
      + stop_on_destroy         = true
      + tablet_device           = true
      + template                = false
      + timeout_clone           = 1800
      + timeout_create          = 1800
      + timeout_migrate         = 1800
      + timeout_move_disk       = 1800
      + timeout_reboot          = 1800
      + timeout_shutdown_vm     = 1800
      + timeout_start_vm        = 1800
      + timeout_stop_vm         = 300
      + vm_id                   = (known after apply)

      + disk {
          + aio               = "io_uring"
          + backup            = true
          + cache             = "none"
          + datastore_id      = "local-lvm"
          + discard           = "on"
          + file_format       = (known after apply)
          + file_id           = (known after apply)
          + interface         = "virtio0"
          + iothread          = true
          + path_in_datastore = (known after apply)
          + replicate         = true
          + size              = 20
          + ssd               = false
        }

      + initialization {
          + datastore_id = "local-lvm"
          + upgrade      = (known after apply)

          + user_account {
              + password = (sensitive value)
              + username = "user"
            }
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.
proxmox_virtual_environment_download_file.ubuntu_cloud_image: Creating...
proxmox_virtual_environment_download_file.ubuntu_cloud_image: Still creating... [10s elapsed]
proxmox_virtual_environment_download_file.ubuntu_cloud_image: Still creating... [20s elapsed]
proxmox_virtual_environment_download_file.ubuntu_cloud_image: Still creating... [30s elapsed]
proxmox_virtual_environment_download_file.ubuntu_cloud_image: Still creating... [40s elapsed]
proxmox_virtual_environment_download_file.ubuntu_cloud_image: Creation complete after 47s [id=local:iso/jammy-server-cloudimg-amd64.img]
proxmox_virtual_environment_vm.ubuntu_vm: Creating...
proxmox_virtual_environment_vm.ubuntu_vm: Still creating... [10s elapsed]
proxmox_virtual_environment_vm.ubuntu_vm: Still creating... [20s elapsed]
proxmox_virtual_environment_vm.ubuntu_vm: Creation complete after 23s [id=100]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
❯ tofu destroy -auto-approve
proxmox_virtual_environment_download_file.ubuntu_cloud_image: Refreshing state... [id=local:iso/jammy-server-cloudimg-amd64.img]
proxmox_virtual_environment_vm.ubuntu_vm: Refreshing state... [id=100]

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

OpenTofu will perform the following actions:

  # proxmox_virtual_environment_download_file.ubuntu_cloud_image will be destroyed
  - resource "proxmox_virtual_environment_download_file" "ubuntu_cloud_image" {
      - content_type        = "iso" -> null
      - datastore_id        = "local" -> null
      - file_name           = "jammy-server-cloudimg-amd64.img" -> null
      - id                  = "local:iso/jammy-server-cloudimg-amd64.img" -> null
      - node_name           = "pve" -> null
      - overwrite           = true -> null
      - overwrite_unmanaged = false -> null
      - size                = 665835520 -> null
      - upload_timeout      = 600 -> null
      - url                 = "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img" -> null
      - verify              = true -> null
    }

  # proxmox_virtual_environment_vm.ubuntu_vm will be destroyed
  - resource "proxmox_virtual_environment_vm" "ubuntu_vm" {
      - acpi                    = true -> null
      - bios                    = "seabios" -> null
      - id                      = "100" -> null
      - ipv4_addresses          = [] -> null
      - ipv6_addresses          = [] -> null
      - keyboard_layout         = "en-us" -> null
      - mac_addresses           = [] -> null
      - migrate                 = false -> null
      - name                    = "test-ubuntu" -> null
      - network_interface_names = [] -> null
      - node_name               = "pve" -> null
      - on_boot                 = true -> null
      - protection              = false -> null
      - reboot                  = false -> null
      - scsi_hardware           = "virtio-scsi-pci" -> null
      - started                 = true -> null
      - stop_on_destroy         = true -> null
      - tablet_device           = true -> null
      - tags                    = [] -> null
      - template                = false -> null
      - timeout_clone           = 1800 -> null
      - timeout_create          = 1800 -> null
      - timeout_migrate         = 1800 -> null
      - timeout_move_disk       = 1800 -> null
      - timeout_reboot          = 1800 -> null
      - timeout_shutdown_vm     = 1800 -> null
      - timeout_start_vm        = 1800 -> null
      - timeout_stop_vm         = 300 -> null
      - vm_id                   = 100 -> null

      - disk {
          - aio               = "io_uring" -> null
          - backup            = true -> null
          - cache             = "none" -> null
          - datastore_id      = "local-lvm" -> null
          - discard           = "on" -> null
          - file_format       = "raw" -> null
          - file_id           = "local:iso/jammy-server-cloudimg-amd64.img" -> null
          - interface         = "virtio0" -> null
          - iothread          = true -> null
          - path_in_datastore = "vm-100-disk-0" -> null
          - replicate         = true -> null
          - size              = 20 -> null
          - ssd               = false -> null
        }

      - initialization {
          - datastore_id = "local-lvm" -> null
          - interface    = "ide2" -> null
          - upgrade      = false -> null

          - user_account {
              - keys     = [] -> null
              - password = (sensitive value) -> null
              - username = "user" -> null
            }
        }
    }

Plan: 0 to add, 0 to change, 2 to destroy.
proxmox_virtual_environment_vm.ubuntu_vm: Destroying... [id=100]
proxmox_virtual_environment_vm.ubuntu_vm: Destruction complete after 1s
proxmox_virtual_environment_download_file.ubuntu_cloud_image: Destroying... [id=local:iso/jammy-server-cloudimg-amd64.img]
proxmox_virtual_environment_download_file.ubuntu_cloud_image: Destruction complete after 0s

Destroy complete! Resources: 2 destroyed.
❯ 

If the issue is still relevant to you, could you please share the terraform apply and terraform destroy outputs? It would help us better understand what’s going wrong in your case.