[Enhancement] Policy folder does not delete when running a destroy

[hawksight]

PROBLEM SUMMARY

Terraform destroy will run and say it has completed and destroyed the Policy... but the policy will remain in TPP.

STEPS TO REPRODUCE

1. Create a policy in terraform
2. Run a terraform -destroy
3. View that your policy folders still exist.

EXPECTED RESULTS

If I destroy it in terraform I expect it to be delete in TPP.

ACTUAL RESULTS

Policy folder resources remain.

Some logs from Terraform:

> TF_CLI_CONFIG_FILE=~/.terraform.rc tf plan -out plan -destroy
╷
│ Warning: Provider development overrides are in effect
│
│ The following provider development overrides are set in the CLI configuration:
│  - venafi/venafi in /Users/peter.fiddes/projects/venafi/terraform-provider-venafi
│
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible
│ with published releases.
╵
venafi_policy.team["team-3"]: Refreshing state... [id=\VED\Policy\Terraform\team-3]
venafi_policy.team["team-2"]: Refreshing state... [id=\VED\Policy\Terraform\team-awesome]
venafi_policy.team["team-1"]: Refreshing state... [id=\VED\Policy\Terraform\team-1]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # venafi_policy.team["team-1"] will be destroyed
  - resource "venafi_policy" "team" {
      - id                   = "\\VED\\Policy\\Terraform\\team-1" -> null
      - policy_specification = jsonencode(
            {
              - defaults = {
                  - domain  = "peter-fiddes-gcp.jetstacker.net"
                  - keyPair = {
                      - ellipticCurve    = "P256"
                      - keyType          = "RSA"
                      - rsaKeySize       = 2048
                      - serviceGenerated = false
                    }
                  - subject = {
                      - country  = "UK"
                      - locality = "Newcastle"
                      - org      = "Jetstack"
                      - orgUnits = [
                          - "CSE",
                          - "PF",
                        ]
                      - state    = "Tyne & Wear"
                    }
                }
              - policy   = {
                  - autoInstalled        = false
                  - certificateAuthority = "\\VED\\Policy\\Certificate Authorities\\digicert-end-date"
                  - domains              = [
                      - "team-1.dev.peter-fiddes-gcp.jetstacker.net",
                      - "another-domain.dev.peter-fiddes-gcp.jetstacker.net",
                    ]
                  - keyPair              = {
                      - ellipticCurves   = [
                          - "P256",
                        ]
                      - keyTypes         = [
                          - "RSA",
                        ]
                      - reuseAllowed     = false
                      - rsaKeySizes      = [
                          - 2048,
                        ]
                      - serviceGenerated = false
                    }
                  - maxValidDays         = 360
                  - subject              = {
                      - countries  = [
                          - "UK",
                        ]
                      - localities = [
                          - "Newcastle",
                        ]
                      - orgUnits   = [
                          - "CSE",
                          - "PF",
                        ]
                      - orgs       = [
                          - "Jetstack",
                        ]
                      - states     = [
                          - "Tyne & Wear",
                        ]
                    }
                  - subjectAltNames      = {
                      - dnsAllowed   = true
                      - emailAllowed = false
                      - ipAllowed    = false
                      - upnAllowed   = false
                      - uriAllowed   = false
                    }
                  - wildcardAllowed      = false
                }
              - users    = [
                  - "team-1",
                ]
            }
        ) -> null
      - zone                 = "\\VED\\Policy\\Terraform\\team-1" -> null
    }

  # venafi_policy.team["team-2"] will be destroyed
  - resource "venafi_policy" "team" {
      - id                   = "\\VED\\Policy\\Terraform\\team-awesome" -> null
      - policy_specification = jsonencode(
            {
              - defaults = {
                  - domain  = "peter-fiddes-gcp.jetstacker.net"
                  - keyPair = {
                      - ellipticCurve    = "P256"
                      - keyType          = "RSA"
                      - rsaKeySize       = 2048
                      - serviceGenerated = false
                    }
                  - subject = {
                      - country  = "UK"
                      - locality = "Newcastle"
                      - org      = "Jetstack"
                      - orgUnits = [
                          - "CSE",
                          - "PF",
                        ]
                      - state    = "Tyne & Wear"
                    }
                }
              - policy   = {
                  - autoInstalled        = false
                  - certificateAuthority = "\\VED\\Policy\\Certificate Authorities\\digicert-end-date"
                  - domains              = [
                      - "team-2.dev.peter-fiddes-gcp.jetstacker.net",
                    ]
                  - keyPair              = {
                      - ellipticCurves   = [
                          - "P256",
                        ]
                      - keyTypes         = [
                          - "RSA",
                        ]
                      - reuseAllowed     = false
                      - rsaKeySizes      = [
                          - 2048,
                        ]
                      - serviceGenerated = false
                    }
                  - maxValidDays         = 360
                  - subject              = {
                      - countries  = [
                          - "UK",
                        ]
                      - localities = [
                          - "Newcastle",
                        ]
                      - orgUnits   = [
                          - "CSE",
                          - "PF",
                        ]
                      - orgs       = [
                          - "Jetstack",
                        ]
                      - states     = [
                          - "Tyne & Wear",
                        ]
                    }
                  - subjectAltNames      = {
                      - dnsAllowed   = true
                      - emailAllowed = false
                      - ipAllowed    = false
                      - upnAllowed   = false
                      - uriAllowed   = false
                    }
                  - wildcardAllowed      = false
                }
              - users    = [
                  - "team-2",
                ]
            }
        ) -> null
      - zone                 = "\\VED\\Policy\\Terraform\\team-awesome" -> null
    }

  # venafi_policy.team["team-3"] will be destroyed
  - resource "venafi_policy" "team" {
      - id                   = "\\VED\\Policy\\Terraform\\team-3" -> null
      - policy_specification = jsonencode(
            {
              - defaults = {
                  - domain  = "peter-fiddes-gcp.jetstacker.net"
                  - keyPair = {
                      - ellipticCurve    = "P256"
                      - keyType          = "RSA"
                      - rsaKeySize       = 2048
                      - serviceGenerated = false
                    }
                  - subject = {
                      - country  = "UK"
                      - locality = "Newcastle"
                      - org      = "Jetstack"
                      - orgUnits = [
                          - "CSE",
                          - "PF",
                        ]
                      - state    = "Tyne & Wear"
                    }
                }
              - policy   = {
                  - autoInstalled        = false
                  - certificateAuthority = "\\VED\\Policy\\Certificate Authorities\\digicert-end-date"
                  - domains              = [
                      - "frontend.dev.peter-fiddes-gcp.jetstacker.net",
                    ]
                  - keyPair              = {
                      - ellipticCurves   = [
                          - "P256",
                        ]
                      - keyTypes         = [
                          - "RSA",
                        ]
                      - reuseAllowed     = false
                      - rsaKeySizes      = [
                          - 2048,
                        ]
                      - serviceGenerated = false
                    }
                  - maxValidDays         = 360
                  - subject              = {
                      - countries  = [
                          - "UK",
                        ]
                      - localities = [
                          - "Newcastle",
                        ]
                      - orgUnits   = [
                          - "CSE",
                          - "PF",
                        ]
                      - orgs       = [
                          - "Jetstack",
                        ]
                      - states     = [
                          - "Tyne & Wear",
                        ]
                    }
                  - subjectAltNames      = {
                      - dnsAllowed   = true
                      - emailAllowed = false
                      - ipAllowed    = false
                      - upnAllowed   = false
                      - uriAllowed   = false
                    }
                  - wildcardAllowed      = false
                }
              - users    = [
                  - "devs",
                ]
            }
        ) -> null
      - zone                 = "\\VED\\Policy\\Terraform\\team-3" -> null
    }

Plan: 0 to add, 0 to change, 3 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan"


> TF_CLI_CONFIG_FILE=~/.terraform.rc tf apply plan
╷
│ Warning: Provider development overrides are in effect
│
│ The following provider development overrides are set in the CLI configuration:
│  - venafi/venafi in /Users/peter.fiddes/projects/venafi/terraform-provider-venafi
│
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible
│ with published releases.
╵
venafi_policy.team["team-1"]: Destroying... [id=\VED\Policy\Terraform\team-1]
venafi_policy.team["team-2"]: Destroying... [id=\VED\Policy\Terraform\team-awesome]
venafi_policy.team["team-3"]: Destroying... [id=\VED\Policy\Terraform\team-3]
venafi_policy.team["team-1"]: Destruction complete after 0s
venafi_policy.team["team-3"]: Destruction complete after 0s
venafi_policy.team["team-2"]: Destruction complete after 0s

Please Note: I am using a local build of the provider based on #154 to fix the certificate authentication for me.

Policy folder still exist:

ENVIRONMENT DETAILS

• TPP 23.3

Terraform v1.9.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v6.7.0
+ provider registry.terraform.io/venafi/venafi v0.21.1

Your version of Terraform is out of date! The latest version
is 1.9.8. You can update by downloading from https://www.terraform.io/downloads.html

COMMENTS/WORKAROUNDS

I have set the following scopes and am using the following customer application in TPP:

variable "venafi_client_id" {
  type        = string
  default     = "terraform-onboard"
  description = "The API Integration to use for the admin use case"
}

variable "venafi_scope" {
  type        = string
  default     = "configuration:manage,delete"
  description = "Applicable scope to restrict usage of token returned"
}

provider "venafi" {
  url          = var.venafi_url
  p12_cert_filename = "./certs/automation-gc-legacy.p12"
  p12_cert_password = "example"
  client_id         = var.venafi_client_id
  scope             = var.venafi_scope
}

No workaround, manual cleanup required.

[luispresuelVenafi]

Hi there @hawksight ,

This is not bug, it's rather intended. I agree that for the regular devops usage for Terraform you'd spec this to be deleted on destroy, but back then we set an agreement to leave this as is (not having implemented the delete function), due to many implications that we would need to handle in Terraform to gracefully delete a "Policy" resource from remote platform using our Terraform provider.

With all this said, I'll remove the "bug" label, as rather this should be considered an enhancement.

Thanks a lot for creating this issue ticket.

[hawksight]

Hey @luispresuelVenafi thanks for taking a look at it. I understand from the Venafi side it's a question of.. :"what do you do with certificates inside that policy if it is deleted"? There's probably options so listing these out might help decide what a reasonable path is. (will try and think of them at some point)

From a Terraform user's perspective there is already a way to prevent destruction of a resource for Production use cases, prevent_destroy:

resource "aws_instance" "example" {
  ...
  lifecycle {
    prevent_destroy = true
  }
}

With this lifecycle provision we can hand of the decision to the Terraform user as with any other terraform resource.

💡 We could enhance or promote it's usage by detecting if not set and emitted a warning to the user if not set? 🤔

[luispresuelVenafi]

Sorry for late response.

That's correct. To take into account this plugin have existed prior Terraform version when life cycle configuration was supported. Per the documention here:

For this tutorial, you will need:

• The Terraform CLI, version 0.14 or later.
  (...)

starting version Terraform version 0.14.0.

It should be ok to support it now, but as you mention, we need to think some ways to let know the user the implications
Maybe a warning letting them know. And even so, maybe introducing this will also involve a full operation to retire or not the certificates inside of it (probably, destroy event for policy should be turned of by default, unlike regular certificate resource retirement destruction, where retirement it's default).

I could think of following aspects:

• Policy destruction turned off by default

• When turned on (this would be the default) :
  Would be retirement of current certificates inside it and also, if possible, a double ask of the question "this operation will also retire certificates since flag/attribute is turned on (boolean equals true), are you really sure? Once starting, as it may take sometime to retire all the certificates, you will need to wait all the time for the operation to finish. Unexpected out come is expected if operation is interrumped"

  • Probably also we will need to print in the standard output or somewhere the list of the certificates we couln't retire if user still, by any reason, interrumps this operation.

• When turned off:
  Will force fully delete the folder/application with out consideration of certificates inside

Not sure if this fully makes sense. What do you think?