Attempting conversion containing for_each with primitives fails

[smithrobs]

What happened?

The pulumi convert --from terraform --language python command does not work for terraform for-loops that have primitives. For example for_each = var.enable ? [1] : [] which we typically do for dynamic blocks such as S3 lifecycle rules which has to be defined in a single resource (ref). The error message is:

error: main.pp:7,40-75: cannot traverse value of type number;
error: main.pp:7,40-75: cannot traverse value of type union(cty.NumberIntVal(1), none);
error: could not generate output program

Example

resource "aws_s3_bucket_lifecycle_configuration" "this" {
  count  = length(var.expirations) > 0 || length(var.transitions) > 0 || length(var.noncurrent_version_transitions) > 0 ? 1 : 0
  bucket = aws_s3_bucket.this.id
  
  # There are some other dynamic rule blocks i have omitted

  dynamic "rule" {
    for_each = var.transitions
    content {
      status = try(rule.value.enabled, true) ? "Enabled" : "Disabled"
      id     = try(rule.value.id, md5("transitions${jsonencode(rule.value)}"))
      dynamic "abort_incomplete_multipart_upload" {
        # following line will fail to convert
        for_each = try(rule.value.abort_incomplete_multipart_upload_days, null) != null ? [1] : []
        content {
          days_after_initiation = rule.value.abort_incomplete_multipart_upload_days
        }
      }
      filter {
        prefix = try(rule.value.prefix, "")
        dynamic "tag" {
          for_each = try(jsondecode(rule.value.tags), [])
          content {
            key   = try(tag.key, null)
            value = try(tag.value, null)
          }
        }
      }
      transition {
        date          = try(rule.value.date, null)
        days          = try(rule.value.days, null)
        storage_class = rule.value.storage_class
      }
    }
  }
}

Output of pulumi about

Not available; assume v3.141.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[joeduffy]

@brandonpollack23 Would love it if you could take a look at this. Sounds like it could be impactful for the customer @smithrobs is working with!

[brandonpollack23]

Taking a look!

[brandonpollack23]

Alright so just a heads up on a few things:

If the variables are not defined in the TF (I assume they are and just ommitted here) then that causes some trouble since our converter doesnt know what the references are to.

Another issue is we've done a bunch of work getting conversions more robust recently but try remains particularly difficult. We have a rudimentary version in pulumi already, but the convert doesn't emit it because there are quite a few cases where it breaks things.

Let me try to see how it handles this case and then I'll see if there are any other things we need to do to get this working and I'll report back shortly

[brandonpollack23]

Alright so the good news is the "try" we have implemented seems to cover this case pretty well (usually this kind of count is 0 or 1 pattern works fine, it's when you need to do .apply that we get hairy).

But there is some kind of typing error on the variable member we're trying to loop on so Ill spend a bit more time on it and see if its a quick fix

[brandonpollack23]

Ok after mulling on this a while I have a hunch:

This type is mistaken as a number, it is assigned as a number, did we actually mean to use the dynamic value of the inner block? I'm trying it out.

It works when I assign just a static block (just one) so perhaps this is it.

Edit:
Stepping through with a debugger now and was able to narrow down where this is happening. Generating the PCL is of course fine, but upon binding it, specifically the object cons part (the value part of the for expression generated). Inside this is the other for loop so I'll keep digging to confirm thats where this is headed.

Specifically the error occurs first in the inner for loop where "entry.value.abortIncompleteMultipartUploadDays" is being bound

digging a bit deeper when scope traversing this expression, entry traverses just fine, but value's type here isn't that of an object but instead a UnionType of either 0 or 1 constant.

This leads to the solution (which in hindsight should have been obvious looking at the PCL...):

Info
When generating for expressions in pcl, the iterator name is always entry, but this overlap causes code generation issues. Here they are both called "entry" so the schemas of what should be an entry overlap, causing the binder to mistake an inner dynamic object for what should be a constant/union

[brandonpollack23]

So to summarize:

1. Try is not yet released, once it is that is the first step: Implement the try TF/PCL function pulumi#18350, enabled on the converter by Enable try and can in the converter now they are both in pu/pu pcl #295
2. Once that is done, the bug found by this needs to be fixed: Converting nested dynamic blocks causes for expression iterator name overlap in PCL pulumi#18718
3. After working around this by editing the pcl Converting for comprehensions in pcl to python/ts that contain an invoke result in calls to non variables pulumi#18649 still seems to occur.

Once these are all fixed this should work.

To give you a little timeline: I'm looking at (2) now, (1) is my next major task and hopefully we have it done in a few weeks. (3) can probably be mixed in between (or will be fixed incidentally by the continuing work).