OAMG team onboarding: match their TF GitHub Workflow

[TomasTomecek]

The team is using a custom GitHub Action which utilizes Action "sclorg/testing-farm-as-github-action@v1.2.10" under the hood.

"oamg/leapp" project invokes two TF test runs:

1. https://github.com/oamg/leapp/blob/835a3d27b52fe44cbca6ed5307e6801981df24d8/.github/workflows/tmt-tests.yml#L154-L180
2. https://github.com/oamg/leapp/blob/835a3d27b52fe44cbca6ed5307e6801981df24d8/.github/workflows/tmt-tests.yml#L182-L209

There are multiple tasks here for us, let's address them with separate PRs or as jira subtasks :)

1. Make sure Packit supports all TF API configuration attributes from above
2. Either enable setting up 2 test jobs that have different configurations and status reporting works correctly (solved in Add a new job config option for identifying the job packit#1506)
3. Or make it a single one and let it be configured on TF/tmt level if the team is okay with that solution
4. Get successful runs

@r0x0d is going to work on the integration from the OAMG side.

TODO

• Support required TF API configuration attributes for OAMG #1534
• Fix reporting from multiple TF jobs #1535
• Triggering testing farm tests with artifacts from another project #1544
• Improve getting of the Testing Farm composes #1653
• Tests not submitted correctly to TF: internal + custom mapping + identifiers #1690

[mfocko]

IIRC 2nd point should be already implemented

[TomasTomecek]

@mfocko wow, that would be perfect, I was worried about that point the most

[mfocko]

added a reference to the PR implementing the 2nd point

[lachmanfrantisek]

Sadly, we need to resolve the test result reporting when using identifiers -- this is the missing point. Probably a new (optional) attribute for pipelines so we are able to map the test result correctly.

[stale]

This issue has been marked as stale because it hasn't seen any
activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed
by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! 🦄 🚀 🤖

(Note: issues labeled with pinned or EPIC are
never marked as stale.)

[r0x0d]

@TomasTomecek this issue will be closed only when all the items regarding the leapp integration is done?

[mfocko]

Looks like the implementation part should be done, at least from my perspective as I do not see anything else linked in the description.

Just say you have CI green and I'll close :)

[r0x0d]

Linked in the TODO, yes, but we had another (two) item(s) that is not included there:

• Triggering testing farm tests with artifacts from another project #1544
• Improve getting of the Testing Farm composes #1653

[TomasTomecek]

thanks @r0x0d! added those two with current state of the work :)

[TomasTomecek]

all work is now complete \o/ 🎉 🎉

[r0x0d]

Thank you everyone for the hard work! This looking super amazing!

I would only ask of you to wait a bit more to close this, as I want to get it merged on leapp side first, and then, I think we are ready to close this long waited issue!

[TomasTomecek]

@r0x0d thanks for the update! we'll wait, Rodolfo

I'm especially curious about the #1544 work, still need to play with it myself :)

[r0x0d]

Hey, @TomasTomecek. Let me ask you something... Did those last two PRs got deployed to production last Tuesday? I'm still seeing the same error in my PoC repository :(

[mfocko]

Hey, @TomasTomecek. Let me ask you something... Did those last two PRs got deployed to production last Tuesday? I'm still seeing the same error in my PoC repository :(

Yes, both should be already deployed

[mfocko]

But I see in Sentry some issues related to composes, I will have a further look into it over the weekend

[r0x0d]

Hey @mfocko! Thanks, hopefully this will turn out to be something easy to fix!

Also, I noticed that the status for my builds/tests are all stuck, don't know if that has something to with the recent GitHub outage or what: r0x0d/leapp-packit-poc#4

[r0x0d]

Hello!

Regarding #1544, as @lbarcziova mentioned the other day in Google Chat, GitHub is replacing the text of namespace/repo#pr to a link, like namespace/repo#1

Take a look at this reproducer: oamg/leapp#804 (comment). This command was typed out and not a copy paste.

[lbarcziova]

Hi! We probably misunderstood, I only meant that raw GitHub comments and what the Github shows may differ, therefore sometimes there can be an error message which can be misleading.

If I write down a PR like lbarcziova/hello-world#1 (where I tested this functionality), the raw comment should not contain a link so it should work fine (there may be only problem when you copy paste).

[r0x0d]

Aha, I see! I probably typed my last comment on oamg/leapp#804 and thought that even by typing the whole packit command it caused the tests to fail, but it didn't! In any case, I think it's good to keep this in mind in case GitHub changes the feature in the future and even typing, they do the replacement for the URL.

Now, one additional thing I just noticed on that PR I posted, the compose for RHEL-8.6-Rhui and RHEL-7.9-Rhui seems to not be available any more? It is strange since there is this other PR from leapp where they are using it, and it seems to be working just fine (oamg/leapp#802 (comment)).

Edit: link for the failed run: https://github.com/oamg/leapp/pull/804/checks?check_run_id=9707064534

[lbarcziova]

Thanks for the feedback! It might make sense at least to mention this in our docs for now and maybe think about improving the format in future.

As for the composes, it is realy strange, but we get the available composes as described in the TF API documentation for each test request https://testing-farm.gitlab.io/api/#operation/composesGet, so the two composes were not there at the time the tests were being triggered by us (and I still don't see them there https://api.dev.testing-farm.io/v0.1/composes/redhat).

[r0x0d]

🤔 okay, I will follow up with the testing farm team. Thanks!

[TomasTomecek]

I think that compose is only available in internal Testing Farm ranch, based on their new webpage: http://storage.tft.osci.redhat.com/composes-production.html (both 7.9 and 8.6 are there)

[r0x0d]

Is there anything we can do about it in packit, @TomasTomecek?

[r0x0d]

Got a response from @thrix, @TomasTomecek. It seems that the list packit uses is a bit outdated. They will update it soon.

[TomasTomecek]

@r0x0d thanks for the headsup! Well, we're just using their API :D nothing else we can do.

[TomasTomecek]

All the tasks in the main TODO list are done: closing. Thank you for the wild ride 🎢

Please create new issues for specific problems.

[r0x0d]

Thank you all for the hard work!