OPCT-226: cmd/report UX enhancements #76
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
mtulio
commented
Aug 8, 2023
internal/pkg/summary/consolidated.go
Outdated
Comment on lines
592
to
597
| err := ps.Documentation.Load() | ||
| if err != nil { | ||
| return err | ||
| } |
There was a problem hiding this comment.
If load failed, we must log the error, but not return error and failed the execution. Considering this is not required by the execution, having tests without docs linked due to external connectivity must be acceptable.
mtulio
commented
Aug 8, 2023
internal/pkg/summary/consolidated.go
Outdated
| } | ||
| err = ps.Documentation.BuildIndex() | ||
| if err != nil { | ||
| return err |
There was a problem hiding this comment.
same for this. What kind of errors we could have here? Is it acceptable to keep running the processing pipeline without this feature? Are we cleaning the test docs fields to prevent odd information in the test attributes when processing the HTML interface?
mtulio
commented
Aug 8, 2023
internal/pkg/summary/mustgather.go
Outdated
| } | ||
| } | ||
| */ | ||
| break |
There was a problem hiding this comment.
Don't want to create directory for each item in the tarball, we'll choose what to filter, and where to save in the next case item.
Suggestion to move that case item to bottom, keeping the example of creating nested directories for directory items in the tarball, continuing the loop in the case - or just remove the case item. :)
mtulio
commented
Aug 8, 2023
internal/pkg/summary/mustgather.go
Outdated
| switch { | ||
| // no more files | ||
| case err == io.EOF: | ||
| return nil |
There was a problem hiding this comment.
maybe we need to exit the loop if we want to return to the regular function execution.
mtulio
commented
Aug 8, 2023
internal/pkg/summary/mustgather.go
Outdated
| } | ||
| log.Debug("Waiting for processing routines") | ||
| waiterProcNS.Wait() | ||
| return nil |
There was a problem hiding this comment.
TODO check what to do when successfully finished the processing
mtulio
commented
Aug 8, 2023
internal/pkg/summary/plugin.go
Outdated
Comment on lines
63
to
68
| for kerr, errName := range test.ErrorCounters { | ||
| if _, ok := ps.ErrorCounters[kerr]; !ok { | ||
| ps.ErrorCounters[kerr] = errName | ||
| } else { | ||
| ps.ErrorCounters[kerr] += errName | ||
| } |
There was a problem hiding this comment.
TODO if it is possible to move/use to shared functions in error.go
mtulio
force-pushed
the
devel-report-html
branch
2 times, most recently
from
b2e1648 to
4426012
Compare
mtulio
commented
Aug 23, 2023
Makefile
Outdated
| @@ -4,55 +4,68 @@ export GO111MODULE=on | |||
| # Disable CGO so that we always generate static binaries: | |||
There was a problem hiding this comment.
TODO move the changes in Makefile to a dedicated PR targeting the "Project rename"
data/templates/report/README.md
Outdated
| @@ -0,0 +1,14 @@ | |||
| # Report HTML app | |||
| @@ -0,0 +1,447 @@ | |||
| <!-- README for template delimiter: This file changed the template delimiter for Golang to '[ [' and '] ]', | |||
There was a problem hiding this comment.
is it possible to mve this file to a menu in default page?
data/templates/report/report.html
Outdated
Comment on lines
10
to
27
| <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-eOJMYsd53ii+scO/bJGFsiCZc+5NDVN2yr8+0RDqr0Ql0h+rP48ckxlpbzKgwra6" crossorigin="anonymous"> | ||
| <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/js/bootstrap.bundle.min.js" integrity="sha384-JEW9xMcG8R+pH31jmWH6WWP0WintQrMb4s7ZOdauHnUtxwoG2vI5DkLtS3qm9Ekf" crossorigin="anonymous"></script> | ||
| <script src="https://cdn.jsdelivr.net/npm/vue@2/dist/vue.js"></script> | ||
| <script src="https://unpkg.com/axios/dist/axios.min.js"></script> | ||
|
|
||
| <!-- Load required Bootstrap and BootstrapVue CSS --> | ||
| <link type="text/css" rel="stylesheet" href="https://unpkg.com/bootstrap/dist/css/bootstrap.min.css" /> | ||
| <link type="text/css" rel="stylesheet" href="https://unpkg.com/bootstrap-vue@latest/dist/bootstrap-vue.min.css" /> | ||
|
|
||
| <!-- Load polyfills to support older browsers --> | ||
| <script src="https://polyfill.io/v3/polyfill.min.js?features=es2015%2CIntersectionObserver" crossorigin="anonymous"></script> | ||
|
|
||
| <!-- Load Vue followed by BootstrapVue --> | ||
| <script src="https://unpkg.com/vue@latest/dist/vue.min.js"></script> | ||
| <script src="https://unpkg.com/bootstrap-vue@latest/dist/bootstrap-vue.min.js"></script> | ||
|
|
||
| <!-- Load the following for BootstrapVueIcons support --> | ||
| <script src="https://unpkg.com/bootstrap-vue@latest/dist/bootstrap-vue-icons.min.js"></script> |
There was a problem hiding this comment.
is there duplicated scripts?
internal/pkg/summary/testdoc.go
Outdated
| func (d *TestDocumentation) Load() error { | ||
| req, err := http.NewRequest(http.MethodGet, *d.SourceBaseURL, nil) | ||
| if err != nil { | ||
| // fmt.Printf("client: could not create request: %s\n", err) |
There was a problem hiding this comment.
Suggested change
| // fmt.Printf("client: could not create request: %s\n", err) |
internal/pkg/summary/testdoc.go
Outdated
| return nil | ||
| } | ||
|
|
||
| func (d *TestDocumentation) BuildIndex() error { |
There was a problem hiding this comment.
TODO add test for this function
pkg/report/cmd.go
Outdated
| } | ||
|
|
||
| if input.saveTo != "" { | ||
| // TODO: ConsolidatedSummary should be migrated to SaveResults |
There was a problem hiding this comment.
Suggested change
| // TODO: ConsolidatedSummary should be migrated to SaveResults |
pkg/report/cmd.go
Outdated
| if p.Name == summary.PluginNameOpenShiftUpgrade || p.Name == summary.PluginNameArtifactsCollector { | ||
| return | ||
| } | ||
| // fmt.Printf(" - Failed (without filters) : %s\n", calcPercStr(int64(len(p.FailedList)), stat.Total)) |
There was a problem hiding this comment.
Suggested change
| // fmt.Printf(" - Failed (without filters) : %s\n", calcPercStr(int64(len(p.FailedList)), stat.Total)) |
pkg/status/status.go
Outdated
| @@ -20,24 +20,40 @@ import ( | |||
| ) | |||
|
|
|||
| const ( | |||
| StatusInterval = time.Second * 10 | |||
| DefaultStatusIntervalSeconds = 10 | |||
| // StatusInterval = time.Second * 10 | |||
There was a problem hiding this comment.
Move this change to isolate PR?
Suggested change
| // StatusInterval = time.Second * 10 |
rvanderp3
pushed a commit
that referenced
this pull request
Aug 24, 2023
This change introduces the document used in the report enhancement in progress at #76 The Rules are a set of tests with fixed acceptance criteria that will evaluate the data collected when running `report` command. To allow flexibility and increase the UX, we are only keeping in the code the "Rule ID", linking it to a "Knowledge Base" to explore why the rule is failing, and the recommended next steps. The version of #76 is tested internally by developers but the document can be available as a dev preview when the report frontend is mounting the hyperlinks for the Rules. The demo is available here (direct 4:40) [Red Hat only]: https://drive.google.com/file/d/1v4EIO1mXesDKKy0meZeiVcZ-kSjf7o_E/view?usp=sharing The rules are implemented here: https://github.com/redhat-openshift-ecosystem/provider-certification-tool/blob/87d4f19f7454ed388d546b0737f7462a6235189f/internal/pkg/summary/checks.go This is how it will be referenced in CLI: `./opct-dev report results-failed-registry.tar.gz --save-to /tmp/results/ --loglevel debug`  This is how it will be linked in the HTML report:  
openshift-merge-robot
added
the
needs-rebase
5 tasks
mtulio
changed the title
mtulio
force-pushed
the
devel-report-html
branch
from
710784d to
a811e31
Compare
openshift-merge-robot
removed
the
needs-rebase
mtulio
force-pushed
the
devel-report-html
branch
from
a811e31 to
0b9fb41
Compare
openshift-merge-robot
pushed a commit
to redhat-openshift-ecosystem/provider-certification-plugins
that referenced
this pull request
Sep 20, 2023
This change introduces the [camgi](https://github.com/elmiko/camgi.rs) report to the artifact collector plugin - results artifacts: ~~~ $ tree plugins/99-openshift-artifacts-collector/ plugins/99-openshift-artifacts-collector/ ├── definition.json ├── results │ └── global (...) │ ├── artifacts_must-gather_camgi.html (...) └── sonobuoy_results.yaml ~~~ CAMGI is a tool broadly used in OCP CI to help to troubleshoot many objects in must-gather with a friendly interface. The binary is pretty small, increasing 700 MiB to the compressed image in the Quay.io, and ~1MiB local. ~~~ $ podman images |grep tools quay.io/ocp-cert/tools v0.1.0 66dad94e6c9f 56 minutes ago 253 MB quay.io/ocp-cert/tools v0.0.0-f38min-oc4133-s05612-v0 58b81800eeca 4 weeks ago 252 MB ~~~ The report camgi.html could be bigger, depending on the size and logs in must-gather. The estimated size would be ~30-40MiB. This image will also produce a human-friendly version (SemVer) for the tools image (base image), preventing adding features to the label. This PR also can help in the report improvement introduced in redhat-openshift-ecosystem/opct#76 . Status: - [x] embed camgi binary to the base image - [x] build a dev image and test it - [x] Merge #43 (and rebase here) - [x] change the tools image from this PR to v0.2.0 (leave v0.1.0 to #43) - [ ] update OPCT documentation
mtulio
force-pushed
the
devel-report-html
branch
from
92e8279 to
91ab22d
Compare
openshift-merge-robot
added
the
needs-rebase
mtulio
force-pushed
the
devel-report-html
branch
from
91ab22d to
7e10673
Compare
openshift-merge-robot
added
needs-rebase
mtulio
force-pushed
the
devel-report-html
branch
from
7e10673 to
69a455b
Compare
This was referenced Sep 22, 2023
mtulio
force-pushed
the
devel-report-html
branch
from
dc21563 to
413e5dd
Compare
openshift-merge-robot
added
the
needs-rebase
mtulio
added a commit
to mtulio/provider-certification-tool
that referenced
this pull request
Aug 8, 2024
This change introduce several improvements in the UX while reviewing the report by: - creationg an intuitive HTML report allowing users to quickly see issues and navigate to the logs for each test failure - introduce several gates/SLO/checks to be used as post-processor and get better visibility in the results, based in existing knowledge base/CI data or external ssytems See the PR with details of improvements: redhat-openshift-ecosystem#76 ---- fix sippy query in the flake filter create a rank of tags, adding percentage of counters create html report saving data to json improve report html add log links and rank by error count update camgi tab add waiting interval support status: improve message field with plugin phase report: generating filter.html file with failures table renameing vfs path for html templates report html - update rank and output as json using report as data source of CLI report; apply docs add timers to the runtime report update report collecting more env information report: update html templates to improve filters adding suite errors menu Update suite error menu using native js create a rank of error counters adding an option to extract errors from must-gather extracting must-gather event filters adding a tab to the report adding the structure of report checks introducing dynamic checks moving report to summary Adding checks with acceptance values from baseline adding alert pop to Checks menu parallelism processing must-gather, loweing 3 times proc time adding support to all plugins in report http file server to serve report review: documenting and linking check rules reorg mkdocs/todo report: embeding etcd error parser to json report report/must-gather: consolidating etd parser report: supporting etcd, network check and embed data remove codegen script / not used distributing report to packages parsing meta/run.log extracting runtime information introducing new parsers: opct and meta config renaming sippy to its package renaming packages and cleanup review report html and threasholds review timers for plugins add etcd checks ; cleanup review thresholds bumping plugins image review documentation and doc strings update check platform type fix tests in tags fix dev docs dev/report: adding initial flow of report Fix Containerfile for CI when releasing Fix windows build update makefile to fix windows build isolating status watch interval from redhat-openshift-ecosystem#87 reverting unrelated docs changes improving unit tests adding metaconfig unit tests and test data adding tests for meta-run.log parser adding parsers and tests for config maps fixes in the opct metrics report gen metrics report adding sample document to generate batch reports Review checks to support attributes review report to improve checks increasing the error pattern information for etcd taking notes for report dev doc adding plugin log extraction and link to the plugin name collecting node info extracting install-config review report html and metrics with adm parse cmd adding charts with ploty remove comments when plotting fix log save opct adm parse-etcd-logs to quickly access parsed logs (#4) increasing documentation coverage bump to use new quay.io org create target directory before extractors rename title of checks on report cli fixes in the report check prevent empty data fix/report/cli: show other plugin than k8s report/check: 012 - check plugin failure fixes after rebase rename namespace Supporting PDB to opct server add new gen openshift-tests plugin based in go allow custom openshift-tests image to devel w/ kind working version with entrypoint for plugin plugin manifests fixes to support remote entrypoint to tests image update plugin manifests to new plugin version refact CLI UI tables to enhance results intro yamllint and fixing yaml assets/manifests plugins working version the default flow enhance pre-run checks for missing config create local log file for all levels updatem plugins according to the latest goplugin version add collector image to the manifest template review packages / moving to report package using ETL-like cleanup must-gather/extractor packages refact/must-gather: isolate and tune leaky bucket processor cmd/run: fix flag to use full image path to allow CI presubmits review the result filter pipeline OPCT-292: cmd/publish - add experimental publish used in CI Add a experimental command to allow Prow CI to publish results in the opct-storage (S3) without needing to install dependencies in the test step, preventing failures. OPCT-292: cmd/publish - add metadata option when publishing artifact doc: add review/support documentations feat: upload baseline artifact on s3 report/feat: opct adm baseline (get|list|indexer|publish) review baseline reitrieving summary/baseline from service/API feat: add replay to filter pipeline feat: introduce adm setup-ndoe to helper in tests review plugin filter order fix checks to good tresholds
mtulio
force-pushed
the
devel-report-html
branch
from
1cc6a1b to
50fe32e
Compare
openshift-merge-robot
removed
the
needs-rebase
mtulio
added a commit
to mtulio/provider-certification-tool
that referenced
this pull request
Aug 8, 2024
This change introduce several improvements in the UX while reviewing the report by: - creationg an intuitive HTML report allowing users to quickly see issues and navigate to the logs for each test failure - introduce several gates/SLO/checks to be used as post-processor and get better visibility in the results, based in existing knowledge base/CI data or external ssytems See the PR with details of improvements: redhat-openshift-ecosystem#76 ---- fix sippy query in the flake filter create a rank of tags, adding percentage of counters create html report saving data to json improve report html add log links and rank by error count update camgi tab add waiting interval support status: improve message field with plugin phase report: generating filter.html file with failures table renameing vfs path for html templates report html - update rank and output as json using report as data source of CLI report; apply docs add timers to the runtime report update report collecting more env information report: update html templates to improve filters adding suite errors menu Update suite error menu using native js create a rank of error counters adding an option to extract errors from must-gather extracting must-gather event filters adding a tab to the report adding the structure of report checks introducing dynamic checks moving report to summary Adding checks with acceptance values from baseline adding alert pop to Checks menu parallelism processing must-gather, loweing 3 times proc time adding support to all plugins in report http file server to serve report review: documenting and linking check rules reorg mkdocs/todo report: embeding etcd error parser to json report report/must-gather: consolidating etd parser report: supporting etcd, network check and embed data remove codegen script / not used distributing report to packages parsing meta/run.log extracting runtime information introducing new parsers: opct and meta config renaming sippy to its package renaming packages and cleanup review report html and threasholds review timers for plugins add etcd checks ; cleanup review thresholds bumping plugins image review documentation and doc strings update check platform type fix tests in tags fix dev docs dev/report: adding initial flow of report Fix Containerfile for CI when releasing Fix windows build update makefile to fix windows build isolating status watch interval from redhat-openshift-ecosystem#87 reverting unrelated docs changes improving unit tests adding metaconfig unit tests and test data adding tests for meta-run.log parser adding parsers and tests for config maps fixes in the opct metrics report gen metrics report adding sample document to generate batch reports Review checks to support attributes review report to improve checks increasing the error pattern information for etcd taking notes for report dev doc adding plugin log extraction and link to the plugin name collecting node info extracting install-config review report html and metrics with adm parse cmd adding charts with ploty remove comments when plotting fix log save opct adm parse-etcd-logs to quickly access parsed logs (#4) increasing documentation coverage bump to use new quay.io org create target directory before extractors rename title of checks on report cli fixes in the report check prevent empty data fix/report/cli: show other plugin than k8s report/check: 012 - check plugin failure fixes after rebase rename namespace Supporting PDB to opct server add new gen openshift-tests plugin based in go allow custom openshift-tests image to devel w/ kind working version with entrypoint for plugin plugin manifests fixes to support remote entrypoint to tests image update plugin manifests to new plugin version refact CLI UI tables to enhance results intro yamllint and fixing yaml assets/manifests plugins working version the default flow enhance pre-run checks for missing config create local log file for all levels updatem plugins according to the latest goplugin version add collector image to the manifest template review packages / moving to report package using ETL-like cleanup must-gather/extractor packages refact/must-gather: isolate and tune leaky bucket processor cmd/run: fix flag to use full image path to allow CI presubmits review the result filter pipeline OPCT-292: cmd/publish - add experimental publish used in CI Add a experimental command to allow Prow CI to publish results in the opct-storage (S3) without needing to install dependencies in the test step, preventing failures. OPCT-292: cmd/publish - add metadata option when publishing artifact doc: add review/support documentations feat: upload baseline artifact on s3 report/feat: opct adm baseline (get|list|indexer|publish) review baseline reitrieving summary/baseline from service/API feat: add replay to filter pipeline feat: introduce adm setup-ndoe to helper in tests review plugin filter order fix checks to good tresholds
mtulio
force-pushed
the
devel-report-html
branch
from
50fe32e to
c5d9ae5
Compare
mtulio
added a commit
to mtulio/provider-certification-tool
that referenced
this pull request
Aug 8, 2024
This change introduce several improvements in the UX while reviewing the report by: - creationg an intuitive HTML report allowing users to quickly see issues and navigate to the logs for each test failure - introduce several gates/SLO/checks to be used as post-processor and get better visibility in the results, based in existing knowledge base/CI data or external ssytems See the PR with details of improvements: redhat-openshift-ecosystem#76 ---- fix sippy query in the flake filter create a rank of tags, adding percentage of counters create html report saving data to json improve report html add log links and rank by error count update camgi tab add waiting interval support status: improve message field with plugin phase report: generating filter.html file with failures table renameing vfs path for html templates report html - update rank and output as json using report as data source of CLI report; apply docs add timers to the runtime report update report collecting more env information report: update html templates to improve filters adding suite errors menu Update suite error menu using native js create a rank of error counters adding an option to extract errors from must-gather extracting must-gather event filters adding a tab to the report adding the structure of report checks introducing dynamic checks moving report to summary Adding checks with acceptance values from baseline adding alert pop to Checks menu parallelism processing must-gather, loweing 3 times proc time adding support to all plugins in report http file server to serve report review: documenting and linking check rules reorg mkdocs/todo report: embeding etcd error parser to json report report/must-gather: consolidating etd parser report: supporting etcd, network check and embed data remove codegen script / not used distributing report to packages parsing meta/run.log extracting runtime information introducing new parsers: opct and meta config renaming sippy to its package renaming packages and cleanup review report html and threasholds review timers for plugins add etcd checks ; cleanup review thresholds bumping plugins image review documentation and doc strings update check platform type fix tests in tags fix dev docs dev/report: adding initial flow of report Fix Containerfile for CI when releasing Fix windows build update makefile to fix windows build isolating status watch interval from redhat-openshift-ecosystem#87 reverting unrelated docs changes improving unit tests adding metaconfig unit tests and test data adding tests for meta-run.log parser adding parsers and tests for config maps fixes in the opct metrics report gen metrics report adding sample document to generate batch reports Review checks to support attributes review report to improve checks increasing the error pattern information for etcd taking notes for report dev doc adding plugin log extraction and link to the plugin name collecting node info extracting install-config review report html and metrics with adm parse cmd adding charts with ploty remove comments when plotting fix log save opct adm parse-etcd-logs to quickly access parsed logs (#4) increasing documentation coverage bump to use new quay.io org create target directory before extractors rename title of checks on report cli fixes in the report check prevent empty data fix/report/cli: show other plugin than k8s report/check: 012 - check plugin failure fixes after rebase rename namespace Supporting PDB to opct server add new gen openshift-tests plugin based in go allow custom openshift-tests image to devel w/ kind working version with entrypoint for plugin plugin manifests fixes to support remote entrypoint to tests image update plugin manifests to new plugin version refact CLI UI tables to enhance results intro yamllint and fixing yaml assets/manifests plugins working version the default flow enhance pre-run checks for missing config create local log file for all levels updatem plugins according to the latest goplugin version add collector image to the manifest template review packages / moving to report package using ETL-like cleanup must-gather/extractor packages refact/must-gather: isolate and tune leaky bucket processor cmd/run: fix flag to use full image path to allow CI presubmits review the result filter pipeline OPCT-292: cmd/publish - add experimental publish used in CI Add a experimental command to allow Prow CI to publish results in the opct-storage (S3) without needing to install dependencies in the test step, preventing failures. OPCT-292: cmd/publish - add metadata option when publishing artifact doc: add review/support documentations feat: upload baseline artifact on s3 report/feat: opct adm baseline (get|list|indexer|publish) review baseline reitrieving summary/baseline from service/API feat: add replay to filter pipeline feat: introduce adm setup-ndoe to helper in tests review plugin filter order fix checks to good tresholds
mtulio
force-pushed
the
devel-report-html
branch
from
c5d9ae5 to
df9de61
Compare
mtulio
added a commit
to mtulio/provider-certification-tool
that referenced
this pull request
Aug 8, 2024
This change introduce several improvements in the UX while reviewing the report by: - creationg an intuitive HTML report allowing users to quickly see issues and navigate to the logs for each test failure - introduce several gates/SLO/checks to be used as post-processor and get better visibility in the results, based in existing knowledge base/CI data or external ssytems See the PR with details of improvements: redhat-openshift-ecosystem#76 ---- fix sippy query in the flake filter create a rank of tags, adding percentage of counters create html report saving data to json improve report html add log links and rank by error count update camgi tab add waiting interval support status: improve message field with plugin phase report: generating filter.html file with failures table renameing vfs path for html templates report html - update rank and output as json using report as data source of CLI report; apply docs add timers to the runtime report update report collecting more env information report: update html templates to improve filters adding suite errors menu Update suite error menu using native js create a rank of error counters adding an option to extract errors from must-gather extracting must-gather event filters adding a tab to the report adding the structure of report checks introducing dynamic checks moving report to summary Adding checks with acceptance values from baseline adding alert pop to Checks menu parallelism processing must-gather, loweing 3 times proc time adding support to all plugins in report http file server to serve report review: documenting and linking check rules reorg mkdocs/todo report: embeding etcd error parser to json report report/must-gather: consolidating etd parser report: supporting etcd, network check and embed data remove codegen script / not used distributing report to packages parsing meta/run.log extracting runtime information introducing new parsers: opct and meta config renaming sippy to its package renaming packages and cleanup review report html and threasholds review timers for plugins add etcd checks ; cleanup review thresholds bumping plugins image review documentation and doc strings update check platform type fix tests in tags fix dev docs dev/report: adding initial flow of report Fix Containerfile for CI when releasing Fix windows build update makefile to fix windows build isolating status watch interval from redhat-openshift-ecosystem#87 reverting unrelated docs changes improving unit tests adding metaconfig unit tests and test data adding tests for meta-run.log parser adding parsers and tests for config maps fixes in the opct metrics report gen metrics report adding sample document to generate batch reports Review checks to support attributes review report to improve checks increasing the error pattern information for etcd taking notes for report dev doc adding plugin log extraction and link to the plugin name collecting node info extracting install-config review report html and metrics with adm parse cmd adding charts with ploty remove comments when plotting fix log save opct adm parse-etcd-logs to quickly access parsed logs (#4) increasing documentation coverage bump to use new quay.io org create target directory before extractors rename title of checks on report cli fixes in the report check prevent empty data fix/report/cli: show other plugin than k8s report/check: 012 - check plugin failure fixes after rebase rename namespace Supporting PDB to opct server add new gen openshift-tests plugin based in go allow custom openshift-tests image to devel w/ kind working version with entrypoint for plugin plugin manifests fixes to support remote entrypoint to tests image update plugin manifests to new plugin version refact CLI UI tables to enhance results intro yamllint and fixing yaml assets/manifests plugins working version the default flow enhance pre-run checks for missing config create local log file for all levels updatem plugins according to the latest goplugin version add collector image to the manifest template review packages / moving to report package using ETL-like cleanup must-gather/extractor packages refact/must-gather: isolate and tune leaky bucket processor cmd/run: fix flag to use full image path to allow CI presubmits review the result filter pipeline OPCT-292: cmd/publish - add experimental publish used in CI Add a experimental command to allow Prow CI to publish results in the opct-storage (S3) without needing to install dependencies in the test step, preventing failures. OPCT-292: cmd/publish - add metadata option when publishing artifact doc: add review/support documentations feat: upload baseline artifact on s3 report/feat: opct adm baseline (get|list|indexer|publish) review baseline reitrieving summary/baseline from service/API feat: add replay to filter pipeline feat: introduce adm setup-ndoe to helper in tests review plugin filter order fix checks to good tresholds
mtulio
force-pushed
the
devel-report-html
branch
from
df9de61 to
3f7c678
Compare
mtulio
added a commit
to mtulio/provider-certification-tool
that referenced
this pull request
Aug 8, 2024
This change introduce several improvements in the UX while reviewing the report by: - creationg an intuitive HTML report allowing users to quickly see issues and navigate to the logs for each test failure - introduce several gates/SLO/checks to be used as post-processor and get better visibility in the results, based in existing knowledge base/CI data or external ssytems See the PR with details of improvements: redhat-openshift-ecosystem#76 ---- fix sippy query in the flake filter create a rank of tags, adding percentage of counters create html report saving data to json improve report html add log links and rank by error count update camgi tab add waiting interval support status: improve message field with plugin phase report: generating filter.html file with failures table renameing vfs path for html templates report html - update rank and output as json using report as data source of CLI report; apply docs add timers to the runtime report update report collecting more env information report: update html templates to improve filters adding suite errors menu Update suite error menu using native js create a rank of error counters adding an option to extract errors from must-gather extracting must-gather event filters adding a tab to the report adding the structure of report checks introducing dynamic checks moving report to summary Adding checks with acceptance values from baseline adding alert pop to Checks menu parallelism processing must-gather, loweing 3 times proc time adding support to all plugins in report http file server to serve report review: documenting and linking check rules reorg mkdocs/todo report: embeding etcd error parser to json report report/must-gather: consolidating etd parser report: supporting etcd, network check and embed data remove codegen script / not used distributing report to packages parsing meta/run.log extracting runtime information introducing new parsers: opct and meta config renaming sippy to its package renaming packages and cleanup review report html and threasholds review timers for plugins add etcd checks ; cleanup review thresholds bumping plugins image review documentation and doc strings update check platform type fix tests in tags fix dev docs dev/report: adding initial flow of report Fix Containerfile for CI when releasing Fix windows build update makefile to fix windows build isolating status watch interval from redhat-openshift-ecosystem#87 reverting unrelated docs changes improving unit tests adding metaconfig unit tests and test data adding tests for meta-run.log parser adding parsers and tests for config maps fixes in the opct metrics report gen metrics report adding sample document to generate batch reports Review checks to support attributes review report to improve checks increasing the error pattern information for etcd taking notes for report dev doc adding plugin log extraction and link to the plugin name collecting node info extracting install-config review report html and metrics with adm parse cmd adding charts with ploty remove comments when plotting fix log save opct adm parse-etcd-logs to quickly access parsed logs (#4) increasing documentation coverage bump to use new quay.io org create target directory before extractors rename title of checks on report cli fixes in the report check prevent empty data fix/report/cli: show other plugin than k8s report/check: 012 - check plugin failure fixes after rebase rename namespace Supporting PDB to opct server add new gen openshift-tests plugin based in go allow custom openshift-tests image to devel w/ kind working version with entrypoint for plugin plugin manifests fixes to support remote entrypoint to tests image update plugin manifests to new plugin version refact CLI UI tables to enhance results intro yamllint and fixing yaml assets/manifests plugins working version the default flow enhance pre-run checks for missing config create local log file for all levels updatem plugins according to the latest goplugin version add collector image to the manifest template review packages / moving to report package using ETL-like cleanup must-gather/extractor packages refact/must-gather: isolate and tune leaky bucket processor cmd/run: fix flag to use full image path to allow CI presubmits review the result filter pipeline OPCT-292: cmd/publish - add experimental publish used in CI Add a experimental command to allow Prow CI to publish results in the opct-storage (S3) without needing to install dependencies in the test step, preventing failures. OPCT-292: cmd/publish - add metadata option when publishing artifact doc: add review/support documentations feat: upload baseline artifact on s3 report/feat: opct adm baseline (get|list|indexer|publish) review baseline reitrieving summary/baseline from service/API feat: add replay to filter pipeline feat: introduce adm setup-ndoe to helper in tests review plugin filter order fix checks to good tresholds
mtulio
force-pushed
the
devel-report-html
branch
from
3f7c678 to
be60edf
Compare
|
This version is accepted. Running it to parse archives recently rehearsed on CI[1][2](which includes plugin upgrades) I am seeing observing results. This PR, and the plugin refact one, are ready to move on. cc @rvanderp3 [1] https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_release/54718/rehearse-54718-periodic-ci-redhat-openshift-ecosystem-provider-certification-tool-main-4.15-platform-none-vsphere/1821048927051321344/artifacts/platform-none-vsphere/provider-certification-tool-results/artifacts/ |
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
mtulio
added
kind/feature
The report feature introduces several improvements in the UX while reviewing the report by: - creationg an intuitive HTML report allowing users to quickly see issues and navigate to the logs for each test failure - introduce several gates/SLO/checks to be used as post-processor and get better visibility in the results, based in existing knowledge base/CI data or external ssytems - providing a better CLI UI exploring results See the PR with details of improvements: redhat-openshift-ecosystem#76
mtulio
force-pushed
the
devel-report-html
branch
from
be60edf to
c02c85c
Compare
mtulio
added a commit
that referenced
this pull request
Aug 8, 2024
The report feature introduces several improvements in the UX while reviewing the report by: - creationg an intuitive HTML report allowing users to quickly see issues and navigate to the logs for each test failure - introduce several gates/SLO/checks to be used as post-processor and get better visibility in the results, based in existing knowledge base/CI data or external ssytems - providing a better CLI UI exploring results See the PR with details of improvements: #76
|
Moving forward this feature. ✅ The CI tests is showing good results with rehearsal jobs:
This feature is also exercised by internal teams and partners since Aug 2023 with |
mtulio
merged commit f321a27
into
redhat-openshift-ecosystem:main
13 of 14 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
opct reportprovides several improvements in the UX while reviewing the conformance results archive using OPCT, such as:issues and navigate to the logs for each test failure
get better visibility in the results, based on existing knowledge
base/CI data or external systems
Changes (overview)
Improvements
report: now the counters are displaying the percentage of it (when compared with the total). So it can quickly have one idea of failures - in general, we expect lower than 1% of failures in a regular installation for OpenShift Conformancereport: a headline with grouped (by test tags) occurrences is displayed before each failed list for both conformance pluginsrun --watch --watch-interval: Status can set custom watch interval to decrease the number of logs in CIreport --diff: created as an alias of--baseline, setting it as deprecatedreportis now showing the "Checks" section, applying some rules to help the reviewer where to focus, and prevent submitting results to the partner support case prematurely.Error counters
report/Plugins: search for failures by pattern in the plugin logs and aggregate it as "Suite Errors"report/Must-gather: search for failures by pattern in the pod logs (must-gather), aggregating it as "Workload Errors"HTML report
report: extracts important information, processes it, and saves it to a local directory to be served by a local web server, allowing one to quickly navigate to the failures for each test using a browser.report: rank by error countreportHTML: scrapes the test documentation for Kuebrnetes Conformance, shows the link for each item when it is available in the Kubernetes suitesreportmenu "Workload errors": has been added showing information about the pod logs countersreportmenu "Suite errors": has been added showing information about the pod logs countersreportmenu "Checks": has been added showing information about the result checklistreporttab "CAMGI": redirects to CAMGI static HTML page extracted from must-gather, when it is present, otherwise shows how to use CAMGI when it is not processed by the pluginreporttab "Filter": redirects to a static HTML page with all tests allowing the user to explore the test detailsreporttab "Events": redirects to a static HTML page with events created by Must-gather (extracted in the runtime)Many other features in the WebUI.
Plugin Runtime
status: replace the messagewaiting for post-processor...tocompletewhen the pod is finished.openshift-tests-plugin) has been refactored to Golang, especially targeting:testsimage as a sidecar container, preventing plugin development when new dependencies are required byopenshift-testsReplayBug fixes
report: Flake tests now is querying to correct OCP version on Sippy APIreport: Flake Filter report prevents duplicated tests from immediate actionreport: Flake Filter report does not show items reporting than 5% of flake count in OCP CI (by Sippy)Done checklist
reportreportchecklist sectionDocumentation checklist
OPCT-226: Introduce knowledge document for OPCT Report Rules #77
OPCT-226: Added check rules for etcd #80
report: OPCT-257: docs review documents for v0.5 report/review feature #113adm baseline *: OPCT-257: docs review documents for v0.5 report/review feature #113reportoptions: OPCT-257: docs review documents for v0.5 report/review feature #113