[test_operator] Add stackviz integration for Tempest test visualization#3642
[test_operator] Add stackviz integration for Tempest test visualization#3642sauragar wants to merge 1 commit intoopenstack-k8s-operators:mainfrom
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/9eebbe61675c4dcc9d088d2c2d5ea48d ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 37m 18s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/8f1eb8e960fb4d9e8ca4f4a98fc80cda ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 19m 50s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/11f7659c426842b28edff49ed1d8f0cf ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 11m 41s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/75bb82277de440b3ac6855780f9c6fa9 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 09m 12s |
sauragar
force-pushed
the
OSPRH-12077
branch
3 times, most recently
from
4000706 to
a8b3e36
Compare
|
PR Summary: Fix Stackviz Report Generation for Tempest Tests Problem Statement The stackviz integration for visualizing Tempest test results was not generating reports in Zuul CI runs. Investigation
Solution Overview This PR fixes the stackviz integration through 13 commits that address all identified issues and improve documentation: Core Fixes (5 commits)
Cleanup & Documentation (8 commits)
Foundation (3 commits)
Configuration Variables Four new variables control stackviz behavior (all documented in roles/test_operator/README.md):
Files Modified Core functionality:
Documentation:
Files Deleted
Testing & Verification ✅ Verified in Zuul CI:
Expected Output After Tempest tests complete, users will find: Single test stage: Multiple test stages (workflows): Report Features The generated stackviz HTML reports include: 📊 Core Visualization:
🎨 Theme Support:
🔍 Interaction:
Benefits
Technical Highlights Python Script Improvements:
Ansible Role Improvements:
Related Issues
|
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/1ad35fe4e1874a5d995b1b081965dc9a ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 18m 11s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/7f226ed574e54971acc665beca6f4750 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 05m 33s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/3f99653ac2a64483bf6d1550aeaf7f6a ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 27s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/d8974b19497244cc8715428016054050 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 07m 15s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/6f2107258d6941c793b79b9718e453e3 ❌ openstack-k8s-operators-content-provider FAILURE in 6m 45s |
|
recheck |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/f9b15630dea8425684d21ce011dd8cf3 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 17m 07s |
sauragar
force-pushed
the
OSPRH-12077
branch
from
137e128 to
85e40e5
Compare
sauragar
force-pushed
the
OSPRH-12077
branch
from
e69acc8 to
46f7d4f
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/7359cdd1f98c4abc8ae2be9866549620 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 51m 04s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/a8f037a31df54b2d8233f253618cee1f ❌ openstack-k8s-operators-content-provider NODE_FAILURE Node request 100-0008160431 failed in 0s |
sauragar
force-pushed
the
OSPRH-12077
branch
from
6dccf3d to
f6d67d3
Compare
| when: subunit_gz_files.files | length > 0 | ||
| block: | ||
| # Dependency checking (done once, not per file) | ||
| - name: Check if Python script exists |
There was a problem hiding this comment.
TBH don't think if the script exists is needed.
| become: true | ||
| when: cifmw_test_operator_stackviz_auto_install_deps | ||
|
|
||
| - name: Verify stackviz dependencies are available |
There was a problem hiding this comment.
We do not make such test to other libraries, so don't know why it needs to be here.
So if you can remove, would be good.
| failed_when: false | ||
| register: stackviz_deps_check | ||
|
|
||
| - name: Fail if stackviz dependencies are not available |
There was a problem hiding this comment.
same - to remove as a comment above.
| mode: '0644' | ||
| when: false # Template will be created separately | ||
|
|
||
| - name: Generate simple summary index HTML |
There was a problem hiding this comment.
please don't treat copy/content as templates. Here is definitely template.
Please move it to be as a template.
| _base_filename: "{{ 'tempest_retry' if 'tempest_retry' in subunit_file.path else 'tempest_results' }}" | ||
| _html_filename: "{{ 'tempest_retry_viz.html' if 'tempest_retry' in subunit_file.path else 'tempest-viz.html' }}" | ||
|
|
||
| - name: Display processing information |
There was a problem hiding this comment.
so the index is only needed to... show debug message? In that case it is not even helpful. If other people will agree, I suggest to remove subunit_index and all relation to it.
| _current_subunit: "{{ _current_dir }}/{{ _base_filename }}.subunit" | ||
| _current_html: "{{ _current_dir }}/{{ _html_filename }}" | ||
|
|
||
| - name: Decompress subunit file if compressed |
There was a problem hiding this comment.
why it can be compressed? I don't believe in that phase it is compressed, or?
| {{ _current_subunit }} | ||
| {{ _current_html }} | ||
| register: _current_stackviz_generation | ||
| when: _current_subunit_stat.stat.exists |
There was a problem hiding this comment.
so if we have a task with _current_subunit_stat.stat.exists maybe we should create a fail task with ignore_errors where we notify end user that file might be missing?
| - _current_stackviz_generation is not skipped | ||
| - cifmw_test_operator_stackviz_debug | default(false) | ||
|
|
||
| - name: Verify HTML report was created |
There was a problem hiding this comment.
oh wait. If generate-stackviz-report.py finish with exit code other than 0, it should fail, right?
In that case, we should assume that the file exists, so checking if file exists where it should exists does not make any sense.
scripts/generate-stackviz-report.py
Outdated
| test["end_time"] = ts | ||
| # If we missed the start time, try to infer or leave incomplete | ||
| if not test["start_time"]: | ||
| # Some streams might not have explicit inprogress? |
scripts/generate-stackviz-report.py
Outdated
| if test_status: | ||
| # Map statuses | ||
| # success, fail, skip are the main ones we care about | ||
| if test_status == "inprogress": |
There was a problem hiding this comment.
do you think it might happen?
scripts/generate-stackviz-report.py
Outdated
| try: | ||
| content = bytes(file_bytes).decode("utf-8", errors="replace") | ||
| test["details"].append(content) | ||
| except: |
There was a problem hiding this comment.
what exception type? Does pep8/flake8 pass?
scripts/generate-stackviz-report.py
Outdated
| data = accumulator.get_results() | ||
|
|
||
| # Use embedded template | ||
| template = REPORT_TEMPLATE |
There was a problem hiding this comment.
nit: suggest to move REPORT_TEMPLATE into separate play, then read and write than have a html template in python code.
sauragar
force-pushed
the
OSPRH-12077
branch
from
f6d67d3 to
6fde0aa
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/0b83fa5ec44a43e0bd59c6c58e12d102 ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 50m 50s |
sauragar
force-pushed
the
OSPRH-12077
branch
from
6fde0aa to
8dfa867
Compare
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/8282119ae1c647a18ef2ba3aa4960298 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 17m 33s |
Add comprehensive stackviz integration for Tempest test visualization with support for multi-stage test execution. This generates interactive HTML reports with test results, timing analysis, and failure details. Key features: - Multi-stage stackviz report generation for main and worker jobs - Dark/light mode toggle for better viewing experience - Separate stackviz reports for tempest retry results - Support for both compressed and uncompressed subunit files - Auto-install of RPM dependencies (python3-subunit, python3-testtools) - Summary index page with links to all generated reports Implementation details: - Uses cifmw_repo variable instead of playbook_dir to reference the stackviz generation script, fixing failures in adoption jobs where playbooks are executed from rdo-jobs repository - Simplified dependency installation using single task with RPM packages - Removes unnecessary step-X directory wrapper in log collection - Uses FQCN (ansible.builtin.include_tasks) for ansible-lint compliance - All stackviz variables follow cifmw_test_operator_stackviz_* naming Configuration variables (all in defaults/main.yml): - cifmw_test_operator_stackviz_generate: Enable/disable report generation - cifmw_test_operator_stackviz_auto_install_deps: Auto-install RPM packages - cifmw_test_operator_stackviz_create_index: Create summary index page - cifmw_test_operator_stackviz_debug: Enable debug output Reviewer feedback addressed: Ansible Tasks: - Remove redundant Python script existence check - Remove dependency verification tasks (let Python fail with ImportError) - Move inline HTML content to proper template file (stackviz-index.html.j2) - Remove unused subunit_index from loop control and debug messages - Add comment explaining compressed .subunit.gz files - Add explicit fail task for missing subunit files with clear error message - Remove redundant HTML report verification (Ansible fails on non-zero exit) Python Script Improvements: - Add shebang: #!/usr/bin/env python3 - Remove questionable comments and dead code for 'inprogress' status - Fix bare except clause to catch specific exceptions - Add warning to stderr for tests with missing status - Refactor timestamp formatting into helper function - Move 800-line HTML template to external file (stackviz-report-template.html) Files created: - roles/test_operator/tasks/generate-stackviz-main.yml - roles/test_operator/tasks/generate-stackviz-worker.yml - roles/test_operator/templates/stackviz-index.html.j2 - scripts/generate-stackviz-report.py - scripts/stackviz-report-template.html All changes improve code maintainability while maintaining backward compatibility with fallback to embedded template if external file missing. Signed-off-by: Saurabh Agarwal <sauragar@redhat.com> Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: sauragar <sauragar@redhat.com>
sauragar
force-pushed
the
OSPRH-12077
branch
from
8dfa867 to
9ae4e41
Compare
| _current_stage_name: "{{ subunit_file.path | dirname | basename }}" | ||
| _current_is_compressed: "{{ subunit_file.path.endswith('.gz') }}" | ||
|
|
||
| - name: Determine file type and output naming |
There was a problem hiding this comment.
nit: this task can be merged with one above
|
|
||
| # Process each subunit file | ||
| - name: Process each subunit file and generate individual reports | ||
| ansible.builtin.include_tasks: generate-stackviz-worker.yml |
There was a problem hiding this comment.
now I see, that in generate-stackviz-worker.yml you are setting facts.
TBH, would be better to make vars, instead of additional tasks with set_facts if that is not needed.
Especially, that those vars are only needed for this task.
For example:
- name: Process each subunit file and generate individual reports
vars:
_current_subunit_source: "{{ subunit_file.path }}"
_current_dir: "{{ subunit_file.path | dirname }}"
_current_stage_name: "{{ subunit_file.path | dirname | basename }}"
_current_is_compressed: "{{ subunit_file.path.endswith('.gz') }}"
_is_retry_file: "{{ 'tempest_retry' in subunit_file.path }}"
_base_filename: "{{ 'tempest_retry' if 'tempest_retry' in subunit_file.path else 'tempest_results' }}"
_html_filename: "{{ 'tempest_retry_viz.html' if 'tempest_retry' in subunit_file.path else 'tempest-viz.html' }}"
ansible.builtin.include_tasks: generate-stackviz-worker.yml
loop: "{{ subunit_gz_files.files }}"
loop_control:
loop_var: subunit_file
then in generate-stackviz-worker.yml leave:
- name: Set output paths for decompressed file and HTML report
ansible.builtin.set_fact:
_current_subunit: "{{ _current_dir }}/{{ _base_filename }}.subunit"
_current_html: "{{ _current_dir }}/{{ _html_filename }}"
as it was
5 participants
Add automatic stackviz report generation for Tempest tests with comprehensive dependency checking and optional auto-installation.
New files:
Modified files:
Features:
Signed-off-by: Saurabh Agarwal sauragar@redhat.com