Testcase titles and testcase descriptions

[Paebbels]

In a discussion with a customer (trainee in an OSVVM class and PoC contributor), the following missing feature has been identified:

With a growing set of test cases, it's hard to describe what are the differences between test cases; especially, when a multidimensional matrix of parameters is tested.

Example:

complex_test_cfg1_cond_2_scenario_4_cornercase7.vhd
complex_test_cfg2_cond_5_scenario_2_cornercase1.vhd

Currently, test cases can only be described by their test case name. This name is quite technical and doesn't allow spaces or special characters as it's limited to the VHDL identifier naming limits.

This proposal is about adding an optional title (string) and an optional description (multi-line string) to each test case.

Source code example:

RunTest myTestcase.vhdl \
  [ generic BITS 8 ] \
  [ title {This is my special testcase} ] \
  [ description {This is
a multi-line
description text
later visible in the report.} ]

It would be beneficial to use these strings in HTML reporting.
This additional data could be easily emitted into the YAML files.

Advantages:

• Integration of test plan items into the final report.
• No need to use an additional external document for decoding TestIDs into descriptions in big test environments, because test names can't summarize the complexity of a test.

@JimLewis what do you think about that proposal?

---

/cc @gmartina

[Paebbels]

A title could also be used to control OSVVMs auto generated testcase names for subtests (testcases with generics).

The following example creates a list with 2 dictionaries. The dict values are handed over to the generic procedure, but are used in parallel to construct a nice formatted title.

Alternatively, a locally defined function could be used, which takes the dictionary and returns the title.

Example:

set mylist [list \
    [dict create \
      Bits 8 \
      Debug True \
      Description "some description 1"] \
    [dict create \
      Bits 16 \
      Debug False \
      Description "some more info 2"] \
]

foreach entry $mylist {
    # This automatically creates variables $Bits, $Debug, and $Description
    dict with entry {
        RunTest myTestcase.vhdl \
            [ generic BITS $Bits ] \
            [ generic Debug $Debug ] \
            [ title "myTestcase bits=$Bits, debug=$Debug - $Description" ]
    }
}

I'm not a TCL programmer. This code was constructed by CI.

[gmartina]

Another solution would be to add API functions like:

  TestProc : process
  begin
    -- Set test name (standard OSVVM practice)
    SetTestName("TestDescription_Demo") ;
    
    -- NEW FEATURE: Set a human-readable description for this test
    -- This description can contain spaces and special characters
    -- and will appear in HTML reports
    SetTestDescription("Demonstrates the test description and tag feature for documenting complex test scenarios with multiple configurations") ;
    
    -- NEW FEATURE: Set configuration tags as key-value pairs
    -- These help categorize tests in a multi-dimensional matrix
    SetTestTag("Engine Mode", "Cruise") ;
    SetTestTag("Altitude", "35000 ft") ;
    SetTestTag("Thrust Setting", "75%") ;
    SetTestTag("Channel", "Channel A") ;
    SetTestTag("Test Type", "Functional") ;
    SetTestTag("Priority", "High") ;
    SetTestTag("Subsystem", "Autopilot") ;
    SetTestTag("Engine Condition", "Nominal") ;
    SetTestTag("Fuel Type", "Jet A-1") ;
    SetTestTag("Payload", "Standard") ;
    SetTestTag("Flight Phase", "Cruise") ;
    SetTestTag("Navigation Mode", "LNAV") ;
    SetTestTag("Autopilot Status", "Engaged") ;

[Paebbels]

There are properties in JUnit XML implemented as key-value-pairs, that could save these information.

<testcase ....>
  <properties>
    <property ...>
    <property ...>
    <property ...>
  </properties>
</testcase>

[gmartina]

I was thinking in something like this. Or maybe using a table for the tags.

[JimLewis]

I like the Test Description in the VHDL code. In many ways it would add to the test case documentation - ie: replace the description of the test case at the top of the file.

In your picture above, I see the Description change based on a generic - does this mean the test case conditionally chooses which to include?

It would be easy to add these to the scripts, but I don't think I like the separation of the VHDL test case and the test description. The script files would be ugly.

For TestDescription and TestTags (particularly what your tags imply), is there a use model where we prove or demonstrate that these settings were the ones actually applied - the old problem of comments not updated when the code is updated.

Looks like you are proposing that the TestDescription goes in both the Build Summary (top level) and Test Case reports, but the test tags only go in the Test Case reports.

[JimLewis]

WRT JUnit XML, it will be put into the YAML files first and then propagated to the HTML and JUnit XML files.

Does the JUnit reporters actually do anything with this information?

[Paebbels]

I think the tags are a description like generics, but some times, tests don't use toplevel generics but still should identify what was a variant.

In OSVVM, you have basic tests and burst tests and option tests. These tests could be grouped or filtered by post-processing tools based on the tags.

Does the JUnit reporters actually do anything with this information?

The offline reporters not.

I know CodeCov has working on online test report inspection and adding advanced filtering.

[Paebbels]

For the new API I propose:

• SetTestTitle(title: string)
• SetTestDescription(description: string)
• AddTestTag(key: string; value: boolean)
• AddTestTag(key: string; value: integer)
• AddTestTag(key: string; value: real)
• AddTestTag(key: string; value: string)

Please note, YAML has native support for these types, thus they shouldn't be converted to string for YAML.

Maybe SetTestName should be extended, so a single call is enough:

• SetTestName(name: string; title: string := ""; description: string := "")

[JimLewis]

What is TestTitle and how does it differ than TestDescription? I see where TestDescription goes, but not TestTitle.

Do you have a use model for TestTag being anything other than type string? They seem to be informational in nature.

[Paebbels]

A test title would override the test name if it's set, but test name is needed as an ID in YAML, XML and other places, because a title (prose text) doesn't fulfill the requirements for an identifier.

A title is a short description / one-liner, while the description is a multi-line string capable of handling multiple paragraphs. Usually, data structures with human readable content distinguish between a title and a description. Moreover a description could be given as embedded Markdown or ReStructuredText, thus a description can be formatted and embedded into automated test reports.

A title can be used in the summary table of all testcases in a test suite, while a title and description is useful on the details page.

Other alternative names are brief or summary.

Do you have a use model for TestTag being anything other than type string?

There doesn't need to be a use model. It's about correct typing of data. Ensuring typed data within VHDL is transferred without information loss (loss of type information) to YAML is essential. 1 is not identical to "1" and not identical to 1.0. In a simplest form it ensures correct ordering of items in a list because number ordering is different from alphabetical ordering. The implementation is very simple in VHDL: just a few overloads.

[JimLewis]

@Paebbels,
On the Tcl side, in answer to "How does Tcl discriminate types?"

Tcl doesn't strictly "discriminate types" like statically typed languages; instead, it's a dynamically typed language where everything is fundamentally a string, but it intelligently interprets data based on context, command usage, and internal type hints (like integer, double, list) for efficient operations, only performing actual type checks or conversions when a specific command (like incr, expr) needs it, preventing ambiguity by using syntax like braces {} for literal strings and parentheses () for command execution.

Basically, if it has a value other than string, it will be converted to string when putting it into the YAML. That way when it is rendered into HTML, the array just needs to be iterated to convert it to HTML and JUnit xml.

If you had a value other than string in YAML, what would you do with that value it?

[Paebbels]

The proposed tags are written by VHDL to YAML, not by TCL. Thus the TCL type system is not involved in that process, right?

JSON, YAML and other file formats distinguish between basic types:

tags:
  boolean: true
  integer: 5
  double:  5.0
  string:  "5"
  empty:   # alternatively null

If you had a value other than string in YAML, what would you do with that value it?

As said before, I can filter, iterate, group and sort testcases based on a predicate. For sorting to work properly, numbers need to stay numbers, otherwise sorting will be broken. An alphabetic ordering is not equal to a numeric ordering.