Code coverage with NVC produces no output

[a-panella]

Hi, hopefully this is not user error...

issue

no code coverage outputs are generated when enabling code coverage

the output I observe

Three directories are created in following structure:

• Code coverage
  • DefaultLib
  • scripts_top_tb
    ...but the directories appear to be totally empty

the output I expect

The directories should have code cov or similar .xml files with which to display

extra info

It seems OSVVM is aware that code coverage is desired, as when I check the .html for the script, CodeCoverage is available as an option to click, but the code coverage html file does not exist, so clicking it results in a 404

I have a simple .pro script that attempts to turn code coverage on for NVC for some simple tests (no testsuites)

SetCoverageEnable
SetCoverageAnalyzeEnable true
SetDebugMode true
SetSaveWaves true
SetLogSignals true
SetVHDLVersion 2008

library some_lib

ChangeWorkingDirectory ../fw/src
analyze synth_pkg.vhd

ChangeWorkingDirectory ../sim
analyze bufgmux_ctrl_mock.vhd

ChangeWorkingDirectory ../src

analyze s2f_cdc.vhd
analyze cdc_vector.vhd
analyze test.vhd
analyze top.vhd

SetCoverageAnalyzeEnable false
SetCoverageSimulateEnable true
ChangeWorkingDirectory ../sim

analyze top_tb_in_sync_test.vhd
TestName top_tb_in_sync_test
simulate top_tb_in_sync_test

analyze top_tb_almost_in_sync_test.vhd
TestName top_tb_almost_in_sync_test
simulate top_tb_almost_in_sync_test

analyze top_tb_out_of_sync_test.vhd
TestName top_tb_out_of_sync_test
simulate top_tb_out_of_sync_test
SetCoverageSimulateEnable false

Possibly related to #43 if std.env.stop(GetAlertCount) is required for code coverage

[JimLewis]

Code coverage for NVC is still a work in progress. If you know the command options I can get it setup - otherwise I will talk to Nick

[a-panella]

Thanks for replying quickly. Looks like when elaborating, the coverage types can be specified like so

nvc -e --cover=statement,branch,toggle,expression <top>

Where I guess you could fetch the coverage options from your OsvvmScript GetCoverageSimulateOptions
It looks like code coverage is an elaboration only thing for NVC. I'm not sure if this is correct, but I suppose this is different from other simulators that support coverage, as you provide SetCoverageAnalyzeEnable.

Looks like you can merge the coverage from each test case too, like so:

nvc -c --merge=merged.covdb --report=<path_to_folder_for_html_report> first.covdb second.covdb third.covdb ...

That is about as much as I can glean from the documentation, I have not tried nvc code coverage feature natively.

More information can be found from Nick's website https://www.nickg.me.uk/nvc/manual.html#CODE_COVERAGE

Thanks for your work on Osvvm

[a-panella]

Update: I have managed to get code coverage results to generate to std output stream, however, there seems to be some issues with writing the output results to a named file. I am uncertain if this is a permission issue in the directory and will revisit this in a few hours. I will make a PR later if I can make it work nicely with OsvvmScripts

[JimLewis]

Hi Aaron
File access issues is often due to no file close on the previous run.

First check your code for file close.

Sometimes this happens due to a previous code version not running to completion.

In either case when developing I have had to delete files from previous runs in the OS to get the next run able to run.

Some times the requires a 2 level call mechanism

Open file
Use catch to run the action procedure
Close file 
Signal errors - depending on setting this may terminate the build - but generally for support stuff this is ignored.

[a-panella]

Thanks, the issue preceeds OsvvmScripts - it is nvc that is apparently not writing an output coverage file to disk, I'm looking at the source now to try and determine if its an oddity in my environment or nvc itself

[JimLewis]

What I have run into is the new file can't be created if the old file exists and is still open.

[a-panella]

Okay, some more info here. I think the manual for NVC was slightly out of date, as the documentation for usage of code coverage appears to have changed since I last accessed Nick's website.

Last week, the command was listed as

nvc --work=mylib -e --cover dut_tb -r --cover-export --format=cobertura -o some_name.xml

but this generated no xml files of the input name, and instead generated a covdb with the name _MYLIB.TB_NAME.elab.covdb

The example usage has now changed to

nvc --work=mylib -e --cover dut_tb -r

where I guess you are no longer supposed to supply a name for the coverage database.

You then merge each .elab.covdb using

nvc -c --merge=merged.covdb --report=cov mylib/_MYLIB.DUT_TB.elab.covdb

So I guess OsvvmScripts can just run the -c command as a post script and then your TestReport html can just point to the generated coverage html file. I'm not sure what you're currently working on for Osvvm - probably more important things than experimental code coverage for nvc. If you like, I can make a PR for Osvvm scripts to support this

[JimLewis]

A PR for this would be great.

What I did for other tools is in the coverage directory, I created a directory for each test suite and put the coverage from each test case there.

When a test suite finishes, the test case reports are merged and put into a test suite coverage report in a directory named to match the build.

When the build finishes, all test suite reports are merged and put into the main coverage directory - and then an html report is generated for the build.

[Blebowski]

Hi,

I ran into this discussion, and dare to add few bits that may help.
The NVCs code coverage interface changed slightly in the 1.14.0.

Commands to manipulate NVCs code coverage database (*.covdb file) are are now:

To export to different coverage format (currently only Cobertura is supported):

nvc --cover-export <format> -o <export_file>

To merge code coverage from multiple simulation runs:

nvc --cover-merge -o <merged_covdb> test_1.covdb test_2.covdb ...

To generate code coverage report:

nvc --cover-report -o <dir_name> --exclude-file <efile> <covdb_to_report>

The -c command has been deprecated, since its behavior was somehow mixture
of "merge" and "report".

It is possible to generate "per-test" report if you use --cover-report with test_X.covdb,
as well as "merged report" if you use --cover-report with <merged_covdb>`.

[Blebowski]

I am tinkering with integrating OSVVMs functional coverage into NVCs code coverage,
any inputs will be appreciated:
NVC + OSVVM functional coverage integration

[a-panella]

@Blebowski Apologies, October was very busy for me and I did not return to this.

I have not tested this, but based on more recent comments by yourself, @Blebowski @JimLewis and @nickg on this nvc issue, nickg/nvc#1011 , Jim updated the CoverageAPI

Closing as a result

[JimLewis]

We worked on functional coverage.

Still need a little help with scripts for code coverage.

[Blebowski]

Yeah, true. @JimLewis sorry, this thing bubbled completely to the bottom of my task list...