Alu

Let’s consider now a variant of the previous example with open-logic. Here, we concentrate on a toy design only for demonstration purposes, a dummy alu emulator, which uses #osvvm as verification framework and relies on a few #openlogic blocs. In this case, its dependencies are defined in a manifest.scm file, including both fw-open-logic and osvvm, among other dependencies.
Install dependencies locally, in a new profile with

cd alu
mkdir _deps
export GUIX_PROFILE=open-logic/_deps
guix install -P $GUIX_PROFILE -m .builds/manifest.scm
. $GUIX_PROFILE/etc/profile

In this case, we will test the design using, first, a custom made makefile. Secondly, we will use hdlmake to automatically produce our makefile. Similarly to previous #openlogic example, two build manifest are used:

profile1
profile2

You’ll realise how some of the tasks are common with the case of previous #openlogic example (update channels, auth and update profile).

osvvm

In this case, we also need to compile osvvm libraries

compile__osvvm, produce a compiled version of #osvvm verification libraries; this is necessary as we are using here the tcl scripts included in the library itself to follow the correct order of compilation. Libraries will appear within the local profile under $GUIX_PROFILE/VHDL_LIBS/GHDL-X.Y.Z

test

test, for a fully custom made testing pipeline; in this case, using a Makefile
Just simply, source the .envrc file where the local $GUIX_PROFILE variable is defined, cd to the ghdl directory and call make to compile the design and run the simulation in two steps: first, clean all and include sources in its corresponding libraries with

sh make __clean_all __include

Then, produce a new Makefile using ghdl.

sh ./makefile.sh # ghdl --gen-makefile ...

Finally, run the simulation with

sh make GHDLRUNFLAGS="--stop-time=4us --disp-time --ieee-asserts=enable" run

This will produce a executable file before running it with the provided parameters.
You may notice that, in this case, you need to produce somehow your own Makefile, or equivalent pipeline, right ?

hdlmake

Wouldn’t it be nice if we had a tool to deploy online which produces makefiles for us ? It exists, and its name is #hdlmake.

test__hdlmake
Source the .envrc file where the local $GUIX_PROFILE variable is defined, cd to the .builds/hdlmake directory where all Manifest.py files are located, and call hdlmake to produce the Makefile. Finally, just run make to compile the design, produce an executable and run it.

Check the resulting logs inline here, for example.

git

We said #git, then, as mandatory when tracking changes (in documentation, project development, taking notes, etc.). Meaningful changes imply new commits (and good commit messages, for what it takes), but this comes along with a risk of introducing issues. Some kind of mechanism is necessary to automatize the execution of a checkout list to be run per new commit. The list is project aware, for sure, but may also be different following the git branch, and even the kind of commit (merges are to be considered differently to regular commits in topic branches, for example). We need to consider what an issue exactly is, and then you’ll need to adopt a different perspective on kinds of checkout lists.

verification

First (ideally), one starts with clear specifications about the goals of current development effort (in practice this never happens in research, and if you ever have it, they’ll evolve with time). These specifications (you’ll figure out where to find them somehow) will define the tests to run. For example, if you need to implement in firmware a deep neural network, you’ll probably have access to a test data set to verify the outcomes are correct. You may tune, improve or even completely change the architecture of your network, at the very end, you’ll have to verify your design with help of the test data set. Additionally, you may define more sophisticated tests: consumption, area, resources, etc. These all fall into the category of verification testing.

unit tests

Secondly, you’ll be running unit tests during your whole design cycle (and they’ll evolve along with it), and target tests (the one we mentioned just before). Does this addition perform correctly ? What if we stress a module with random inputs ? Are we going through all code in a given design unit ? Do we cover all values of some input/output signal in this important module ? These are all unit testing checkouts, and they’ll help us to detect issues in an early stage of design.

codesign

Codesign falls somewhere in between the two previous: as a testing methodology, it includes concepts of verification and unit testing (and can be combined with them). It is way more ambitious and complex, but also more powerful. No matter your testing strategy, the point here is that you’ll be running these tests (fully or partially) automatically at the several different stages of your development cycle. If they fail, you’ll have to be warned.

guix

Guix, as a package manager, provides all necessary software to deploy our tests (and can be extended with additional tooling). It also includes all that's necessary to create a running environment where we will execute our tests. Most importantly, #guix does so in a #deterministic and #reproductible way: we will be able to reproduce our tests in the future under exactly the same conditions. Shell containers, profiles and the time machine mechanism allow the degree of #reproducibility we need here. All it takes is a couple of text files.

Most usually, we will focus on two strategies to seek for issues: local, and remote. Local strategies are greatly based on git hooks, and will be topic of another post. Let’s see now in practice what can be done with help of remote tools, based on #ci, understood as a methodology consisting on automatically executing a set of tests procedures on a digital design.
#ciseries

osvvm

First, we have osvvm. #Osvvm is a modern verification #vhdl library using most up-to-date language constructs (by the main contributor to the vhdl standard), and I’ll mention it frequently in this #modernhw posts series. Well documented and being continuously improved, it provides a large set of features for natively verifying advanced designs, among them, a constrained random facility, transactions, logging, functional coverage, scoreboards, FIFOs, sophisticated memory models, etc. Even some co-simulation capabilities are included here. Refer to the documentation repository for up-to-date details about osvvm.
You’ll be able to install osvvm with

# guix search osvvm
guix install osvvm-uart osvvm-scripts

You have a simple use of the osvvm vhdl library in the #aludesign, where the random feature is used to inject inputs to a dut unit. Testing runs for as long as every combination of two variables hasn’t been fully covered. This provides a means to be sure that all cases have been tested, regardless of random inputs. You’ll see an example simulation log here, using the remote ci builds facility of sourcehut.

vunit

Then, we have Vunit as a complete single point of failure framework. It complements traditional test benches with a software oriented approach, based on the “test early and test often” paradigm, a.k.a. unit testing. Here, a pre-built library layer on top of the vhdl design scans, runs and logs unit test cases embedded in user test benches. This approach seeks for an early way to detect as soon as possible conception errors. It performs random testing, advanced checking, logging, advanced communication and an advanced api to access the whole from python. It may be called from the command line, adding custom flags, and configured from a python script file where one defines libraries, sources and test parameters. Simple, elegant and efficient as a testing framework, if you want my opinion. Check the documentation for details.
Install it as usual with

guix install python-vunit

A clever example of its use is provided by the fw-open-logic firmware package (also included in the electronics channel). When you install it, you’ll need to build the package once, which gets installed in the guix store for you to use. During the process, the whole testing of its constituent modules is performed. You may have an overview of how it goes with:

guix build fw-open-logic:out

By the way, if you need the simulation libraries, they are available too.

guix install fw-open-logic:out
# guix install fw-open-logic:sim  # sim libraries

Additionnaly, #vunit is compatible with running a testing #ci pipeline online, as explained here.

cocotb

Finally, we have the interesting and original cocotb. It groups several construct providing a set of facilities to implement coroutine-based cosimulation of vhdl designs. Cosimulation, you say ? Yes. It requests on demand #ghdl simulation time from software (python, in this case), dispatching actions as the time advances. Afterward, based on events’ triggers, you’ll stop simulation coming back to software. This forth and back dance goes on, giving access to advanced testing and verification capabilities. Flexible and customizable as much as needed, in my opinion. Go read the documentation to understand how powerful cosumulation approach can reveal. By the way, install it with

guix install python-cocotb

From the previous, you’ll have understood that having access to verification, unit testing and cosimulation libraries is paramount in #modernhw digital design. Independly or combined (be careful!), they provide powerful tools to detect issues (of any kind) in your design. And yet, this is not enough, as the question arises about where, and when do we run these tests ? From the previous logs in the examples, you’ll have noticed that tests run online in #ci infrastructure. How it goes ? This is the topic of the ci posts in this series.