Gitlab

The gitlab #gitforge includes tones of features. Among these, a facility called the container registry, which stores per project container images. Guix pack allows the creation of custom #reproductible environments as images. In particular, it is possible to create a docker image out of our manifest and channels files with

guix time-machine -C channels.scm -- pack --compression=xz --save-provenance -f docker -m manifest.scm

Check the documentation for options.
Remember that there are obviously alternative methods to produce docker images. The point on using guix resides on its reproducibility capabilities: you’ll be able to create a new, identical docker image, out of the manifest and channels files at any point in time. Even more: you’ll have the capacity to retrieve your manifest file out of the binary image in case your manifest file gets lost.
Then, this image must be loaded into the local docker store with

docker load < IMAGE

and renamed to something meaningful

docker tag IMAGE:latest gitlab-registry.whatever.fr/domain/group/NAME:TAG

go remote

Finally, pushed to the remote container registry of your project with

docker push gitlab-registry.whatever.fr/domain/group/NAME:TAG

At this point, you have an environment where you’ll run your tests using gitlab's ci features. You’ll set up your gitlab’s runners and manifest files to use this container to execute your jobs.
As an alternative, you could use a ssh executor running on your own fast and powerful hardware resources (dedicated machine, shared cluster, etc.). In this case, you’d rather produce an apptainer container image with:

guix time-machine -C channels.scm -- pack -f squashfs ...

scp this container file to your computing resources and call it from the #gitlab runner.

Github

The github is probably the most popular #gitforge out there. It follows a similar to #gitlab in its conception (pull requests and merge requests, you catch the idea ?). It also includes a container registry, and the set of features if offers may be exchanged with ease with any other #gitforge following the same paradigm. No need to go into more details.
There is a couple of interesting tips about using #github, though. It happens more usually than not that users encounter frequently problems of #reproducibility when using container images hosted on ghcr.io, the hosting service for user images. These images are usually employed for running #ci testing pipelines, and they usually break as upstream changes happen: updates, image definition changes, image packages upgrades, etc. If you read my dependencies hell post, this should ring a bell.
What can be done about in what concerns #modernhw ? Well, we have #guix. Let’s try a differente approach: building an image locally, and pushing it to #github registry. Let’s see how.

in practice

An example repository shows tha way to proceed. Its contents allow to create a docker container image to be hosted remotely. It includes all that’s necessary to perform remote #ci testing of a #modernhw #vhdl design.

docker pull ghcr.io/csantosb/hdl
docker images # check $ID
docker run -ti $ID bash

It includes a couple of #plaintext files to produce a #deterministic container. First, the channels.scm file with the list of guix chanels to use to pull packages from. Then, a manifest.scm, with the list of packages to be install within the container.
The image container may be build with

image=$(guix time-machine --channels=channels.scm -- \
             pack -f docker \
             -S /bin=bin \
             --save-provenance \
             -m manifest.scm)

At this point, it is to be load to the docker store with

docker load < $image
# docker images

Now it is time to tag the image

docker tag IMID ghcr.io/USER/REPO:RELEASE

and login to ghcr.io

docker login -u USER -p PASSWORD ghcr.io

Finally, the image is to be push remotely

docker push ghcr.io/USER/HDL:RELEASE

test

You’ll may test this image using the neorv32 project, for example, with:

docker pull ghcr.io/csantosb/hdl
docker run -ti ID bash
git clone --depth=1 https://github.com/stnolting/neorv32
cd neorv32
git clone --depth=1 https://github.com/stnolting/neorv32-vunit test
cd test
rm -rf neorv32
ln -sf ../../neorv32 neorv32
python3 sim/run.py --ci-mode -v

git

We said #git, then, as mandatory when tracking changes (in documentation, project development, taking notes, etc.). Meaningful changes imply new commits (and good commit messages, for what it takes), but this comes along with a risk of introducing issues. Some kind of mechanism is necessary to automatize the execution of a checkout list to be run per new commit. The list is project aware, for sure, but may also be different following the git branch, and even the kind of commit (merges are to be considered differently to regular commits in topic branches, for example). We need to consider what an issue exactly is, and then you’ll need to adopt a different perspective on kinds of checkout lists.

verification

First (ideally), one starts with clear specifications about the goals of current development effort (in practice this never happens in research, and if you ever have it, they’ll evolve with time). These specifications (you’ll figure out where to find them somehow) will define the tests to run. For example, if you need to implement in firmware a deep neural network, you’ll probably have access to a test data set to verify the outcomes are correct. You may tune, improve or even completely change the architecture of your network, at the very end, you’ll have to verify your design with help of the test data set. Additionally, you may define more sophisticated tests: consumption, area, resources, etc. These all fall into the category of verification testing.

unit tests

Secondly, you’ll be running unit tests during your whole design cycle (and they’ll evolve along with it), and target tests (the one we mentioned just before). Does this addition perform correctly ? What if we stress a module with random inputs ? Are we going through all code in a given design unit ? Do we cover all values of some input/output signal in this important module ? These are all unit testing checkouts, and they’ll help us to detect issues in an early stage of design.

codesign

Codesign falls somewhere in between the two previous: as a testing methodology, it includes concepts of verification and unit testing (and can be combined with them). It is way more ambitious and complex, but also more powerful. No matter your testing strategy, the point here is that you’ll be running these tests (fully or partially) automatically at the several different stages of your development cycle. If they fail, you’ll have to be warned.

guix

Guix, as a package manager, provides all necessary software to deploy our tests (and can be extended with additional tooling). It also includes all that's necessary to create a running environment where we will execute our tests. Most importantly, #guix does so in a #deterministic and #reproductible way: we will be able to reproduce our tests in the future under exactly the same conditions. Shell containers, profiles and the time machine mechanism allow the degree of #reproducibility we need here. All it takes is a couple of text files.

Most usually, we will focus on two strategies to seek for issues: local, and remote. Local strategies are greatly based on git hooks, and will be topic of another post. Let’s see now in practice what can be done with help of remote tools, based on #ci, understood as a methodology consisting on automatically executing a set of tests procedures on a digital design.
#ciseries

usual suspects

Other than guix channel itself, you may have a feeling of what a guix channel looks like taking a look at the guix-science organization. In there, you’ll find collections of package definitions intended to be used in a context of specific research domains, both in its free and nonfree forms. For one reason or another, they have not their place in the main guix repository, and they live their life in here. Interesting enough, guix-past channel brings old-fashioned software to the present, which reveals useful to reproduce results older than guix itself. Another interesting examples are guixrus, by the ~whereiseveryone collective, providing packages not yet fully tested, nightly releases or alpha quality software; or the guix-hpc channel, with packages related to high performance computing; also guix-bioc, containing bioconductor packages, etc. You get the idea.
To include additional channels, other than the default ones, in your ~/.config/guix/channels.scm file, follow the documentation, and check with guix describe: you’ll get a list of all the channels in use next time you do a guix pull. At this point, you’ll have plenty of new packages to install at your hands ! Just remember the discussion about substitutes: some channels provide substitute servers, some not. This might be an issue, or not, depending of the computing resources necessary to build certain packages.

your first personal channel

Guix channels are simple to create and use. Just put your #guile modules, containing your package definitions, in a git repository. To use them, complete the default guix load path as in

guix install -L ~/guix-channel PACKAGE

You may opt to include this local channel into your ~/.config/guix/channels.scm file, or include the -L flag in your commands. As an alternative, guix searches for package definitions in the $GUIX_PACKAGE_PATH, so you may put your channel path in this variable. Remember that guix builds packages by compiling source code: no substitute server will do the job for you in this case, so plan accordingly with your local resources and patience.

electronics channel

This is about #modernhw after all, so let’s see an example more on detail, my own guix-electronics channel which complements the ./gnu/packages/electronics.scm guix module. This is the channel I’m using daily, where I include all the packages I need and doesn’t yet exist upstream. When they get merged somewhere else, I remove them from here. In the meantime, I’m a happy user of my custom packages.
I classify all my package definitions in #guile modules under ./electronics/packages/MODULE.scm. This is the reason why the compilers module, for example, starts by

(define-module (electronics packages compilers)

Then, proceed as usual (check the guix repo itself for inspiration), putting your packages below

(define-public mypackage
  (package
    (name "mypackage")
    ...
    ))

I create a git repo, commit my changes and push it online to a #gitforge.
I need a .guix-channel file with the url of my channel and a version number. If the code in the channel depends on other channels other than main guix channel, one needs to specify this information in this file too. In this case, for example, several of the package definitions include a dependency on ghdl-clang, provided by guix-science channel, so I need to declare it in here too.
It is also necessary a .guix-authorizations file, with the (GPG key) list of developers with right to push to this channel. This is how guix authenticates channels. Finally, it is important to create a readme file or similar where introduce the channel (first commit, signing key fingerprint). That’s it. Include the channel introduction into your ~/.config/guix/channels.scm file, and you’ll be pulling from your online guix channel.

contributing

The advantage of using a custom channel is to benefit from an increased degree of freedom when using guix, without being dependent of someone’s else time availability to consider your particular needs. Remember, guix is a huge project which follows closely the #freesoftware paradigm, and collaboration works in two directions. You take advantage of other developers contributions to guix, while you participate yourself to improving guix repositories with your package definitions, once they have been tested.
Take for example the list of packages in my electronics channel. I have included here utilities for #vhdl code editing under #emacs, specific to the electronic digital design field, which are not yet available in guix upstream channel. There is also a cosimulation library, #cocotb, which allows coupling #python and #ghdl to simulate a design. I have submitted a patch to include this in guix itself, similarly to python-vunit; once merged, I’ll remove them from here.
Some packages, like open fpga loader are now part of guix. Some others, like the gnat ada compiler and ghdl-clang stable have been merged already on guix-science, as gnat package is obtained from binaries (and so ghdl-clang is also impacted, from guix perspective). I include here a more recent ghdl-clang, non yet released, as it includes some fixes and improvements I am interested in, without having to wait until guix incorporates a new release.
Catch the idea ?
Contributing your code to guix comes to sending #email with your patches attached, it’s that simple. Dont be intimidated by the workflow (this is used by the linux project, for example). Once your patches are submitted, a review of your code folllows, see details. Some tools, like mumi, are helpful to that purpose.
These simple steps will allow you to start contributing to guix:

• git clone guix itselft
  
• add and commit your changes, watch the commit message
  
• don’t forget to use guix lint/style/refresh -l to check your code
  
• use git send-email to create and send a patch
  
• cosider reviews, if any, updating your patch accordingly with git ammend
  
• resend a new patch including a patch version (v1, v2 ...)
  

Interested ? Consult the doc for details, you’ll learn a lot about how to contribute to a common good and collaboration with other people.