How to create a self-checking testbench

A self-checking testbench is a VHDL program that verifies the correctness of the device under test (DUT) without relying on an operator to manually inspect the output. The self-checking testbench runs entirely on its own, and prints an “OK” or “Failed” message in the end.

Every VHDL module should have an associated self-checking testbench. It’s important to be able to verify that all modules have the intended behavior at any time. For example, when you make changes to the DUT, a sub-module, or an interfacing module. We all know that stuff can break, and your best tool for catching these problems is the self-checking testbench.

The device under test

Let’s jump right in and create an example of a self-checking testbench. First, we need something to test, a DUT. For that I have created the module in the code below. It’s a binary to Gray code converter.

library ieee;
use ieee.std_logic_1164.all;

entity gray_converter is
  port (
    bin : in std_logic_vector;
    gray : out std_logic_vector
  );
end gray_converter; 

architecture rtl of gray_converter is
begin

  process(bin) is
  begin
    gray(gray'high) <= bin(bin'high);

    for i in bin'high - 1 downto bin'low loop
      gray(i) <= bin(i + 1) xor bin(i);
    end loop;

  end process;

end architecture;

Gray code is an alternative number coding scheme, different from regular binary encoding. The main property and purpose of Gray code is that only one bit changes when counting between adjacent number values.

The table above shows how Gray code differs from binary code.

The testbench

We shall start by creating the basic testbench and instantiating the DUT within it. The code below shows the testbench file with the DUT instantiated and all the necessary imports.

library ieee;
use ieee.std_logic_1164.all;
use ieee.numeric_std.all;

use std.env.finish;

entity gray_converter_tb is
end gray_converter_tb;

architecture sim of gray_converter_tb is

  signal bin : std_logic_vector(3 downto 0) := (others => '0');
  signal gray : std_logic_vector(3 downto 0);

begin

  DUT : entity work.gray_converter(rtl)
  port map (
    bin => bin,
    gray => gray
  );

end architecture;

Note that we are importing std.env.finish which requires VHDL-2008. If you try to compile the testbench in ModelSim without changing anything, you will get the following error:

# ** Warning: gray_converter_tb.vhd(6): (vcom-1516)
Package "STD.ENV" does not exist in this language version.

Fortunately, this can easily be fixed by setting the VHDL version for the testbench file to VHDL-2008. Right-click the testbench .vhd file and select Properties→VHDL→Use 1076-2008->OK.

You don’t need to change anything for the DUT. It’s normal to use a higher version of VHDL for testbenches than for the RTL modules. You always want to be able to use the latest VHDL constructs in your testbench, but most synthesis tools don’t support them.

Generating the input

Our next addition to the testbench will be the process that generates inputs for the DUT. It’s always best to create an exhaustive test, a test that tries all possible input values. Although, if there are too many permutations, you may be restricted to doing only corner cases.

The input and outputs of our DUT is of unspecified range. However, I will make an educated guess that testing with vector lengths of four bits is enough to reveal any possible issue with this module.

The code below contains the entire process for generating the input sequence.

PROC_SEQUENCE : process
begin

  -- Test all possible input values
  for i in 0 to 2**bin'length - 1 loop
    bin <= std_logic_vector(to_unsigned(i, bin'length));
    wait for 10 ns;
  end loop;

  -- Finally, test the wrapped value
  bin <= (others => '0');
  wait for 10 ns;

  report "Test: OK";
  finish;

end process;

The first code chunk is a For loop which generates the counting sequence from the lowest possible value to the highest possible value. In between the values, we wait for 10 nanoseconds to allow the DUT to react. Although, any nanosecond value greater than 0 would have worked, because the logic inside of the DUT is purely combinational.

The second chunk of code is for testing the situation where the input counter goes back to 0, which was the initial input value. After that, there is no need for further testing because the DUT will just continue to yield the same results over and over again.

The final two code lines within the process is for ending the test gracefully. The text “Test: OK” is printed to the console, and then the simulation is stopped by using the VHDL-2008 keyword “finish”.

Note that if you run ModelSim with the default Run-button, ModelSim will prompt you with the quit dialog after the testbench completes successfully. This behavior may be changed when launching Vsim from a script or from the command line. Add the “-onfinish stop” switch to the Vsim command, as described in the ModelSim command reference.

Checking the output

Now we are supplying the DUT with inputs, but there is no checking of the output at all. The testbench will print out “Test: OK”, regardless if the output is correct or not. Let’s do something about that.

When creating a verification algorithm, you should always try to implement the test differently than in the DUT. Otherwise, a fundamental flaw in the logic may go unnoticed because it’s present in the DUT as well as in the testbench algorithm.

Following this principle, we are going to test the DUT output not by checking that the Gray code is correct, but rather that only one bit changes from one number to the next. After all, that’s the whole reason for using Gray code in the first place. The code below shows a process that performs this kind of checking.

PROC_CHECKER : process
    variable prev : std_logic_vector(gray'range);
    variable count : integer;
begin
  wait on bin;

  prev := gray;

  -- Wait for all delta cycles to propagate
  wait for 1 ns;
  
  -- Count the number of changed bits
  count := 0;
  for i in gray'range loop
    if gray(i) /= prev(i) then
      count := count + 1;
    end if;
  end loop;

  assert count = 1
    report integer'image(count) & " bits changed, should have been 1"
    severity failure;
  
end process;

The process is sensitive to the bin signal, the input to the DUT. We could have used a sensitivity list with the same result, but I prefer to use only Wait statements in testbench code. It’s a convention that makes it easy to know if you’re dealing with a testbench or an RTL module just by looking at how the code is written.

On the seconds line, we copy the previous DUT output. Remember, this is the first delta-cycle after the bin signal has changed, and the DUT cannot possibly react yet. So, it is safe to copy the DUT output with the assumption that this is the old value.

Then, we wait for 1 nanosecond to allow all combinational logic in the DUT to complete. Now, the DUT output should be stable, and we can safely examine its value.

In the next chunk of code, we use a For loop for counting the number of changed bits on the DUT output.

Finally, comes the Assert statement that checks that the number of changed bits is exactly 1. Assert statements work by checking the condition, which is this case is count = 1. If the condition evaluates to false, an assertion will be raised, and the simulator will stop before the “Test: OK” message is printed out.

It’s wise to include the optional report statement with the assert statement. This text will be printed out if the assertion fails. In our example, I provide a brief explanation of the event that caused the testbench to fail.

Running the testbench

It’s time to run our testbench to verify that the DUT is working correctly. After starting the simulation in ModelSim, and pressing the “run -all” button, we see that the “Test: OK” message is printed to the console.

VSIM 1> run -all
# ** Note: Test: OK
#    Time: 170 ns  Iteration: 0  Instance: /gray_converter_tb

Testing the testbench

I always like to create a failing condition in the DUT just to see that the testbench works. Sometimes this comes naturally due to actual errors in the DUT while you are developing it. But if it doesn’t, just create an error briefly to test the testbench.

To achieve this, I will edit the DUT code to create a stuck at 0 error on bit number 3. After this test, I will remove the error with the knowledge that the testbench is able to detect such errors. The code below shows the DUT process with the additional code line.

process(bin) is
begin
  gray(gray'high) <= bin(bin'high);

  for i in bin'high - 1 downto bin'low loop
    gray(i) <= bin(i + 1) xor bin(i);
  end loop;

  -- Emulate a stuck at zero error
  gray(3) <= '0';

end process;

When we run the testbench now, we can see that the testbench stops, and the error is printed out before the “Test: OK” line is reached. The transcript from the ModelSim console is shown below.

VSIM 2> run -all
# ** Failure: 0 bits changed, should have been 1
#    Time: 81 ns  Iteration: 0
Process: /gray_converter_tb/PROC_CHECKER File: gray_converter_tb.vhd
# Break in Process PROC_CHECKER at ray_converter_tb.vhd line 61

Getting started with self-checking testbenches

You should be able to create a self-checking testbench by using what you have learned in this article. Make it a habit to always create self-checking testbenches for all of your VHDL modules. It will save you time in the long run.

It’s OK to be creative when writing testbenches. More so than when writing RTL modules because not all VHDL constructs can be synthesized, but the testbench doesn’t need to be synthesizable. Expect to spend at least as much time writing tests as you spend on writing the DUT.

If you want to get serious about testing, you may be interested in my *upcoming VHDL and FPGA course. In the course, I will walk you through the complete design process from idea to the physical working FPGA prototype. We will create multiple self-checking testbenches.

Updated 12th of October 2020: I have completed the course. Click the image below to learn more.

I will teach you the best practices for successfully creating VHDL and FPGA designs the right way. Knowledge that I have acquired through many years in academia and after working as a hardware engineer in the defense industry will be passed on to you.

Read more about the Dot Matrix VHDL and FPGA course here!