[Simh] Unit/functional testing simulators

[Mark Pizzolato - Info Comm]

Hi Seth,

On Tuesday, December 02, 2014 at 5:54 PM, Seth Morabito wrote:
> This last weekend I decided to take a stab at implementing my first
> simulator using the SIMH framework. My eventual goal is an AT&T 3B2
> 400 simulator, though clearly this will take quite some time to get
> working.

It is always good to see new folks jumping in to write another simulator!

> Programming is my day job, I've been doing it for years. Over the last
> decade or so, I've become addicted to testing my software, and I've
> fully embraced test-driven development -- I tend to write a test
> before I write a line of code. SIMH is entirely without test suites,
> so of course I've had to forge on ahead without that safety net. I've
> started writing my CPU code without tests. Since the 3B2's WE32100 CPU
> is a fairly complex beast, I feel extraordinarily uncomfortable about
> this.

At least several simulators have been 'tested' with some of the original hardware diagnostic tests that the manufacturers wrote way back in the old days.  Dave Bryan runs the HP diagnostics to verify the correct operation of the HP2100 simulator.  Running these needs some sort of scripting paradigm to load, start and examine the results of such tests.   Dave Bryan added the ASSERT command added to the scp framework way back in simh V3.4.  ASSERT provides a means of evaluating conditional expressions on simulator state variables aborting a current command procedure.  

The current simulator framework (available at https://github.com/simh/simh) has significant support which you could use to write test scripts for your simulator.

Recent extensions to simh provide various ways test scripts can be implemented extending on the initial ASSERT to include an IF command and GOTO as well as means to evaluate the data emitted out the console (or other serial device port) and injecting input into one of these ports with the EXPECT and SEND commands.

The goal of most simulators is, often not to completely run every detail of the original vendor provided diagnostics, but to run the operating systems and other software which ran on that hardware.  Getting every bit of the hardware 'right' is often not necessary and sometimes too much overhead in the simulator implementation.  The things which software running in the simulator needs to see are what most simulators target to implement.
 
> I'd feel a lot better if I could just programmatically set up a test
> harness to call sim_instr() in a single step. It looks like there's a
> lot of SCP setup that needs to be done before sim_instr() can be
> called. I may have to wade in and just try to suss out exactly how to
> do it myself, but before I go down that route I thought I would ask if
> anyone else has testing strategies they've been working on?

Well, if you're trying to 'test' a particular instruction implementation.  You can drive that test with SCP commands which:
 1) load some memory with the instruction to be executed.
        sim> DEP address <instruction-value>
2) load processor registers with the values you need:
        sim> DEP Rx value1
        sim> DEP Ry value2
        sim> DEP PC address
3) single step into the instruction:
        sim> STEP
4) check for expected results:
       sim> IF Rz=value3 echo Good!

Any such commands can be in a script which the simulator framework will gladly execute with the DO command.  Many of these can be strung together or nested in any way you want.

> I'd love to see testing come to SIMH. It's never too late!

If you need something else, please make suggestions...

If you need help getting enough pieces put together so you can actually do some of the above, feel free to contact me offline and I can make specific suggestions.

- Mark