[Simh] Unit/functional testing simulators

[J. David Bryan]

On Tuesday, December 2, 2014 at 19:54, Seth Morabito wrote:

> SIMH is entirely without test suites, so of course I've had to forge
> on ahead without that safety net. 

I'm not sure I understand your comment.  Do you mean that there are no 
tests for the SIMH framework -- for example, that the ATTACH command 
correctly attaches a file to a unit?


> I've started writing my CPU code without tests. Since the 3B2's WE32100
> CPU is a fairly complex beast, I feel extraordinarily uncomfortable
> about this.

If there are any standalone (i.e., "runs on the bare machine") diagnostics 
for this CPU, I would strongly recommend obtaining and running them during 
development.  A manufacturer's diagnostic, if it is well written, can 
unearth a multitude of sins, especially those where the machine operation 
conflicts with the published specification.

I started work on the HP 2100/1000 simulator several years after Bob Supnik 
had built the CPU and a number of peripheral simulations.  At that time, it 
was already capable of running the multiuser HP Time-Shared BASIC system.  
However, running the HP diagnostics suite revealed over 100 bugs in the 
simulation that I was able to identify and fix.  HP's diagnostics and their 
accompanying manuals were very good at indicating exactly what was being 
tested, what the expected results were, and how the actual results 
deviated.  The current simulator now runs the full set of HP operating 
systems over a range of machine variants.  As Mark noted, I continue to run 
the diagnostics as a regression test for subsequent changes made to the 
simulator.


> I'd feel a lot better if I could just programmatically set up a test
> harness to call sim_instr() in a single step.

Mark suggested a good way to do that.  But from my experience with the HP 
simulator, I would have to credit over 50% of the ultimate success to 
having the service diagnostics for the system, and maybe another 20% to 
having the source code for both the diagnostics and the OS that eventually 
ran on the simulator.  There were just enough "gotchas" in the hardware 
design to make a successful simulation problematic without these aids and 
by working only from the published documentation.


> ...I thought I would ask if anyone else has testing strategies they've
> been working on?  

I'd also recommend generously instrumenting the simulator as you go along 
with the debug controls mentioned in the "Writing a Simulator for the SIMH 
System" manual.  Much as a logic analyzer would give you on a real machine, 
I've found that debug logs that trace machine operations to be invaluable 
in both developing the simulations and in chasing down problems.  Also, 
when tracking down a problem induced by a code change, I've generally found 
it easier to skim a log, or especially to compare a "bad" log with a "known 
good" one, than to step through the simulator code with a debugger.

For example, I've instrumented the CPU of a new simulator under development 
to trace CPU instructions executed; the log excerpt below (from running the 
CPU diagnostic) shows the address, data, and actions taken:

  >>CPU stack: 00.006526  000000    stack registers empty
  >>CPU fetch: 00.015137  004500    instruction fetch
  >>CPU instr: 00.015136  024402  PSHR Q
  >>CPU  data: 00.006527  004500    stack write
  >>CPU stack: 00.006527  000000    stack registers empty
  >>CPU fetch: 00.015140  022402    instruction fetch
  >>CPU instr: 00.015137  004500  DUP,NOP
  >>CPU  data: 00.006527  004500    stack read
  >>CPU stack: 00.006526  000002    stack register A 004500, B 004500
  >>CPU fetch: 00.015141  004500    instruction fetch
  >>CPU instr: 00.015140  022402  ADDI 2
  >>CPU stack: 00.006526  000002    stack register A 004502, B 004500
  >>CPU fetch: 00.015142  027402    instruction fetch

Another example of an instrumentation log for mag tape (MS) and I/O 
processor (IOP) simulations, excerpted from a diagnostic run:

  >>MS   cmds: Unit 0 attached and set online
  >>IOP   irq: Device number 6 asserted INTREQ at priority 11
  >>MS   cmds: Unit 0 requested attention
  >>MS  iobus: Received data 000000 with signals INTPOLLIN
  >>MS  iobus: Returned data 000006 with signals INTACK | JMPMET
  >>IOP   irq: Device number 6 acknowledged request at priority 11
  >>IOP   dio: Test I/O order sent to device number 6
  >>MS  iobus: Received data 000000 with signals DSTATSTB
  >>MS   cmds: Unit 0 status is 1600 bpi | load point | ready
  >>MS  iobus: Returned data 100600 with signals JMPMET
  >>IOP   dio: Control I/O order sent to device number 6
  >>MS  iobus: Received data 040000 with signals DCONTSTB
  >>MS   cmds: Unit 0 control is reset interrupts
  >>MS  iobus: Returned data 000000 with signals JMPMET
  >>IOP   dio: Reset Interrupt order sent to device number 6
  >>MS  iobus: Received data 000000 with signals DRESETINT
  >>MS  iobus: Returned data 000000 with signals JMPMET

I've found simulator debugging to be substantially easier when such aids 
are built-in early in the development process.

                                      -- Dave