[Simh] Unit/functional testing simulators

[Seth Morabito]

* On Wed, Dec 03, 2014 at 02:27:33AM -0500, J. David Bryan <jdbryan at acm.org> wrote:
> On Tuesday, December 2, 2014 at 19:54, Seth Morabito wrote:
> 
> > SIMH is entirely without test suites, so of course I've had to forge
> > on ahead without that safety net. 
> 
> I'm not sure I understand your comment.  Do you mean that there are no 
> tests for the SIMH framework -- for example, that the ATTACH command 
> correctly attaches a file to a unit?

I meant unit tests in the Kent Beck sense of unit testing, i.e. code
that is automatically run at or shortly after compile time to verify
correctness of "units" (likely C functions, in this case).  This would
include expectations of certain outputs given certain inputs,
expectation of side effects, etc. -- something that makes assertions
about expected behavior and fails when those expectations aren't met,
and does so as an integral part of the code/compile/test cycle.

The important thing about these tests is that they test very small,
discrete parts of code. A specific example I had in my head is
instruction decode. For the WE32100, I would probably have a
subroutine that reads the current opcode at the PC, then decodes the
instruction and all of its operands into some kind of intermediate
structure for further use. That specific subroutine would be under a
unit test to verify that it reads in the correct number of operands,
sets the right addressing mode for each operand, and sets the byte
length of the opcode + operands so that the PC can be post-incremented
after execution.

But, that said...

> > I'd feel a lot better if I could just programmatically set up a test
> > harness to call sim_instr() in a single step.
> 
> Mark suggested a good way to do that.

And I'm gratful for that, thanks! I did not realize how advanced the
SCP scripting was these days. I think I can construct an entire suite
of functional / integration tests. If I combine debug logging, SCP
scripting, and some kind of 'expect' harness, it should really help me
out and I can live without the unit testing I described above.

> If there are any standalone (i.e., "runs on the bare machine") diagnostics 
> for this CPU, I would strongly recommend obtaining and running them during 
> development.  A manufacturer's diagnostic, if it is well written, can 
> unearth a multitude of sins, especially those where the machine operation 
> conflicts with the published specification.

There are diagnostics for the 3B2, but I won't actually be able to run
them until I have most of the simulator to the point where I can load
software off a disk image. Once I get to that point, I think the AT&T
diagnostic disk will be invaluable. But that's a long way off :)

> There were just enough "gotchas" in the hardware 
> design to make a successful simulation problematic without these aids and 
> by working only from the published documentation.

I'm pretty worried about this for the 3B2 emulator. There's precious
little documentation. The most valuable thing I have so far is
the SysVR3 source code itself. Without that I wouldn't even know
the memory map for the system -- none of the doucmentation I have
access to even includes that. Yikes!


-Seth