[Simh] How do we test the emulators?

[Peter Lund]

(originally only sent to David by mistake)

On Tue, 2007-09-18 at 15:19 -0500, Hittner, David T. wrote:

> Bob has been wanting Integration tests for a long time now, so that we
> can quickly detect if something has been broken.
> However, the Java JUnit concept doesn't translate very well here - we do
> need a series of integration tests, but there is no testing framework(s)
> for C and simulation that has been found yet.

CUnit actually exists ;)

More to the point, we don't actually need an external module for this.
Almost all of it can be done with a small module containing
start_test(), stop_test(), check(), etc functions and some test code
inside each module.  Each module can be compiled in two modes: the
normal one and a stand-alone test mode.  I've used that approach in both
C and Pascal since around '92 and it works beautifully.

I got it from Kevin Weeks, "Is Your Code Done Yet?", Computer Language,
April 1992.

> We need to test:
> 	simulator framework
> 	simulated instruction sets
> 	simulated register behavior
> 	simulated i/o devices (console, disk, tape, serial, parallel,
> ethernet, etc.)
> 	simulated device timing characteristics (see Bob's paper on Code
> Rot :-))
> 	simulated OS behavior
> 	code coverage

Yep, I agree on all of that.  But the perfect must not be allowed to be
the enemy of the good.  Even a bunch of lame smoke tests à la "boot this
OS image and see if we get this particular prompt" would help a lot --
and would probably be easier to set up.

> What may have to happen is someone might have to write a framwork that
> starts up the simulator in a subprocess and monitors the subprocess(es),
> and then gracefully shuts the simulator down after all the tests pass.
> 
> I don't believe Bob would be averse to a testing framework written in
> another language,
> as long as it can successfully drive the C-based simulators.

It should still be something that is easy to set up and run (which
actually kinda rules out Java).  Perl would do the job.  So would
Python.  Posix shell would too but the code ends up looking weird, is
very fragile, and nobody knows how to program in it.

My preference would be C for some kinds of tests and Perl or Python for
the rest.  Perl code can actually be very readable :)

If I get a bunch of suggested images to boot up I would love to hack up
a smoke testing suite.

-Peter