[Simh] How do we test the emulators?

[Hittner, David T.]

You've certainly hit the hardest problem for developers on this project.

Bob has been wanting Integration tests for a long time now, so that we
can quickly detect if something has been broken.
However, the Java JUnit concept doesn't translate very well here - we do
need a series of integration tests, but there is no testing framework(s)
for C and simulation that has been found yet.

We need to test:
	simulator framework
	simulated instruction sets
	simulated register behavior
	simulated i/o devices (console, disk, tape, serial, parallel,
ethernet, etc.)
	simulated device timing characteristics (see Bob's paper on Code
Rot :-))
	simulated OS behavior
	code coverage

The testing framework(s) need to have:
	method-level testing
	Process I/O watcher to monitor external I/O streams to see if
they are sending/receiving the correct data
		("EXPECT -text- ON -stream- TIMEOUT -secs- ")
	Process stimulator to send input into the I/O streams
		("PROVIDE -input- ON -stream- AFTER -prompt-" DELAY
-msec-)
	Process breakpoint monitor to watch registers
		("BREAK -mem-loc- AFTER -repetition-count- VERIFY
-register-loc- CONTAINS -value-")

What may have to happen is someone might have to write a framwork that
starts up the simulator in a subprocess and monitors the subprocess(es),
and then gracefully shuts the simulator down after all the tests pass.

I don't believe Bob would be averse to a testing framework written in
another language,
as long as it can successfully drive the C-based simulators.
	
Dave


-----Original Message-----
From: simh-bounces at trailing-edge.com
[mailto:simh-bounces at trailing-edge.com] On Behalf Of Peter Lund
Sent: Tuesday, September 18, 2007 2:44 PM
To: simh
Subject: [Simh] How do we test the emulators?

While mucking around with the build script(s) it annoyed me that I
basically didn't have any tests I could run on the resulting binaries.

Jason Stevens seems to have had the same problem when he recently built
lots and lots of binaries for various platforms.  If it had been dead
easy to test an emulator I don't think he would have written:
"Again these binaries are largely compiled and untested.."

So how could we test them?  I'm not really thinking about huge, in-depth
tests.  More like small and easy smoke tests.

Most of us probably only really care about a small number of emulators
that emulate interesting machines or machines one used to work on for
years, a long time ago.  So we have in-depth knowledge of a few of the
emulators and practically none about the others.

Even the ones we know well require active work to test: find this OS
image, attach it to the right emulated peripheral device, boot it, run
this command, then that command, pretend to send something to an output
device, etc...

Are there a bunch of smallish programs we can gather somewhere on a web
site (or in a tarball, sorry, zipfile) and script so anybody can run the
smoketests automatically without having to know anything about all the
emulators he or she doesn't care about?

I think we should start small and then (slowly) build upwards.  Simple
things like OS images we can boot test automatically or machine code
sorting and searching programs for the really old machines.

We can add coverage testing later on and gradually improve the tests
until we know all instructions have been tested, all addressing modes,
all external hardware, etc.

Anyone?

-Peter

_______________________________________________
Simh mailing list
Simh at trailing-edge.com
http://mailman.trailing-edge.com/mailman/listinfo/simh