[Simh] pdp11 i/o addressing

[Timothe Litt]

On 16-Feb-18 14:51, Clem Cole wrote:
> curmudgeon warning below.....
>
> On Fri, Feb 16, 2018 at 11:06 AM, Ethan Dicks <ethan.dicks at gmail.com
> <mailto:ethan.dicks at gmail.com>> wrote:
>
>
>     I started on a VAX with 2MB of physical memory in a 16MB physical
>     address space but with 4GB virtual addresses.  Switching over to the
>     PDP-11 was odd from that.
>
>
> ​Sigh... I fear that is a fault of your education.  
>
> If you ask many (younger) programmers what VM was designed to solve
> (particularly those that never memory constrained systems such as you
> get in 8, 12, 16 or 18 bit processors), they will tell you 'So can
> have more addressable memory.'​  The problem is said programmers never
> experienced or learned about overlays.   Conceptually, a PDP-11 can
> allow a lot more than the 64Ks physical limit by 'swapping out' and
> 'overlaying parts' and calling subroutines through 'thunks' [which to
> quote my old friend Paul Hilfinger from page 427 of his book:  /"an
> onomatopoetic reference to the sound made by a pointer as it moves
> instantaneously up and down the stack"/].  A process has be allowed to
> be larger that 64K, but only 64K (128K on seperate I/D systems) in the
> set up memory maps at a time.    If you need to call a subroutine to
> (optionally) bring in the routines and its data into memory if it is
> not already there, and then set up the map to point the routine in
> question.
>
> BTW: If you play with BSD 2.11 or the like, it uses overlays to allow
> programs to grow in size.   This was needed as people started to try
> to move features from 4BSD and later back to the PDP-11.   At this
> point, I believe you must have what was sometimes referred too as
> '17th address bit - i.e. I/D space which gives you 128K bytes of
> mapped in memory at a time.    But you can (with care) let you
> programs grow.   
>
> The point is that VM is a mechanism/to automatically manage
> overlays/.   The implementation of this management gets easier if
> there are more address bits than physical address bit, but that is the
> key item that is happening.
>
You have to pick a frame of reference.  There were overlays before VM. 
But people may have experience VM before overlays, or vice-versa.  It is
true that overlays aren't well known or understood by the younger crowds.

But, your observation that "VM is a mechanism/to automatically manage
overlays/" is an over-simplification.

Overlays were used to compensate for limited virtual address space.  VM
expands the limit.  But, more than that, it does it more efficiently
than overlays.  Overlays generally are implemented in user mode by a
linker (task builder).  They are not visible to the OS, so limit
sharing.  In addition, they force code (in the overlay areas) to be
writable.  And the overlay loader (invoked by thunks) makes
cross-overlay calls considerably more expensive than an ordinary call -
even when the target call is frequent.  They work better for code than
for data; identifying & swapping out writable data in user mode for an
overlay is non-trivial.  (Sane overly systems didn't encourage this.) 
Although there were some attempts to automate generating an overlay
structure, getting reasonable performance requires a lot of analysis and
programmer effort.  This grows (at least) exponentially as the size and
complexity of a program grows and the overlay structure moves from a
partition or two to a tree structure.  And worse with writable overlays.

VM expands the address space *more efficiently*.  The OS knows about it;
sharing is more efficient; code can be read-only.  Physical memory can
be used for caching recently used pages, reducing I/O.  And a programmer
can do less (not to be confused with no) work.

Overlays had a place - but VM does a lot more than automatically manage
them.   It provides much finer granularity than is practical for
overlays, does so more efficiently (at both the application and system
level), and used thoughtfully saves programmer effort.  Overlays are, in
many respects, a "poor man's VM".  When you don't have paging hardware,
overlays can do a  lot.  But at a steep price.  Initially, paging
hardware wasn't cheap, neither was the OS complexity to implement it
well.  And it took a long time to figure out how to do it well.  (Er,
well enough.  We're still learning.)

That said, VM used carelessly can perform much, much worse than an
overlay structure.  An early version of an operating that I won't name
decided that "everything is a page fault".  Including disk I/O.  So to
run a program, you associated it with an address space, and - well. Page
fault reading the start address.  PF reading the instruction at the
start address.  PF reading the operand of the instruction at the start
address.... It was academically beautiful - but hopelessly
non-performant.  As were programs that figured that since memory is
free, might as well allocate GB sparse arrays.

I remember when VAX's 32-bit VM was supposed to be "infinite" and
overlays "obsolete"; divided into 4 segments, it didn't take long before
1GB was "too small", and thoughts of overlays had a brief resurgence. 

Still, I miss telling Link-10 to provide a plot (on a drum pen plotter)
of the overlay structure of a really big program, like SPSS.  It could
tell you a lot about how the developer thought it worked.

I don't miss the RSX-11/TKB overlays, even though they did force
programmers to think about their code.

And I cede curmudgeon points and status to no one :-)

("Thunk" is also used for the code used in call by name - which has
another hollow sound.)

> Sadly, since people stopped learning about overlay management, the
> context of what it was doing under the covers was lost.
>
> Clem
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.trailing-edge.com/pipermail/simh/attachments/20180216/12301a89/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4577 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://mailman.trailing-edge.com/pipermail/simh/attachments/20180216/12301a89/attachment-0001.bin>