Enter your search terms Submit search form
	Web pdp-10.trailing-edge.com

.! DYNLIB proposal.  See also DYNLIB.PRO, DYNLIB.CTL
.! Issue 1.2
.page size 58, 72
.subtitle
.flag quote `
.flag capitalize
.flag index
.set paragraph 0, 1, 3
.autoparagraph
.style headers 5, 1, 4
.title Dynamic Libraries

#
.page
.number page 1
.HL 1 INTRODUCTION

.hl 2 Purpose of this document

  This document is a proposal for a dynamically linked library
system. This proposal is being made because the original proposal
[1] as implemented in [2] does not provide all of the facilities
needed to use dynamic libraries in the layered product
environment proposed in [4].  Where possible, the additional
capabilities are made as extensions to [1].  

.hl 2 History and Related Documents

  In March of 1981, Dan Murphy wrote a proposal called "Dynamically
Linked Libraries in TOPS-20" [1].

  From late 1981 through August 1982, Kevin Wallace wrote "Canonical
Terminal Support in the TOPS-20 Operating System" [2].  He used an
implementation of the dynamic library concept described in [1].
[5], [6], and [7] are further descriptions of this project.

  In January of 1983, Mike Uhler wrote "Extended Addressing" [3],
which is the most current description of the way extended addressing
works on KL processors (and is proposed to work on KC processors).

  In May of 1983, Peter Mierswa sent out the latest revision of
"Layered Architecture" [4].  This document is a proposal for how to
use the tools available to us, and some new ones, to make products
that are truly layered.

.hl 2 Extensions to original DYNLIB proposal

.hl 3 Digital transfer vector

  For proper restartability of programs using dynamic libraries, it
was necessary to add a concept of "master initialization" of a
library.  The dynamic library mechanism needs to know how to
master-initialize a library.  

  To allow for the possibility of discovering further such required or
optional functions that the library mechanism should know about, we
created the concept of "DIGITAL-specified" entry points.  To make it
possible to add to these without disturbing existing libraries, they
were put into a separate transfer vector.

.hl 3 Service class

  This is related to the "known library" concept described but not
implemented for the DYNLIB% jsys (existing unofficial implementation).

  Certain types of dynamic libraries (particularly those that perform
some sort of process-wide resource allocation function) must exist in
only one copy in a process.  On the other hand, some libraries provide
only a single "stream" of operations but maintain an internal state
(for example, record sort, in which you call sort once for each record
to be passed in, once to terminate the list of records, and then once
for each record in sorted order).  If a stream is in progress when a
new caller attempts to start a stream, another copy of the library
should be loaded to satisfy the new demand.

  Thus, when an unmodified local transfer vector is used, the dynamic
library mechanism must first determine if there are any copies of the
library currently in memory able to provide the service requested.  We
feel that this should be done not by checking precise file identity,
but rather by having a library declare what "class" of service it
performs.  This is represented by a string, and is encoded into a PDV
name so that the library mechanism can easily determine the list of
providers present.

.hl 3 Busy bit

  Some libraries have no internal state.  Some provide a single stream
of operations.  Some provide multiple streams.  To allow the mechanism
to know if it may route a stream to a provider, a flag has been
defined in the dynamic library descriptor block to say if this copy of
the library can accept another stream.  This bit should be manipulated
by the library to correctly reflect its state.

.hl 3 Master Init

  To make programs restartable (important in a multi-forking
environment) and keep the cost of the restart within reasonable
bounds, each library routine must be provided with a master-init
routine which restores the library to a virgin state.  This routine
will be called once when the library is first mapped in, and once
thereafter each time the program is restarted.

  Other conventions, such as having the init routine of each library
call the init routine of every library it may call, are too
inflexible and expensive.

.hl 3 Save command

  The existing DYNLIB% implementation includes automatic unlinking of
dynamic libraries on execution of a save command.  This is
specifically undesirable for two reasons:
.list 1
.le;The command sequence "`^C", "SAVE", "Continue" should work.  If
libraries get unlinked on the save command, it won't.
.le;Many sites instruct their users to save their core image when a
program exhibits "unusual" behavior (when a bug is suspected), and to
deliver the .exe file produced along with their bug report to help the
local support people see what happenned.  If the save command de-links
dynamic libraries, then this procedure becomes much less useful.
.end list

  We cannot see any reason why this troublesome function is desirable
or even useful, so we request that it be removed.

.hl 3 Galactic variables

  "Galactic" variables are variables with a wider scope than "global"
variables -- in particular, variables whose scope exceeds a single
`.exe file.  On VMS, these are called "universal" variables, but that
term means something else entirely on TOPS-20 systems.

  Galactic variables are the "other half" of dynamic linking not
mentioned in previous proposals -- access to data in other libraries.
Previous proposals have only given access to routines in other
libraries.

  This is needed for "logical completeness" (never a big draw in
getting funding) and for at least one specific need in the global
interrupt/trap manager currently being designed.

  This function is very cheap to provide, as explained below.

.hl 3 Library Version matching

  The library itself and the .REL file defining the LDLBLK will
contain library version information.  At run-time this information is
checked to see if the version of the library actually found is
acceptable to the program requesting it.  The version matching rule is
specified in the library but can be overridden in the LDLBLK.

.hl 3 Dynamic library mechanism version matching

  The dynamic library blocks (LDLBLK and DLBLK) contain version number
information.  This can be checked to make sure that the blocks found
are compatible with the code massaging them.

.hl 2 Benefits of dynamic libraries

  Better independence of packages.  Living in your own section,
which you control, gives better isolation than sharing that
section with other code.

  Easier software updates.  A new version of a dynamic library
can be introduced into all programs calling it by simply placing
it on the directory from which dynamic libraries are loaded. 
There is no need to relink programs using it.

  More consistent program behavior.  If a facility is always
provided by the same package of code, then it will always be
provided in the same way (this isn't really a benefit of dynamic
libraries per se, the same thing could be achieved by linking the
code that provides the facility with the caller).

  Easier maintenance.  A crash of from a program structured
from dynamic libraries will show clearly and neatly what version
of each library was loaded, what the internal state of that
library is, and so forth.  Within a library, things will always
be in the same place.

  More efficient real memory use.  If a facility provided by a
dynamic library is never called during a given run of a program,
that library will never be mapped in.  

.HL 2 Conventions used

.hl 3 Indirect words

  ALL of the indirect words (EFIW, IFIW) mentioned in this document
must have the index register bits set to ZERO!!!  No indexing should
be used in the dynamic library tables.  The reason for this is simple:
the contents of the registers aren't globally constant, so the result
of the indexing may not be what was intended.

  Indirection is permitted.  None of the further indirect words
reached through indirection should perform indexing either.

.hl 1 Dynamic Library functions

  The following functions must be provided by the dynamic library
mechanism:

.hl 2 Calling a routine in a dynamic library

  To be able to call routines in a dynamic library, you must:
.list 1
.le;Be running in a non-zero section
.le;Have a global-format stack pointer in AC17
.le;Have linked your program with a library-specific .REL file for
each of the libraries you may call directly (you do not need to
consider libraries that may be called by libraries you call)
.end list
  A routine in a dynamic library is called by performing a PUSHJ
through a "local transfer vector".  This local transfer vector is
provided by the library-specific .REL file mentioned above.

  Suppose the dynamic library EXAMPL is documented as containing a
routine RNDNAM.  The argument-passing rules for this routine and what
registers it preserves should be documented as well.

  To call this routine from MACRO, you must set up the arguments as
specified for the routine, and then say
.I 5;PUSHJ P, @RNDNAM

  You may use any type and number of levels of indexing and
indirection in the instruction that calls a routine in a dynamic
library.  It is also perfectly acceptable for the PUSHJ instruction to
be XCT'd from somewhere else.

.hl 2 Referring to a galactic variable

  To refer to a data location in another library, you must:
.list 1
.LE;Be running in a non-zero section
.le;Have linked your program with a library-specific .REL file for the
library which defines the galactic variable to which you wish to refer
.end list

  To refer to the galactic variable, make an indirect reference
through a "local galactic vector" which is provided by the library-specific
`.REL file mentioned above.

  Referring to a galactic variable defined in a library not yet mapped
in will cause that library to be mapped in (and its master init
routine to be run).

.hl 2 Global master init

  This routine should be called from the initialization routine of
every top-level program (since on tops-20 top-level programs are
entered through entry vectors, whereas dynamic libraries are entered
through transfer vectors pointed to by PDV's, a package can always
know how it was entered, and thus whether it is running at the top
level or not).

  This finds and calls the master init routine of every copy of every
dynamic library mapped into the process address space.

  This is necessary to make restarting work right without incurring
unacceptable overhead.

.hl 2 Overloading

  Merge information about several dynamic libraries into one local
transfer vector.

  This is an extension of the capabilities you get by manually moving
some EFIW into the local transfer vector in locations where you want
to supercede the routine provided by the library.  For example, you
can cause your transfer vector to contain EFIW's to your private
(presumably improved; perhaps debugging versions?) of some routines
and point to the library copies of other routines.

  The normal rules for overloading are as follows:

.s 1.list 1
.le;LTVEC entries not containing their default initial state will not
be altered.  The default initial state is "IFIW 7777,,LDLBLK" for the
monitor implementation, or "IFIW LDLBLK-1" for the user-mode
implementation.
.le;If an MTVEC entry contains 0, the corresponding LTVEC entry is not
altered. 
.le;If the LTVEC is longer than the MTVEC, excess LTVEC locations are
not altered.
.le;If the MTVEC is longer than the LTVEC, the additional MTVEC
entries are ignored.
.end list

.hl 3 Load MTVEC into LTVEC

  You specify a master transfer vector and a local transfer vector.
The master transfer vector is loaded over the local transfer vector
according to the usual rules.

.hl 3 Load library into LDLBLK

  You specify a LDLBLK whose transfer vectors you want to load over,
and an LDLBLK specifying the library you want to load from.  The
overloading is done according to the usual rules.

.hl 2 De-linking

  The monitor DYNLIB% jsys provides functions for de-linking a dynamic
library.  We think that this function is not fully thought out.  In
particular, it is not clear how the monitor will find all the LTVEC's
pointing to any given MTVEC.

  We do not see any need for this function.

.hl 2 Finding a free provider

  When a streaming (single or multi) library is entered at the point
that causes a stream to be initiated, if the library is not capable of
initiating another stream, it should return an error.  Newly-written
dynamic libraries should all be non-streaming or multi-streaming, so
this should not come up in practice.

  If this became an important consideration, it might be possible to
write a routine that the library stream-initialization routine could
call which would find the LTVEC the call came through (by analyzing
the call instruction, whose successor address is on the stack), find a
free provider, and re-map the LTVEC to point to this provider.  Note
that this would lose the effects of any overloading performed on that
LTVEC.  This would need a lot more thought before I wanted to commit
to it.  

.hl 2 Library version checking

  The master DLBLK in the library .EXE file contains the library
version number in location .DYVER.  It contains the default version
checking rule for that library within the flag word (.DYFLG) in field
DY%VER.

  The LDLBLK in the calling program contains the library version
number that it corresponds to in location .LDVER.  It may optionally
contain a version checking rule (indicated by LD%VMA in .LDVER) in
field LD%VER of .LDFLG, which overrides the default rule specified in
the DLBLK.

  Based on this information, a check is performed when the library is
loaded to see if it matches the version the calling program was
compiled against well enough to be used.  If it does not, an error is
printed and the program terminated.

  The version numbers are in standard TOPS-20 version word format.
For version matching purposes, only VI%MAJ and VI%MIN are considered.

  If the major and minor versions from both places are the same, then
the library is accepted.

  In addition, the library may optionally be accepted if:
.s 1.LIST 0, "o"
.le;Library major version greater than program major version (VM%MAG)
.LE;Library major version less than program major version (VM%MAL)
.LE;Major versions equal and library minor version greater than
program minor version (VM%MIG)
.LE;Major versions equal and library minor version less than program
minor version (VM%MIL)
.end list
Any combination may be set.

.hl 2 Dynamic library mechanism version checking

  The master DLBLK in the library .EXE file contains the library
mechanism version number in location .DYFVN.

  The LDLBLK in the calling program contains the library mechanism
version number in location .LDFVN.

  Based on this information, a check is performed when the library is
loaded to see if it matches the version of the dynamic library
mechanism (FAKDYN in the user mode implementation) that is being used.
If it does not, an error is printed and the program terminated.

  The version numbers are in standard TOPS-20 version word format.
For version matching purposes, only VI%MAJ and VI%MIN are considered.

  If the major and minor versions from both places are the same, then
the library is accepted.

  In addition, the library may optionally be accepted if:
.s 1.LIST 0, "o"
.le;Library major version greater than mechanism major version (VM%MAG)
.LE;Library major version less than mechanism major version (VM%MAL)
.LE;Major versions equal and library minor version greater than
mechanism minor version (VM%MIG)
.LE;Major versions equal and library minor version less than mechanism
minor version (VM%MIL)
.end list
Any combination may be set.

  This is applied both to the LDLBLK and the DLBLK.

.hl 1 Implementation

  First I will describe the proposed changes for monitor-implemented
dynamic libraries.

  Later, I describe a way to implement the same capabilities in user
code, leaving an easy way to take advantage of monitor capabilities
when they become available.

.hl 2 Monitor-mode implementation

.hl 3 In the user program

  I suggest that all dynamic libraries consist of two user-visible
parts: the dynamic library .exe file, and a .rel file that defines the
proper local dynamic library block for that dynamic library.  Having
the local dynamic library block available from a rel file removes the
responsibility of constructing it (and the possibility of making it
wrong) from the users of dynamic libraries.  It also makes it easier
to change the local dynamic library block, which will probably be
necessary to do when making the transition from user-code to
monitor-supported dynamic libraries.

  I propose that the local dynamic library block should look like this
(this is a considerable change in format from our current version):

.s 1.LITERAL
LDLBLK:	length (must be 11 currently)
	Block format version number (to be used in the future to
	    provide fancier kinds of dynamic library services such as
	    calls by name instead of by offset)
	Flag word (DIGITAL-defined only)
	    Initialized: library has been loaded and vectors updated 
	User word
	Pointer to "service class" string
	Pointer to file spec string
	Library version number
	IFIW Address of DIGITAL-specified entry-point vector (LDTVEC)
	IFIW Address of library-specific entry-point vector (LCTVEC)
	Reserved to DIGITAL
	IFIW Address of library-specific galactic vector (LCGVEC)
	[The possibility of additional entries is reserved to DIGITAL]
.END LITERAL

  Digital-specified entry points will likely include several flavors of
initialization and cleanup.  They have been made a separate vector so
that it will be possible to add additional ones in the future if
needed.  Note that there is nothing requiring that each offset in each
transfer vector point to a unique routine.  

.s 1.LITERAL
LDTVEC:	diglen+1
	7777,,LDLBLK
	. . .
	7777,,LDLBLK

LCTVEC:	cuslen+1
	7777,,LDLBLK
	. . .
	7777,,LDLBLK
LCGVEC:	cgalen+1
	7777,,LDLBLK
	. . .
	7777,,LDLBLK
.END LITERAL

.hl 3 In the library

  I propose that the dynamic library block (DLBLK) in the dynamic
library itself should change similarly:

.s 1.LITERAL
DLBLK:	length (must be 9 currently)
	Block format version number
	Flag word (DIGITAL defined only)
	    Busy: library can't start a stream
	    Version match: what are acceptable matches between DLBLK
		version and LDLBLK version?
	Library-use word
	Library version number
	IFIW Address of DIGITAL-specified entry-point vector (DTVEC)
	IFIW Address of library-specific entry-point vector (CTVEC)
	Reserved to DIGITAL
	IFIW Address of library-specific galactic variable vector (CGVEC)
	[The possibility of additional entries is reserved to DIGITAL]

DTVEC:	diglen+1
	IFIW routine entry-point
	. . .
	IFIW routine entry-point

CTVEC:	cuslen+1
	IFIW routine entry-point
	. . .
	IFIW routine entry-point

CGVEC:	cgalen+1
	IFIW address
	. . .
	IFIW address
.END LITERAL

  In addition to the PDV "Dynamic Library" which points to the DLBLK,
each dynamic library will have a PDV named "DYNLIB$class", where
"class" is the name of the service class provided.  Class names
containing a "%" are reserved to DIGITAL.  For example, the sort
service dynamic library would have a PDV "Dynamic Library", and a PDV
"DYNLIB$SORT%".

  The service-class PDV should point to the DLBLK just as the "Dynamic
Library" PDV does, so that the DLBLK can be found directly from the
service class name.  It should point to THE DLBLK, not to another
copy.  There is only one DLBLK in a dynamic library.

  Note: some convention for tops-20 name-space should be adopted and
applied to these PDV names as well as to all global symbol names.  The
exact standard adopted is much less important than the adoption of
SOME standard.  The details above are primarily intended as examples.

  Note: The PDV name "Dynamic Library" should belong to DIGITAL name
space, not customer name space.  Rather than explicitly listing it as
an exception to the rules, the name should be changed so that it falls
in DIGITAL name space.

.hl 3 <JSYS interface

  Not yet defined.  The details of this aren't nearly as important to
settle as the capabilities and data structures for normal use.  Note
that explicit calls to the JSYS interface will be very much the
exception, not the norm.

.hl 2 User-mode implementation

  One goal of our FAKDYN procedure must be that it can be replaced by a
monitor-supported dynamic library procedure with minimum effort.  This
FAKDYN proposal could be converted to a monitor-supported DYNLIB
(assuming the monitor buys the changes proposed above, which are
independent of FAKDYN anyway) by re-linking existing programs with a
new LDLBLK rel file.  No code changes in any program would be required.

  To use FAKDYN, there will be the additional requirement that all
packages calling dynamic libraries be linked with FAKDYN.REL.  

  There is one restriction under FAKDYN:  a library cannot be loaded
in response to a reference to a galactic variable defined in that
library.  The recommended workaround is to call a routine in the
library before attempting reference to a galactic variable.  Testing
the "initialized" bit in the LDLBLK could increase the efficiency of
this.

  First, extend the LDLBLK with a FAKDYN kludge word:

.s 1.LITERAL
|	PUSHJ P, FAKDYN
LDLBLK:	length (must be 11 currently)
	Block format version number
	Flag word (DIGITAL-defined only)
	User word (Can be address of user data block)
	Pointer to "service class" string
	Pointer to file spec string
	Library version number
	IFIW Address of DIGITAL-specified entry-point vector (LDTVEC)
	IFIW Address of library-specific entry-point vector (LCTVEC)
	Reserved to DIGITAL
	IFIW Address of library-specific galactic vector (LCGVEC)
	[The possibility of additional words is reserved to DIGITAL]

LDTVEC:	diglen+1
	IFIW LDLBLK-1
	. . .
	IFIW LDLBLK-1

LCTVEC:	cuslen+1
	IFIW LDLBLK-1
	. . .
	IFIW LDLBLK-1

LCGVEC:	cgalen+1
	IFIW LDLBLK-1
	. . .
	IFIW LDLBLK-1
.END LITERAL

  At FAKDYN entry, stack will contain:

.s 1.LITERAL
	LDLBLK			-1(SP)
	user return		0(SP)
.END LITERAL

  LDLBLK contains all the information necessary to map in (if necessary)
a dynamic library, and to update the local transfer vectors.  After
doing this, FAKDYN decrements the user return to point back to the
instruction that called the routine, and returns to re-execute that
instruction.  Since the local transfer vectors have now been updated,
the re-execution of the instruction will result in calling the
specified routine.

.hl 1 section allocation

  Another area that needs addressing is section allocation.  Since
which sections are in use has a serious impact on performance, the
section allocator should be a piece of digital-supplied code,
preferably in the monitor (where it will know what processor it is
running on and can allocate accordingly).  (Another advantage of
putting this code in the monitor is that an EXE file can be
transported to another system with a different processor and start
allocating sections in a way appropriate for the processor it is
running on.)

.hl 1 <Digital entry points

  A need for the following DIGITAL-specified entry points has been
determined.

.hl 2 Master Init

  DIGITAL entry point 1 is "master init".  Master init is called
automatically by FAKDYN (and eventually by the monitor) when a library
is first mapped in.  If a library does not require master
initialization, that entry in the DTVEC should point to a routine that
simply returns.  The entry should not be 0.  (This restriction
eliminates the need to check before each and every call whether the
entry vector is valid.)

  The master init routine does not accept any arguments.  It must
preserve all registers.  It may assume that there is a global stack
pointer in AC17.  What, if any, assumptions may be made about the
amount of space available on the stack have not yet been decided.

  (In addition to the master init entry point, most libraries will have
an ordinary init or a stream-init entry point.  Do not confuse these
limited initializations with the master init performed once per run or
start command.)

  A routine will be provided which runs the master init on every library
currently mapped in.  This routine should be called by every top-level
program that uses dynamic libraries in its initialization section
(before it calls on any dynamic library services).  

  This takes care of the need to re-initialize on restarts without
having to re-map all the libraries in use, and without having to know
in advance which libraries are going to be used.

  Note that, on TOPS-20, a program must be written to run either as a
top-level, or as a subroutine.  A program can be written to be invoked
either way, but in that case it always knows which way it was in fact
invoked on this run.  Thus, it is meaningful to specify actions to be
taken only if you were called as a top level program.

.appendix Bibliography
.lm 5
.set paragraph -5, 1, 3
.nofill

.p;[1]
Murphy, Dan
Dynamically Linked Libraries in TOPS-20 
DIGITAL interoffice memorandum, March 1981

.p;[2]
Wallace, Kevin Girard
Canonical Terminal Support in the TOPS-20 Operating System
Master's thesis, MIT, May 1982

.p;[3]
Uhler, Mike
Extended Addressing
DIGITAL interoffice memorandum, January 1983

.p;[4]
Mierswa, Peter
Layered Architecture
DIGITAL interoffice memorandum, May 1983

.p;[5]
Wallace, Kevin G.
More on Dynamically Linked Libraries
DIGITAL interoffice memorandum, December 1981

.p;[6]
Wallace, Kevin G.
Functional Specification for Canonical Terminal Support
DIGITAL interoffice memorandum, August 1982

.p;[7]
Wallace, Kevin G.
Design Specification for Canonical Terminal Support
DIGITAL interoffice memorandum, August 1982