Google
 

Trailing-Edge - PDP-10 Archives - BB-JF18A-BM - documentation/dynlib.mem
There are 4 other files named dynlib.mem in the archive. Click here to see a list.



 
Dynamic Libraries                                                 Page 1
INTRODUCTION


1.0  INTRODUCTION

1.1  Purpose Of This Document

This document is a proposal for a  dynamically  linked  library  system.
This  proposal  is  being  made  because  the  original  proposal [1] as
implemented in [2] does not provide all of the facilities needed to  use
dynamic  libraries  in  the layered product environment proposed in [4].
Where possible, the additional capabilities are made  as  extensions  to
[1].



1.2  History And Related Documents

In March of 1981, Dan Murphy wrote a proposal called "Dynamically Linked
Libraries in TOPS-20" [1].

From late 1981 through  August  1982,  Kevin  Wallace  wrote  "Canonical
Terminal  Support  in  the  TOPS-20  Operating  System" [2].  He used an
implementation of the dynamic library concept described  in  [1].   [5],
[6], and [7] are further descriptions of this project.

In January of 1983, Mike Uhler wrote "Extended Addressing" [3], which is
the  most current description of the way extended addressing works on KL
processors (and is proposed to work on KC processors).

In May of 1983, Peter Mierswa sent out the latest revision  of  "Layered
Architecture" [4].  This document is a proposal for how to use the tools
available to us, and some new ones, to  make  products  that  are  truly
layered.



1.3  Extensions To Original DYNLIB Proposal

1.3.1  Digital Transfer Vector

For proper restartability of programs using dynamic  libraries,  it  was
necessary to add a concept of "master initialization" of a library.  The
dynamic library mechanism needs  to  know  how  to  master-initialize  a
library.

To allow for the possibility of discovering  further  such  required  or
optional  functions  that  the  library  mechanism should know about, we
created the concept of "DIGITAL-specified" entry  points.   To  make  it
possible  to  add  to  these without disturbing existing libraries, they
were put into a separate transfer vector.
Dynamic Libraries                                                 Page 2
INTRODUCTION


1.3.2  Service Class

This is related  to  the  "known  library"  concept  described  but  not
implemented for the DYNLIB% jsys (existing unofficial implementation).

Certain types of dynamic libraries (particularly those that perform some
sort  of  process-wide  resource allocation function) must exist in only
one copy in a process.  On the other hand, some libraries provide only a
single  "stream"  of  operations  but  maintain  an  internal state (for
example, record sort, in which you call sort once for each record to  be
passed in, once to terminate the list of records, and then once for each
record in sorted order).  If a stream is in progress when a  new  caller
attempts to start a stream, another copy of the library should be loaded
to satisfy the new demand.

Thus, when an unmodified local transfer  vector  is  used,  the  dynamic
library  mechanism  must  first determine if there are any copies of the
library currently in memory able to provide the service  requested.   We
feel that this should be done not by checking precise file identity, but
rather by having a library declare what "class" of service it  performs.
This  is represented by a string, and is encoded into a PDV name so that
the library  mechanism  can  easily  determine  the  list  of  providers
present.



1.3.3  Busy Bit

Some libraries have no internal state.  Some provide a single stream  of
operations.   Some  provide multiple streams.  To allow the mechanism to
know if it may route a stream to a provider, a flag has been defined  in
the  dynamic library descriptor block to say if this copy of the library
can accept another stream.   This  bit  should  be  manipulated  by  the
library to correctly reflect its state.



1.3.4  Master Init

To make programs restartable (important in a multi-forking  environment)
and  keep the cost of the restart within reasonable bounds, each library
routine must be provided with a master-init routine which  restores  the
library  to  a  virgin state.  This routine will be called once when the
library is first mapped in, and once thereafter each time the program is
restarted.

Other conventions, such as having the init routine of each library  call
the  init  routine  of every library it may call, are too inflexible and
expensive.
Dynamic Libraries                                                 Page 3
INTRODUCTION


1.3.5  Save Command

The existing DYNLIB%  implementation  includes  automatic  unlinking  of
dynamic  libraries on execution of a save command.  This is specifically
undesirable for two reasons:

     1.  The command sequence "^C", "SAVE", "Continue" should work.   If
         libraries get unlinked on the save command, it won't.

     2.  Many sites instruct their users to save their core image when a
         program  exhibits "unusual" behavior (when a bug is suspected),
         and to deliver the .exe file  produced  along  with  their  bug
         report to help the local support people see what happenned.  If
         the  save  command  de-links  dynamic  libraries,   then   this
         procedure becomes much less useful.


We cannot see any reason why this troublesome function is  desirable  or
even useful, so we request that it be removed.



1.3.6  Galactic Variables

"Galactic" variables are variables with  a  wider  scope  than  "global"
variables  -- in particular, variables whose scope exceeds a single .exe
file.  On VMS, these are called "universal"  variables,  but  that  term
means something else entirely on TOPS-20 systems.

Galactic variables are the "other half" of dynamic linking not mentioned
in  previous  proposals  -- access to data in other libraries.  Previous
proposals have only given access to routines in other libraries.

This is needed for "logical completeness" (never a big draw  in  getting
funding) and for at least one specific need in the global interrupt/trap
manager currently being designed.

This function is very cheap to provide, as explained below.



1.3.7  Library Version Matching

The library itself and the .REL file defining the  LDLBLK  will  contain
library version information.  At run-time this information is checked to
see if the version of the library actually found is  acceptable  to  the
program  requesting  it.   The version matching rule is specified in the
library but can be overridden in the LDLBLK.
Dynamic Libraries                                                 Page 4
INTRODUCTION


1.3.8  Dynamic Library Mechanism Version Matching

The dynamic library blocks (LDLBLK and  DLBLK)  contain  version  number
information.  This can be checked to make sure that the blocks found are
compatible with the code massaging them.



1.4  Benefits Of Dynamic Libraries

Better independence of packages.  Living in your own section, which  you
control,  gives  better  isolation  than sharing that section with other
code.

Easier software updates.  A new version of  a  dynamic  library  can  be
introduced  into  all  programs  calling  it by simply placing it on the
directory from which dynamic libraries are loaded.  There is no need  to
relink programs using it.

More consistent program behavior.  If a facility is always  provided  by
the  same  package  of code, then it will always be provided in the same
way (this isn't really a benefit of dynamic libraries per se,  the  same
thing  could  be achieved by linking the code that provides the facility
with the caller).

Easier maintenance.  A crash of from a program structured  from  dynamic
libraries  will show clearly and neatly what version of each library was
loaded, what the internal state  of  that  library  is,  and  so  forth.
Within a library, things will always be in the same place.

More efficient real memory use.  If a facility  provided  by  a  dynamic
library  is  never  called during a given run of a program, that library
will never be mapped in.



1.5  Conventions Used

1.5.1  Indirect Words

ALL of the indirect words (EFIW, IFIW) mentioned in this  document  must
have  the index register bits set to ZERO!!!  No indexing should be used
in the dynamic library tables.  The reason  for  this  is  simple:   the
contents of the registers aren't globally constant, so the result of the
indexing may not be what was intended.

Indirection is permitted.  None of the further  indirect  words  reached
through indirection should perform indexing either.
Dynamic Libraries                                                 Page 5
DYNAMIC LIBRARY FUNCTIONS


2.0  DYNAMIC LIBRARY FUNCTIONS

The  following  functions  must  be  provided  by  the  dynamic  library
mechanism:



2.1  Calling A Routine In A Dynamic Library

To be able to call routines in a dynamic library, you must:

     1.  Be running in a non-zero section

     2.  Have a global-format stack pointer in AC17

     3.  Have linked your program with a library-specific .REL file  for
         each of the libraries you may call directly (you do not need to
         consider libraries that may be called by libraries you call)


A routine in a dynamic library is called by performing a PUSHJ through a
"local  transfer vector".  This local transfer vector is provided by the
library-specific .REL file mentioned above.

Suppose the dynamic library EXAMPL is documented as containing a routine
RNDNAM.   The argument-passing rules for this routine and what registers
it preserves should be documented as well.

To call this routine from MACRO,  you  must  set  up  the  arguments  as
specified for the routine, and then say
     PUSHJ P, @RNDNAM

You may use any type and number of levels of indexing and indirection in
the  instruction  that calls a routine in a dynamic library.  It is also
perfectly  acceptable  for  the  PUSHJ  instruction  to  be  XCT'd  from
somewhere else.



2.2  Referring To A Galactic Variable

To refer to a data location in another library, you must:

     1.  Be running in a non-zero section

     2.  Have linked your program with a library-specific .REL file  for
         the  library  which  defines the galactic variable to which you
         wish to refer


To refer to the galactic variable, make an indirect reference through  a
"local  galactic  vector" which is provided by the library-specific .REL
file mentioned above.
Dynamic Libraries                                                 Page 6
DYNAMIC LIBRARY FUNCTIONS


Referring to a galactic variable defined in a library not yet mapped  in
will  cause that library to be mapped in (and its master init routine to
be run).



2.3  Global Master Init

This routine should be called from the initialization routine  of  every
top-level  program  (since  on  tops-20  top-level  programs are entered
through entry vectors, whereas dynamic  libraries  are  entered  through
transfer  vectors  pointed to by PDV's, a package can always know how it
was entered, and thus whether it is running at the top level or not).

This finds and calls the master init routine  of  every  copy  of  every
dynamic library mapped into the process address space.

This is necessary  to  make  restarting  work  right  without  incurring
unacceptable overhead.



2.4  Overloading

Merge  information  about  several  dynamic  libraries  into  one  local
transfer vector.

This is an extension of the capabilities you get by manually moving some
EFIW  into  the  local  transfer  vector  in locations where you want to
supercede the routine provided by the library.   For  example,  you  can
cause your transfer vector to contain EFIW's to your private (presumably
improved;  perhaps debugging versions?) of some routines  and  point  to
the library copies of other routines.

The normal rules for overloading are as follows:


     1.  LTVEC entries not containing their default initial  state  will
         not   be   altered.    The   default  initial  state  is  "IFIW
         7777,,LDLBLK"  for  the  monitor   implementation,   or   "IFIW
         LDLBLK-1" for the user-mode implementation.

     2.  If an MTVEC entry contains 0, the corresponding LTVEC entry  is
         not altered.

     3.  If the LTVEC is longer than the MTVEC, excess  LTVEC  locations
         are not altered.

     4.  If the MTVEC is longer than the  LTVEC,  the  additional  MTVEC
         entries are ignored.
Dynamic Libraries                                                 Page 7
DYNAMIC LIBRARY FUNCTIONS


2.4.1  Load MTVEC Into LTVEC

You specify a master transfer vector and a local transfer  vector.   The
master  transfer  vector  is  loaded  over  the  local  transfer  vector
according to the usual rules.



2.4.2  Load Library Into LDLBLK

You specify a LDLBLK whose transfer vectors you want to load  over,  and
an LDLBLK specifying the library you want to load from.  The overloading
is done according to the usual rules.



2.5  De-linking

The monitor DYNLIB% jsys provides functions  for  de-linking  a  dynamic
library.   We  think  that  this  function is not fully thought out.  In
particular, it is not clear how the monitor will find  all  the  LTVEC's
pointing to any given MTVEC.

We do not see any need for this function.



2.6  Finding A Free Provider

When a streaming (single or multi) library is entered at the point  that
causes  a  stream  to  be  initiated,  if  the library is not capable of
initiating another stream, it should  return  an  error.   Newly-written
dynamic  libraries  should  all  be non-streaming or multi-streaming, so
this should not come up in practice.

If this became an important consideration, it might be possible to write
a  routine  that  the  library  stream-initialization routine could call
which would find the LTVEC the call came through (by analyzing the  call
instruction,  whose  successor  address  is  on  the stack), find a free
provider, and re-map the LTVEC to point to  this  provider.   Note  that
this  would lose the effects of any overloading performed on that LTVEC.
This would need a lot more thought before I wanted to commit to it.



2.7  Library Version Checking

The master DLBLK in the library .EXE file contains the  library  version
number  in  location  .DYVER.   It contains the default version checking
rule for that library within the flag word (.DYFLG) in field DY%VER.
Dynamic Libraries                                                 Page 8
DYNAMIC LIBRARY FUNCTIONS


The LDLBLK in the calling program contains the  library  version  number
that  it corresponds to in location .LDVER.  It may optionally contain a
version checking rule (indicated by LD%VMA in .LDVER) in field LD%VER of
.LDFLG, which overrides the default rule specified in the DLBLK.

Based on this information, a check is  performed  when  the  library  is
loaded to see if it matches the version the calling program was compiled
against well enough to be used.  If it does not, an error is printed and
the program terminated.

The version numbers are in standard TOPS-20 version  word  format.   For
version matching purposes, only VI%MAJ and VI%MIN are considered.

If the major and minor versions from both places are the same, then  the
library is accepted.

In addition, the library may optionally be accepted if:

      o  Library  major  version  greater  than  program  major  version
         (VM%MAG)
      o  Library major version less than program major version (VM%MAL)
      o  Major versions equal and library  minor  version  greater  than
         program minor version (VM%MIG)
      o  Major versions  equal  and  library  minor  version  less  than
         program minor version (VM%MIL)

Any combination may be set.



2.8  Dynamic Library Mechanism Version Checking

The master DLBLK in the library .EXE file contains the library mechanism
version number in location .DYFVN.

The LDLBLK in the calling program contains the library mechanism version
number in location .LDFVN.

Based on this information, a check is  performed  when  the  library  is
loaded to see if it matches the version of the dynamic library mechanism
(FAKDYN in the user mode implementation) that is being used.  If it does
not, an error is printed and the program terminated.

The version numbers are in standard TOPS-20 version  word  format.   For
version matching purposes, only VI%MAJ and VI%MIN are considered.

If the major and minor versions from both places are the same, then  the
library is accepted.

In addition, the library may optionally be accepted if:
Dynamic Libraries                                                 Page 9
DYNAMIC LIBRARY FUNCTIONS


      o  Library major version  greater  than  mechanism  major  version
         (VM%MAG)
      o  Library  major  version  less  than  mechanism  major   version
         (VM%MAL)
      o  Major versions equal and library  minor  version  greater  than
         mechanism minor version (VM%MIG)
      o  Major versions  equal  and  library  minor  version  less  than
         mechanism minor version (VM%MIL)

Any combination may be set.

This is applied both to the LDLBLK and the DLBLK.



3.0  IMPLEMENTATION

First I will  describe  the  proposed  changes  for  monitor-implemented
dynamic libraries.

Later, I describe a way to implement the same capabilities in user code,
leaving  an easy way to take advantage of monitor capabilities when they
become available.



3.1  Monitor-mode Implementation

3.1.1  In The User Program

I suggest that all dynamic libraries consist of two user-visible  parts:
the  dynamic  library .exe file, and a .rel file that defines the proper
local dynamic library block for that dynamic library.  Having the  local
dynamic   library   block   available   from  a  rel  file  removes  the
responsibility of constructing it (and  the  possibility  of  making  it
wrong)  from the users of dynamic libraries.  It also makes it easier to
change the local dynamic library block, which will probably be necessary
to  do  when  making  the transition from user-code to monitor-supported
dynamic libraries.

I propose that the local dynamic library block  should  look  like  this
(this is a considerable change in format from our current version):

LDLBLK: length (must be 11 currently)
        Block format version number (to be used in the future to
            provide fancier kinds of dynamic library services such as
            calls by name instead of by offset)
        Flag word (DIGITAL-defined only)
            Initialized: library has been loaded and vectors updated 
        User word
        Pointer to "service class" string
        Pointer to file spec string
        Library version number
        IFIW Address of DIGITAL-specified entry-point vector (LDTVEC)
Dynamic Libraries                                                Page 10
IMPLEMENTATION


        IFIW Address of library-specific entry-point vector (LCTVEC)
        Reserved to DIGITAL
        IFIW Address of library-specific galactic vector (LCGVEC)
        [The possibility of additional entries is reserved to DIGITAL]

Digital-specified entry points will likely include  several  flavors  of
initialization  and  cleanup.   They have been made a separate vector so
that it will be possible to add additional ones in the future if needed.
Note  that  there is nothing requiring that each offset in each transfer
vector point to a unique routine.

LDTVEC: diglen+1
        7777,,LDLBLK
        . . .
        7777,,LDLBLK

LCTVEC: cuslen+1
        7777,,LDLBLK
        . . .
        7777,,LDLBLK
LCGVEC: cgalen+1
        7777,,LDLBLK
        . . .
        7777,,LDLBLK



3.1.2  In The Library

I propose that the dynamic library block (DLBLK) in the dynamic  library
itself should change similarly:

DLBLK:  length (must be 9 currently)
        Block format version number
        Flag word (DIGITAL defined only)
            Busy: library can't start a stream
            Version match: what are acceptable matches between DLBLK
                version and LDLBLK version?
        Library-use word
        Library version number
        IFIW Address of DIGITAL-specified entry-point vector (DTVEC)
        IFIW Address of library-specific entry-point vector (CTVEC)
        Reserved to DIGITAL
        IFIW Address of library-specific galactic variable vector (CGVEC)
        [The possibility of additional entries is reserved to DIGITAL]

DTVEC:  diglen+1
        IFIW routine entry-point
        . . .
        IFIW routine entry-point

CTVEC:  cuslen+1
        IFIW routine entry-point
        . . .
Dynamic Libraries                                                Page 11
IMPLEMENTATION


        IFIW routine entry-point

CGVEC:  cgalen+1
        IFIW address
        . . .
        IFIW address

In addition to the PDV "Dynamic Library" which points to the DLBLK, each
dynamic  library  will have a PDV named "DYNLIB$class", where "class" is
the name of the service class provided.  Class names  containing  a  "%"
are  reserved to DIGITAL.  For example, the sort service dynamic library
would have a PDV "Dynamic Library", and a PDV "DYNLIB$SORT%".

The service-class PDV should point to the DLBLK  just  as  the  "Dynamic
Library"  PDV  does,  so  that  the DLBLK can be found directly from the
service class name.  It should point to THE DLBLK, not to another  copy.
There is only one DLBLK in a dynamic library.

Note:  some convention for tops-20  name-space  should  be  adopted  and
applied  to  these PDV names as well as to all global symbol names.  The
exact standard adopted is much less important than the adoption of  SOME
standard.  The details above are primarily intended as examples.

Note:  The PDV name "Dynamic Library"  should  belong  to  DIGITAL  name
space, not customer name space.  Rather than explicitly listing it as an
exception to the rules, the name should be changed so that it  falls  in
DIGITAL name space.



3.1.3  JSYS Interface

Not yet defined.  The details of this  aren't  nearly  as  important  to
settle  as  the  capabilities  and data structures for normal use.  Note
that explicit calls  to  the  JSYS  interface  will  be  very  much  the
exception, not the norm.



3.2  User-mode Implementation

One goal of our FAKDYN procedure must be that it can be  replaced  by  a
monitor-supported  dynamic  library procedure with minimum effort.  This
FAKDYN  proposal  could  be  converted  to  a  monitor-supported  DYNLIB
(assuming  the  monitor  buys  the  changes  proposed  above,  which are
independent of FAKDYN anyway) by re-linking existing programs with a new
LDLBLK rel file.  No code changes in any program would be required.

To use FAKDYN,  there  will  be  the  additional  requirement  that  all
packages calling dynamic libraries be linked with FAKDYN.REL.
Dynamic Libraries                                                Page 12
IMPLEMENTATION


There is one restriction under FAKDYN:  a library cannot  be  loaded  in
response  to a reference to a galactic variable defined in that library.
The recommended workaround is to call a routine in  the  library  before
attempting  reference to a galactic variable.  Testing the "initialized"
bit in the LDLBLK could increase the efficiency of this.

First, extend the LDLBLK with a FAKDYN kludge word:

|       PUSHJ P, FAKDYN
LDLBLK: length (must be 11 currently)
        Block format version number
        Flag word (DIGITAL-defined only)
        User word (Can be address of user data block)
        Pointer to "service class" string
        Pointer to file spec string
        Library version number
        IFIW Address of DIGITAL-specified entry-point vector (LDTVEC)
        IFIW Address of library-specific entry-point vector (LCTVEC)
        Reserved to DIGITAL
        IFIW Address of library-specific galactic vector (LCGVEC)
        [The possibility of additional words is reserved to DIGITAL]

LDTVEC: diglen+1
        IFIW LDLBLK-1
        . . .
        IFIW LDLBLK-1

LCTVEC: cuslen+1
        IFIW LDLBLK-1
        . . .
        IFIW LDLBLK-1

LCGVEC: cgalen+1
        IFIW LDLBLK-1
        . . .
        IFIW LDLBLK-1

At FAKDYN entry, stack will contain:

        LDLBLK                  -1(SP)
        user return             0(SP)

LDLBLK contains all the information necessary to map in (if necessary) a
dynamic  library, and to update the local transfer vectors.  After doing
this, FAKDYN decrements the user return to point back to the instruction
that  called  the  routine,  and returns to re-execute that instruction.
Since the local transfer vectors have now been updated, the re-execution
of the instruction will result in calling the specified routine.
Dynamic Libraries                                                Page 13
SECTION ALLOCATION


4.0  SECTION ALLOCATION

Another area that needs addressing is section allocation.   Since  which
sections  are  in  use  has a serious impact on performance, the section
allocator should be a piece of digital-supplied code, preferably in  the
monitor  (where  it  will  know  what processor it is running on and can
allocate accordingly).  (Another advantage of putting this code  in  the
monitor  is that an EXE file can be transported to another system with a
different processor and start allocating sections in a  way  appropriate
for the processor it is running on.)



5.0  DIGITAL ENTRY POINTS

A need  for  the  following  DIGITAL-specified  entry  points  has  been
determined.



5.1  Master Init

DIGITAL  entry  point  1  is  "master  init".   Master  init  is  called
automatically  by  FAKDYN (and eventually by the monitor) when a library
is  first  mapped  in.   If  a   library   does   not   require   master
initialization,  that  entry in the DTVEC should point to a routine that
simply  returns.   The  entry  should  not  be  0.   (This   restriction
eliminates  the  need  to  check  before each and every call whether the
entry vector is valid.)

The master init routine does not accept any arguments.  It must preserve
all  registers.   It  may assume that there is a global stack pointer in
AC17.  What, if any, assumptions may be made about the amount  of  space
available on the stack have not yet been decided.

(In addition to the master init entry point, most libraries will have an
ordinary  init  or  a  stream-init  entry  point.   Do not confuse these
limited initializations with the master init performed once per  run  or
start command.)

A routine will be provided which runs the master init on  every  library
currently  mapped  in.  This routine should be called by every top-level
program that  uses  dynamic  libraries  in  its  initialization  section
(before it calls on any dynamic library services).

This takes care of the need to re-initialize on restarts without  having
to  re-map  all  the  libraries  in  use,  and without having to know in
advance which libraries are going to be used.

Note that, on TOPS-20, a program must be written  to  run  either  as  a
top-level,  or  as a subroutine.  A program can be written to be invoked
either way, but in that case it always knows which way it  was  in  fact
invoked  on  this  run.  Thus, it is meaningful to specify actions to be
taken only if you were called as a top level program.











                               APPENDIX A

                              BIBLIOGRAPHY



[1]
     Murphy, Dan
     Dynamically Linked Libraries in TOPS-20 
     DIGITAL interoffice memorandum, March 1981

[2]
     Wallace, Kevin Girard
     Canonical Terminal Support in the TOPS-20 Operating System
     Master's thesis, MIT, May 1982

[3]
     Uhler, Mike
     Extended Addressing
     DIGITAL interoffice memorandum, January 1983

[4]
     Mierswa, Peter
     Layered Architecture
     DIGITAL interoffice memorandum, May 1983

[5]
     Wallace, Kevin G.
     More on Dynamically Linked Libraries
     DIGITAL interoffice memorandum, December 1981

[6]
     Wallace, Kevin G.
     Functional Specification for Canonical Terminal Support
     DIGITAL interoffice memorandum, August 1982

[7]
     Wallace, Kevin G.
     Design Specification for Canonical Terminal Support
     DIGITAL interoffice memorandum, August 1982