Trailing-Edge
-
PDP-10 Archives
-
QT020_T20_4.1_6.1_SWSKIT_851021
-
swskit-documentation/performance.memos
There are 5 other files named performance.memos in the archive. Click here to see a list.
MEMOS CONTAINED IN THIS FILE:
o DECNET-IO-PERFORMANCE.MEM
o OFN Managenment changes
----------------------------
I have made some tests of DECnet IO performance for 6.1. These test
try only to quantify the DECnet throughput, not the total system
throughput. The first section contains a summary, the second the
actual figures and the third one describes exactly how I ran the tests
so someone else can reproduce them.
Summary
=======
I ran three different tests: SENDER/SERVER, DTS/DTR and NFT/FAL.
SENDER/SERVER is a program that just measures DECnet throughput. It
does not write to disk or verify the data; it justs sends data at
maximum rate to the other end.
DTS/DTR is the traditional DECnet test. It does not write to disk,
but it uses more cpu resources than SENDER/SERVER when it runs.
NFT/FAL is the standard file transfer, and is as such interesting from
a performance point of view. It writes to disk, and checksums each
message so several other resources of the system than DECnet are
exercised.
Now for the figures:
DTE 51 kbaud with all tests. (56Kbaud connections).
CI 435 kbaud with SENDER/SERVER,
365 kbaud with DTS/DTR
190 kbaud with NFT/FAL
NI 860 kbaud with SENDER/SERVER
485 kbaud with DTS/DTR
220 kbaud with NFT/FAL
It seems as if we easily can saturate the DTE. The difference between
the various tests across the CI and NI is caused by the CPU usage by
the respective tests. While SENDER/SERVER runs approximately twice as
fast on the NI than on the CI, the CPU usage in NFT/FAL will smooth
out the difference to be only 30 kbaud.
The big difference between the NI and CI speeds comes mainly from
using "big buffers" on the NI. The buffer size for the DTE and CI is
576 bytes, while each message across the NI is packaged in 1467 bytes
large chunks.
Note: the CI does one more copy of each message on output than what is
needed for the NI, and the CI is therefore less efficent than the NI.
This difference, however, does not show up as any performance
difference when the CI and NI use the same buffer sizes.
The actual figures
==================
The monitor used was a DECnet, non-ARPA, DEBUG=0 monitor, version
6.1(6572).
All systems used (ETHER/CHIP/DALE) have 1.5 Meg MF20, no MCA25 and
runs KL ucode 352 or 400.
Normal timesharing was up, i.e. GALAXY and other system components
were running. No other users were present on the system.
DTE and CI test:
The systems ran as DECnet routers during the DTE and CI tests, but the
systems were disconnected from the Engineering net so the only nodes
present were the participating ones. The CI circuit was turned off
while the DTE test were run, and vice versa. CI and CFS were enabled
on the systems.
NI test:
The systems ran as ETHERNET endnodes on the private ETHERNET during
the tests. LATOUR was the only other DECnet node up. Large buffers
were used. CI was disabled on one of the systems.
DTE: CHIP <-> CHIPDN <-> DALEDN <-> DALE
========================================
Test 1 Test 2
Bps # of msg % cpu Bps # of msg % cpu
SENDER/SERVER 51570 3448 4 51375 3433 4
DTR/DTS 51169 3433 51535 3426
NFT/FAL 51013 50252
CI: CHIP <-> DALE
=================
Test 1 Test 2
Bps # of msg % cpu Bps # of msg % cpu
SENDER/SERVER 434726 29066 49 435274 29200 49
DTR/DTS 370097 24811 362888 24353
NFT/FAL 194244 183649
--o--------o--
| |
ETHERNET: CHIP ETHER
========================
Test 1 Test 2
Bps # of msg % cpu Bps # of msg % cpu
SENDER/SERVER 852909 22058 38 876013 22580 39
DTR/DTS 475976 12313 500475 12945
NFT/FAL 229562 214909
Test procedure
==============
SENDER/SERVER
-------------
The sources to the SENDER/SERVER programs are in SNARK:<PERF-TOOLS>
DECNET-SENDER.MAC and DECNET-SERVER.MAC.
The SENDER/SERVER programs tries to drive DECnet at maximum speed.
They will not write to disk, or verify the data, but only send data
from one end (SENDER) to the other (SERVER). Messages are not looped,
all traffic is one way. The message size used will be the maximum
allowed by the segment size.
The tests were run twice for each circuit type (DTE/CI/NI) for 5
minutes each.
To run this test, do the following:
On the SERVER side:
RUN SERVER
Nothing else is needed.
On the SENDER side:
RUN SENDER
Remote node name: TEST
Number of segments per message: 1
Link quota: 16 (means 8 for xmit, 8 for receive)
Flow control to use (No = 1, Segment = 2) : 1
Number of seconds for test: 300
And then wait till the test is over. Verify that you get the segment
size you expected (SENDER/SERVER will tell you).
The number 8 was selected because that is the default buffer quota for
users. A system manager can change this by a SETSPD command.
Increasing the buffer quota improves the DECnet performance
substantially.
DTR/DTS
-------
These are the traditional DECnet test programs. A DTR (receiver) is
usually started on all systems as part of the system startup. If it
is not, do the following:
@DTR
*DECL 63/BYTE:8
*WAIT
I ran two tests of DTS/DTR for five minutes each.
To run DTS:
@ena
$dts
3-Mar-85 13:05:54 * [DTS STARTING]
DTS>test ronco::/test:data
DATA_TEST>sink /msg:559 /rflow:none /rqueue:8 /sflow:none /squeue:8 /time:0:5:0
# DTS: DATA TEST STARTING
# DTS: NODE NAME - ronco
.....
Use /MSG:1450 when you run it across the NI with large buffers.
NFT/FAL
-------
I simply copied a large file over and over and timed the elapsed time
with a simple DAYTIME command. I then took the average of all the
transfers. Make sure that you use a text file; I first used a dump
but NFT/FAL was smart enough to realize that there were some zero
pages in the EXE file, and used that fact to optimize!
Notes
-----
I ran into two different little problems: disk space was low on one
system and a global expunge happened during one of the tests. No
message was generated because the system was set to suppress output of
such message. DEcnet performance was only half of what it typically
was, and it took me a long time to figure out what had gone wrong.
The other thing was that when bringing up the systems as endnodes with
large buffers, you will not use large buffers the very first time you
run your tests. The other system is not in the "cache" at that point
so the system will not know that it is safe to use the larger buffers.
So make sure that you get the larger buffer size during the NI tests.
OFN.MEM
Design and internal description of the
Release 6.1 OFN management changes
1-Aug-85
Version 1 - JFNOFN and OFNJFN
13-Nov-84 Tom Moser
Version 2 - File system "caches"
1-Aug-85 Tom Moser / Dave Lomartire
CHANGES TO OFN MANAGEMENT FOR PERFORMANCE Page 2
CONTENTS
1.0 JFNOFN AND ASSIGNING OFNS . . . . . . . . . . . . . 3
2.0 OFNJFN . . . . . . . . . . . . . . . . . . . . . . . 3
3.0 FILE SYSTEM CACHES . . . . . . . . . . . . . . . . . 4
3.1 OFN SHARE COUNT BECOMES 0 . . . . . . . . . . . . 5
3.1.1 RELOFN . . . . . . . . . . . . . . . . . . . . . 5
3.1.2 RELP4 . . . . . . . . . . . . . . . . . . . . . 5
3.2 COUNT OF REAL OPENERS BECOMES 0 . . . . . . . . . 6
3.3 COUNT OF REAL OPENERS BECOMES <> 0 . . . . . . . . 6
3.4 What Is OFN Caching . . . . . . . . . . . . . . . 6
3.5 FREOFN . . . . . . . . . . . . . . . . . . . . . . 6
3.6 CSHOFN . . . . . . . . . . . . . . . . . . . . . . 7
3.7 RELOFN - Disc File CLOSF . . . . . . . . . . . . . 7
3.8 UNCOFN . . . . . . . . . . . . . . . . . . . . . . 7
3.9 OPEN FILE COUNT FOR OFN2XB . . . . . . . . . . . . 8
3.10 IMPLEMENTATION . . . . . . . . . . . . . . . . . . 8
4.0 CFS FILE ACCESS TOKEN CACHING . . . . . . . . . . . 8
4.1 General CFS Token Management . . . . . . . . . . . 8
4.2 File Access Token Management . . . . . . . . . . . 8
4.3 File Access Token Caching . . . . . . . . . . . . 9
4.3.1 HSFLAG . . . . . . . . . . . . . . . . . . . . . 9
4.3.2 CFSRTV . . . . . . . . . . . . . . . . . . . . . 9
4.3.3 CFSAWT/CFSAWP . . . . . . . . . . . . . . . . . 9
4.3.4 CFSUWT . . . . . . . . . . . . . . . . . . . . . 9
CHANGES TO OFN MANAGEMENT FOR PERFORMANCE Page 3
The following describes the changes that were made to OFN handling to
improve performance of the 6 series TOPS-20 monitors.
1.0 JFNOFN AND ASSIGNING OFNS
JFNOFN is called whenever a user attempts to convert from JFN,,PN to
OFN,,PN. It is a fast subroutine in the case of short files or, where the
OFN is in the same file section (Group of 512 pages represented by 1 slot
in the Page table). When the OFN is that for a different file section, the
monitor must "switch" the OFN. The switching is accomplished by calling
NEWLFS, which eventually leads to ASLOFN.
ASLOFN assigns OFNs, but in order to do this, it must scan all OFNs in
use. Typically there are 200-400 OFNs in use, but this can rise to over
1000 on some systems. OFNs are searched by a linear scan. We can estimate
the following behavior:
(Let MAXSPL = number of OFNs in use)
A successful search should take on average:
MAXSPL / 2 lookups (actually because of OFN allocation on TOPS-20 I expect
more like 2/3 MAXSPL) (100-250)
All unsuccessful searches take MAXSPL lookups (200-400 up to 1000)
The linear scan was replaced with a "hash and link" algorithm, such as
the one used by CFSSRV for resource lookup. This causes:
(Let HS represent hash table size, perhaps 59)
A successful search should take on average:
MAXSPL / (HS*2) if MAXSPL = 300 (this is about 2.5)
A unsuccessful search would take on average:
MAXSPL/HS if MAXSPL = 300 (this is about 5)
This change benifits short file opens as well as long file accesses, since
all OFNs must be searched when a short file is opened.
2.0 OFNJFN
When the monitor needs to find a JFN that is associated with a particular
OFN it calls OFNJFN. This routine is fairly fast if the user has nothing
open except short files. For long files, it scans each PTT looking for the
disk address that matches the OFNs. This can be VERY time consuming.
CHANGES TO OFN MANAGEMENT FOR PERFORMANCE Page 4
Users who have long files open may see performance degradation even if they
are accessing only short files because of the scanning in OFNJFN. Instead
we remember the OFN of the PTT for all long file OFNs, and scanning the
JFNs for this OFN in the case of OFNJFN on a long file second level PT.
This requires that we remember the PTT OFN for all second level OFNs. This
new data is kept in anew monitor table SPTO4 which is parrellel to the OFN
area of the SPT. SPTO4 has the following format.
OFN of PTT,,File section number (file is a 2nd level OFN for long file)
or
0 (PTT OFN or reguler short file OFN)
or
-1 (OFN is unassigned)
3.0 FILE SYSTEM CACHES
The monitor incurs some overhead involved with the creation and
deletion of certain objects (notably OFNs and CFS "tokens"). The overhead
can become significant in the cases of random long file access and using
the OPENF Jsys as a file locking mechanism.
For OFNs most of the overhead occurs when the "OFN open count" or the
"OFN share count" transitions from or to 0 (decremented to 0 or incremented
from 0). These counts are determined as follows:
OFN share count (OFNOPC) =
Number of pages mapped in this OFN +
Current OFN count (number of users of this OFN who have it stored in FILOFN) +
1 If this OFN is file section 0 of a long file
OPEN file count (OFOPC) =
Current OFN count (number of users of this OFN who have it stored in FILOFN) +
1 If this OFN is file section 0 of a long file
NOTE: THE OPEN FILE COUNT IS ONLY INCREMENTED BY USERS WHO HAVE THE
FILE OPEN FOR A MODE OTHER THAN "READ UNRESTRICTED". IT IS COMETIMES
CALLED THE "COUNT OF REAL OPENERS"
A list of the significant modifiers of of the overhead items appears
here along with the routine active and a description of the interesting
modification.
CHANGES TO OFN MANAGEMENT FOR PERFORMANCE Page 5
3.1 OFN SHARE COUNT BECOMES 0
3.1.1 RELOFN -
Routine RELOFN label RLOF00. This routine is the most common cause of
this event. OFNs are released by the File close code and by the Long file
PT switching code (NEWLFx). RELOFN performs the following actions when
OFNSHC becomes 0.
1. SCNOFN is called to move all of the file pages to disc (SCNOFN
will be discussed later in detail).
2. CFS resources are freed (FRECFS).
3. OFN slot is deleted and pages are removed from core (DASOFN).
4. Cached disc pages are freed (DSKUGT).
5. ALLOC entry is deassigned (DASALC from DASOFN).
6. OFN is updated if not deleting (UPDOFI)
3.1.2 RELP4 -
Routine RELP4. This code is only executed if the OFN share count
becomes 0 during the unmapping of a page contained in that OFN. For Short
files this SHOULD NEVER OCCUR and for long files this occurs only when a
page is mapped in a non-first file section and the file section is
"switched" causing the share count to be decremented. RELP4 does the
following when OFNSHC becomes 0.
1. Cached pages are freed (DSKUGT) [only if opened DUD!]
2. ALLOC entry is cleared (DASALC [direct call]).
3. SCNOFN is called IF the file is opened DUD. This is a strange
thing to do but is somewhat consistent with one view of DUD usage.
4. If the OFN is modified (OFNWRB=1) then the OFN is written to disc
5. The OFN is NOT deleted (semi cached!) it can be removed by FREOFN
later if needed.
6. ***** FRECFS is NOT called - This may be a bug!!!
CHANGES TO OFN MANAGEMENT FOR PERFORMANCE Page 6
3.2 COUNT OF REAL OPENERS BECOMES 0
This event only occurs in routine RELOFN when it occurs the file is
"unrestricted" by calling routine CFSURA to "downgrade" the file access to
promiscous read.
3.3 COUNT OF REAL OPENERS BECOMES <> 0
This event occurs only in routine ASNOFN when this occurs the monitor
assumes that a real opener of the file has come along after the file was
open for "unrestricted read" this causes a call to UPDPGY to dump pages
todisk and to force a rereading of their contents on the next access. This
is fine for the case where a true unrestricted reader existed but causes
much unneeded disc traffic when accessing 2nd level OFNs.
3.4 What Is OFN Caching
Ofn caching is the artificial keeping of OFNs as if their use counts
were non-zero. The monitor insures that data integrety is maintained but
does not remove the OFN data. Cached objects are marked as such and can be
considered as "unused" when assigned and available when requested. The
keeping of data about OFNs speeds up user access to file data by reducing
the overhead of creating and deleting data and by eliminating the need to
unnecessarily read and write the disc.
Caching requires that disc integrety is maintained and that the
resource is marked as cached by the routines that formerly managed the
relenquishment of and deletion of a resource. It also requires that the
routines which manage the creation or use of a resource correctly "uncache"
resources which transition from "cached" to "in use" (cache hits). This
requires changes in FREOFN, SCNOFN, RELOFN, RELP4 (RELMPG), and ASNOFN. It
requires implementing new routines to cache and uncache OFNs, CSHOFN and
UNCOFN.
3.5 FREOFN
FREOFN is currently called whenever OFNs run out. OFN caching will cause
OFNs to run out very frequently because the OFN table will fill up with
"stale" OFNs. The following algorithm will implement periodic and
on-demand gagbage collection within FREOFN.
1. Called at a different entry every 10 mins from CLK2CL.
2. Marks all cached ofns with 10 minute timestamps
3. If an OFN so marked is DUMPABLE then DUMP it (not used for 10-20)
mins
CHANGES TO OFN MANAGEMENT FOR PERFORMANCE Page 7
4. If a cached marked OFN is found (not used for 20-30 mins) then
remove it (UNCOFN)
5. If OFNs run out during assignment by ASNOFN then
1. remove all "10+ minute OFNs"
2. If this does not work remove all cached OFNs
3.6 CSHOFN
This new routine is called by RELP4 and RELOFN when the share count is
about to become 0. This routine does the foollowing:
1. frees cached disc pages DSKUGT
2. Marks OFN as cached and dumpable (New bits in SPTO2)
3. CFS resources are freed (but remain cached see section 4)
This preserves the ALLOC entry, the OFN itself and any in core disk
pages.
3.7 RELOFN - Disc File CLOSF
RELOFN is called by the file close code and is responsible for
updating files on disk during the final close. The Disc close clode will
be changed to call RELOFN at a special entry point to cause SCNOFN to be
run. SCNOFN will cause the disc to be updated. SCNOFN will also clear the
OFN dumpable bit indicating that the OFN needs dumping to disc.
3.8 UNCOFN
UNCOFN will be called to totally remove a cached OFN. It will preform
the following actions:
1. CALL SCNOFN to insure disc integrity if the OFN is dumpable
2. CALL UPDOFI
3. CALL DASOFN (and indirectly DASALC)
CHANGES TO OFN MANAGEMENT FOR PERFORMANCE Page 8
3.9 OPEN FILE COUNT FOR OFN2XB
The code will be changed to base all decisions on the OFOPC of the
second level OFN for page table 0 or failing that the OFOPC for the OFN of
the super XB. CFSURA and UPDPGX routines will be changed to perform
special actions when called with a second level OFN. They will loop
through all second level OFNs for that file performing their correct
actions.
3.10 IMPLEMENTATION
This will be implemented for Autopatch tape 1 of release 6.1. All
needed bits will be added to SPTO2 and if necessary SPTO4.
4.0 CFS FILE ACCESS TOKEN CACHING
4.1 General CFS Token Management
In CFSSRV, the routines CFSRMV and CFSRMX remove CFS tokens from the
hash table. CFSRMX is exactly like CFSRMV except that CFSRMX will "post"
the removal. (File access tokens and directory allocation tokens are the
only ones to have a post address and to call CFSRMX.) However, not all
tokens are removed from the hash table when the system is done with them.
Directory locks are a good example. If a system is trying to acquire a
directory lock token, and it is already on the system and not being used,
then the system instantly acquires it. No voting or resource block
allocation is necessary. This is correct because this system has not lost
it's previous access to the directory since the token is still on the
system. The only way that the directory lock token will be removed from
the hash table via a call to CFSRMV is by the voter (when it is going to
give up the token to another system) and by the garbage collection routine,
CFSRSE, which gets rid of "stale" entries whenever more space is required.
This scheme allows for quick resource acquisition in cases where the system
stops using a resource but never loses access.
4.2 File Access Token Management
CFS handles file access tokens a bit differently than the others. The
routine CFSUWT is responsible for calling CFSRMV to remove file access
tokens once they are deemed as no longer needed. The PAGUTL routine FRECFS
calls CFSUWT. Therefore, whenever OFNs are being released, FRECFS will be
called and the file access token for that OFN will be removed from the
system. Any subsequent references to the same resource will require voting
and resource block creation regardless of whether or not the system ever
actually lost it's access. CFS has no memory with respect to file access
tokens.
CHANGES TO OFN MANAGEMENT FOR PERFORMANCE Page 9
4.3 File Access Token Caching
This proposal details a scheme by which file access tokens would be
cached in order to reduce needless voting and resource block creation. The
changes required would be as follows:
4.3.1 HSFLAG -
HSFLAG would have a bit added to it to indicate that this token has
been cached. The voter and file access token acquisition routines
(CFSAWT/CFSAWP) would be the primary routines to check this bit.
4.3.2 CFSRTV -
The voter would check to see if the incoming request specifies a
cached token. If so, then CFSRMV would be called to remove it from the
system and the voter would immediately return a "Yes" to the vote request.
4.3.3 CFSAWT/CFSAWP -
These routines would notice that the system was attempting to acquire
a cached token. Therefore, the block would be "uncached" and the routine
would continue just as if this block had never been cached. The
"uncaching" will set the block back into the state that CFS expects a
normal file access token to be in. Namely, HSHKPH will be set and HSHCOD
will be updated with the current OFN. (Later in the CFSAWT routine, HSHPST
will also be reset.) The only voting which is now required is in the case
of the new access not being the same as the access held when the token was
cached. If the access acquired upon caching was exclusive (.HTOEX), no
voting will be required.
4.3.4 CFSUWT -
This routine will no longer call CFSRMV to remove the resource block.
Instead, it will cache the block by setting the "cache" bit. Also, the
HSHKPH bit (prevent this block from being garbage collected) will be
cleared to allow these blocks to be removed in the event that the system
gets low on space and garbage collection is required. Normally, file
access tokens always have HSHKPH set. The count (HSHCNT) and the post
address (HSHPST) will be zeroed as part of the caching.