by Richard F. Drushel (drushel@apk.net)
As promised in last week's TWWMCA (9710.06), I will respond here to some interesting and important questions raised by Ron Mitchell in the 5 October 1997 installment of his article series, Ron's Week'n'ADAM. I will do this in USENET style, with quotes from Ron's article prefaced by >, and my commentary following. This is also known as Point-Counterpoint style, after the old segment on the CBS news magazine 60 Minutes featuring opposing viewpoints by (I think) James Kilpatrick (a curmudgeon) and Shana Alexander (Bambi in the headlights of the oncoming truck). Unlike the originals (who inspired a parody on NBC's Saturday Night Live in which Dan Akroyd, in the Kilpatrick role, responded to Jane Curtin with "Jane, you ignorant slut!"), I will not be mean or nasty. While Ron says, about his original article,
>There is gut reaction >and there is, I think, more mellow and considered comment.
I have no difficulty in looking past the "gut reaction" and recognizing the "mellow and considered comment".
The following analysis is intended to be objective. It may come across as cold-blooded and cruel, but that is a by-product of my desire to be objective, not from a desire to denigrate the work of others. I'm putting on my "scientist" writing hat here, so my own work is as much under the gun as anybody else's.
A few TWWMCAs ago, in a survey of ADAM operating systems, I opined that TDOS (Tony's DOS, a CP/M-compatible operating system) in its current source code state is an unmaintainable mess of "spaghetti code", and that it would be better to start anew to build an ADAM-compatible CP/M than to make further use of the existing TDOS source code. To which Ron responded:
>The part about TDOS in Rich's work may indeed be accurate, but >the manner in which it is presented leaves the impression that >TDOS co-authors Tony Morehen and Guy Cousineau somehow produced a >product that was less than satisfactory. Having known these two >talented men as friends for several years, I do not believe they >wrote any 'spaghetti' code whatever. I believe that they wrote >code that made sense to them, and I know that code continues to >perform yeoman service on several ADAM computers here. In fact, >had it not been for TDOS, I would have dumped ADAM long ago.
I'm sure Ron knows this anyway, but I'll say it for the record: I am *not* knocking Tony Morehen and Guy Cousineau. Both are skilled programmers whose works are still in active, productive use. The ADAM community would be much diminished were it not for their efforts.
However, both Tony and Guy have moved on, leaving behind their programs for others (namely, me) to maintain. What does "maintain" mean? It means:
fix any bugs that may be discovered;
modify to support additional hardware that may be invented;
modify to add extra features that may be desired by users; and
modify to coexist with any changes to system software (e.g., operating system patches) which may be required by applying (1)-(3) to the operating system itself.
All program maintenance requires adding or modifying program code. This may be done in two ways:
modify the source (SmartBASIC, C, Pascal, Z80 assembler) and recompile/reassemble to a new executable binary file; or
patch the current executable binary file directly.
Modifying the source is the method of choice, because the source is typically written in a high-level language (like SmartBASIC) or, if in Z80 assembler, with useful variable names and subroutine names and some comments to explain how the program works. If all you have is a binary file, however, you must disassemble the binary to recreate a source file, then attempt to follow the program flow yourself (as if you were a Z80 CPU), and attempt to recreate and discover the thinking of the programmer. Even if you aren't doing a complete reverse-engineering of the program to source code (rather just disassembling what you think is the part you need to know about for a particular patch or bugfix), you have a difficult task--even with a "smart" disassembler tool like Kenneth Gielow's Z80DIS22 (which runs under CP/M).
In any case, whether you have the original source or a reverse-engineered source, your task as maintainer of someone else's code is to be able to read the code well enough to know what it's doing, how it's doing it, and whether there are any important side effects, so that the changes you make to it (1) do what you intend, and (2) don't break anything else by accident.
This task is greatly influenced by the underlying organization of the original code. If the original code is well-organized, hierarchical, modular, with related subroutines located close together, then it is easy to read. If, however, the code is cluttered, inconsistent, tricky (e.g., uses overlapping reading frames, self-modifying code, self-relocating code, overlays, conditional assembly, or many disjoint source files), and jumps all over the place, then it is very difficult to read. If the original source lacks comments (text explanations of what the program is doing at a particular point, or overviews of its organization), then the source is little better than a disassembly listing in terms of its readability.
The latter kind of code is the programming style referred to as "spaghetti", because like a plate of spaghetti noodles, it winds and twists and turns all around, with little hint of organization (and no comments or explanations from the programmer as to why it works or even what it's trying to do).
The TDOS source is such a "spaghetti" program, because
its internals are mostly uncommented;
it has an "inline" versus "modular" design;
it has a confusing amount of conditional assembly; and
the source is broken up into a large number of files without an obvious organization.
Ron asks:
>1) [...B]y what standard is [spaghetti code] judged?
The answer is, by the people who have to maintain the code :-) Writing code in any assembly language is a difficult task, and even the best-designed, best-commented code may be rather unintelligible to its author 6 months later. (This is absolutely true for me.) Anything which makes the code easier to understand, be it comments or helpful variable names, is a boon to the code maintainer (even if that is the original author).
>2) What activities are included in the process of managing or > maintaining source code, and why are they important, given > that the product is out there and serving?
A program can be perfectly functional and easy-to-use from the end user standpoint, yet a nightmare of twisted "spaghetti" on the inside, completely hidden from the end user. If the end users are completely satisfied with the current operation of the program, and never demand either a change in its features or a change in its operating environment, then there is nothing to maintain, and the internal organization of the program is irrelevant. It is only when a maintainer is asked to fix a bug, or add a new feature, or ensure a backward compatibility, that the aims of the end user and the aims of the maintainer come into conflict. The first bugfix or hack or two can usually be made easily; but repeated patching upon patching results in a very "fragile" program--like a house of cards, the next card added may bring the entire structure down. And patches may have unforseen consequences which limit future patches.
>3) What is modular code, and why is having too many modular > source files a bad thing? After all, shouldn't we be > carrying only the baggage that we need for a given application?
"Modular code" means that the code is organized into basic building blocks of functionality, and that unlike functions are not grouped together. The modules are often hierarchical, and higher-level functions are built up from calls to lower-level functions. Consider the following functional organization of the file system of ADAM's EOS operating system:
_MAKE_FILE create a file _DELETE_FILE delete a file _OPEN_FILE open a file _READ_FILE read from a file _WRITE_FILE write to a file _CLOSE_FILE close an open file _POSITION_FILE move the read/write pointer in a file _RESET_FILE reset the read/write pointer to the beginning _TRIM_FILE free up unused blocks at the end of a file _QUERY_FILE ind a file without file type _FILE_QUERY find a file with file type _FMGR_INIT initialize the file control blocks (FCBs) _INIT_TAPE_DIR initialize the directory of a disk or tape
_SCAN_FOR_FILE see if a file exists _SET_FILE update the directory entry for a file _CHECK_FCB see if a FCB is in use (i.e., if a file is open) _MODE_CHECK see if an open file is open for read/write/execution
_READ_BLOCK read a block from disk or tape (with 3 retries) _WRITE_BLOCK write a block to disk or tape (with 3 retries)
_RD_1_BLOCK read a block from disk or tape (no retries) _WR_1_BLOCK write a block to disk or tape (no retries)
_START_RD_1_BLOCK ADAMnet: start reading a block _END_RD_1_BLOCK ADAMnet: check if we're done reading a block _START_WR_1_BLOCK ADAMnet: start writing a block _END_WR_1_BLOCK ADAMnet: check if we're done writing a block _FIND_DCB ADAMnet: find the device control block (DCB) _REQUEST_STATUS ADAMnet: check the ADAMnet status of a device _SOFT_RES_DEV ADAMnet: reset a device _READ_DEV_DEP_STAT ADAMnet: read device-dependent status
Note that higher levels provide more and more abstraction and insulation from the underlying hardware. This allows for portability for programs which only use the high levels; the low levels can be replaced with driver code for totally-different, non-ADAMnet hardware, as long as the same parameter passing conventions are observed. For example, ADAMserve breaks into _RD_1_BLOCK and _WR_1_BLOCK to reroute requests for ADAMnet device I/O to server devices, if appropriate. (It also maintains some dummy DCBs for the benefit of programs that try to read status data from ADAMnet directly by PEEKing at the DCBs.)
To return to TDOS, part of the reason that I cannot "easily" come up with an ADAMserve version of TDOS, even given the complete original source code, is that nowhere in TDOS is there the equivalent of _RD_1_BLOCK and _WR_1_BLOCK. Perhaps to save space or gain speed, TDOS talks directly to ADAMnet, without any layers of abstraction like _READ_BLOCK (for block devices like disks and tapes) or _READ_CHAR_DEV (for character devices like keyboards and printers). I have not been able to spare the time or mental effort to find a "safe" way to either add this layer of abstraction or branch around the direct ADAMnet I/O code. It is not impossible, just not as straightforward and guaranteed-to-be-foolproof as rerouting _RD_1_BLOCK and _WR_1_BLOCK in EOS.
These direct accesses to hardware are called "inline" program design, because the code isn't written as a subroutine which is CALLed; it's in the direct line of program flow. When optimizing a program for speed, inline design is the way to go, because you avoid the overhead of the CALL and the RETurn from the subroutine. If you must do the same thing at several different points in the program, however, you end up repeating the same bits of code over and over; thus, the program is larger than if you have a single subroutine which is CALLed whenever needed. Subroutines are a way to optimize a program for size, because the overall code takes fewer bytes of machine code.
The programmer must decide where to optimize for speed and where to optimize for size. Optimizing for size can create dense, tricky, unreadable code that exploits every possible side effect of each machine code instruction. This is where helpful comments that describe the intent of the code, perhaps even show it in a straightforward, unoptimized version, are invaluable to the code maintainer. Optimizing for speed is important for timing-dependent operations (such as reading characters from a serial port at high bitrate and storing them in a buffer). For instance, the code in the terminal mode of ADAMlink IValpha and V has a 1-byte subroutine call (RST) to the serial port read/buffer routine about every 7 other instructions--the Orphanware and MI serial cards do not support interrupt-driven I/O, so the serial port must be polled at a fast rate in order to avoid losing characters at 19200 bps.
Ron comments:
>In all fairness, it could be noted here that Guy Cousineau once >remarked (mused, lamented, complained) that TDOS was approaching the >state where perhaps too many variations were 'out there' being >used for one purpose and another. I think he realized that the >'monster' was growing. In my mind however, that in no way >diminishes the immense utility of TDOS in this place here. Each >of my ADAM systems has a different setup. I have boot disks for >each variation. It is not a problem.
Again, end-user utility is not necessarily a function of the elegance (or not) of the internal organization of the program. As for having boot disks for each hardware setup variation, this is fine as long as either (1) someone has assembled a version of TDOS for your particular setup, or (2) you have the TDOS source yourself and you "roll your own" version as needed. (2) is the typical situation for the general CP/M user these days; however, the TDOS source has *not* been released for general distribution. (I don't know if Dale Wick was "authorized" to let me have a copy, but that's where my copy came from.) In addition, the TDOS source is set up exclusively for the Z80ASM+ assembler from SLR, which, while an excellent and powerful assembler, (1) cost $150 when it was still available (I bought a copy at full price from Jay Sage at Sage Microsystems East), and (2) is no longer available now that SLR has sold out to Symantec. Porting the source to a different (freeware) assembler does not look to be an easy task. The upshot is that the average TDOS user does not have the means to create non-standard versions of TDOS for unusual hardware configurations, or novel configurations (such as ADAMserve).
Ron writes:
>Backward compatibility may be a problem for programmers, but >programmers need to keep in mind that some users cling >tenaciously to what they're familiar with unless there is a grand >and handsome payoff. Perhaps the payoff has not been adequately >sold.
Believe me, as a would-be operating systems programmer, I understand that users like to keep the applications they already have, and are using very productively. But it's extremely frustrating to me to find programs that, for no "good" reason (i.e., no payoff in speed or ease of use or novel use of system resources), do "bad" things like bypass the operating system, write directly to the hardware, etc.
But Ron also asks:
>Why would a program written in the early and mid-eighties do >anything other than assume that it had the whole computer to >itself? Was that not in fact true?
In the short-sighted short term, yes. However, even in the limited EOS operating system, there was some thought to what might come in future. The _READ_BLOCK and _WRITE_BLOCK functions take a 4-byte block number in the BC and DE registers; with 1K blocks, this is a maximum of 4 *gigabytes* of storage per block device. ADAMnet itself is a plug-and-play peripheral device network which modern Wintel systems have yet to achieve. And even though no communications protocol was ever defined, an ADAMnet device number (15) was reserved for a Gateway device which would have allowed ADAM-to-ADAM networking.
Some of these "hooks" for future expansion can be exploited today. Under ADAMserve, a 4GB ADAMnet hard drive could be created, or access provided to standard 650MB CD-ROMs--all because of support for "ridiculously high" block numbers in EOS-5. (Consider that tapes were 256K and original Coleco disk drives were 160K. Block 256 is B=1, C=0, D=0, E=0; a rather large "waste"!)
In the long term, however, the sloppy thinking that "the machine is *mine* and I can do what I want" is limiting. Part of the task of paving the way for future expansion is identifying all the "badly-behaved" programs and fixing them to be more considerate, when possible.
>When I >hear this talk about so-called 'badly behaved programs' I have to >wonder 'badly behaved' for whom?
They are badly-behaved for those who would try to provide additional features in operating systems while maintaining the user-demanded "backward compatibility".
>Speedywrite has often been cited as a fine example of a 'badly >behaved program' in the ADAM world. The fact that it provided >superior wordprocessing capability to anything then available for >EOS seemed to be completely irrelevant. Speedywrite provides it's >own methods of dealing with the computer's resources and by doing >so provided an impressive example - in my estimation - of what a >creative imagination can do with limited computing resources if >left unfettered by conventions and rules.
Yes--from the end user's perspective. And as long as that end user is content to use SpeedyWrite in its original environment--taking over the entire machine, pull the reset switch when you're done--he has nothing to complain about. If he wants SpeedyWrite to run from a command line shell under an enhanced EOS which lives in XRAM, however, or can read files from a CD-ROM accessible by an ADAMserve connection, or can use an 80-column terminal, then he is (probably) out of luck, unless someone is willing to take SpeedyWrite to pieces and reassemble it in a "well-behaved" mode.
>In my humble >estimation, a program of Speedywrite's quality, under any >operating system, would be difficult replace.
And *that's* why, if there are too many "badly-behaved" legacy programs that end users want to keep using in their original form, then there is no point to trying to forge ahead with anything new and wonderful at the operating system level, because ultimately the result will be a nifty operating system which nobody will use but its author. I did that once with SmartBASIC 1.x; I don't wish to repeat the exercise.
Even Ron admits, after a laundry list of EOS's many shortcomings,
>it's probably fairly safe to say that >there's plenty wrong with EOS about which plenty could be done. >Whether or not it is worth doing except as an academic exercise >remains debatable.
If the important applications that everybody wants to run had simply obeyed the injunction of the EOS programmers at the start of the EOS-6 code listing, it would have been soooo much easier to make enhancements to EOS:
This absolute listing was generated to ease software
development on the ADAM. This listing provides the location
of both released and unreleased entry points. Released
entry points begin immediately in this file with the jump
table and end before the first code segment listed.
Released entry points include the jump table, common data
areas (EOS_COMN), common data tables, and equates which
describe the released data structures. Direct access
to code segments is STRONGLY DISCOURAGED and may make
your application incompatible with some ADAMs. There is
more than one version of EOS on the market at this time
and updates are planned.
Much the same message is found at the beginning of the BIOS source code listing for the original IBM-PC (1983):
THE BIOS ROUTINES ARE MEANT TO BE ACCESSED THROUGH
SOFTWARE INTERRUPTS ONLY. ANY ADDRESSES PRESENT IN
THE LISTINGS ARE INCLUDED ONLY FOR COMPLETENESS,
NOT FOR REFERENCE. APPLICATIONS WHICH REFERENCE
ABSOLUTE ADDRESSES WITHIN THE CODE SEGMENT
VIOLATE THE STRUCTURE AND DESIGN OF BIOS.
And dealing with all the early MS-DOS applications which did *not* heed this injunction continues to cripple Wintel operating systems development.
In closing, Ron has an excellent idea, which it's a pity that nobody had about 10 years ago:
>But perhaps we need something more. Perhaps we need not just a >standard, or specification covering EOS and how it will work. >Perhaps we also need a standard for programmers proposing to >develop new software. (I'm assuming there will be some; perhaps >that's a leap of faith). At the very least we need some sort of >common understanding as to how ADAM's resources are to be >accessed.
Those few of us who are left as programmers and stewards of the programs of others pretty much know what the standards should have been, and what the crippling no-nos are. Short of a massive effort to reverse-engineer the top five EOS applications to complete, reassemblable source code (PowerPaint, ADAMcalc, SpeedyWrite, SmartWriter, and ADAMlink V--I omit SmartBASIC and File Manager because I've already done them), there is no common base of "well- behaved" applications. *All* of the above 5 do bad things, or have bad internal organizations which make further (desired) expansion impossible. The ADAMserve version of File Manager (4.0, derived from a disassembly of 3.1) is now totally EOS-compliant; it does not bypass EOS to access hardware in any way at all. Its usage of XRAM buffers could be make RAMdisk-friendly without too much trouble, now that I have regenerated the source.
I can invent standards and promulgate/publish them through my TWWMCA articles; but unless part of my audience is ready to start writing some new applications (or willing to help reverse-engineer some existing ones), it is really just vanity press...
On this note, Ron ends:
>'Nuff said. I do hope this discussion continues. I think it's >marvellous that Dr. Drushel has initiated some thinking about >these things. I hope we can all follow along, contribute to the >best of our ability, and learn in the process.
As I said at ADAMcon 08, I want to use my ADAM time and talents on a project which the remaining ADAM community deems to be of sufficient importance that it will actually *use it* if the project is completed, not simply ooh and ahh over it. Attaboys are *not* what I'm looking for. Frankly, I get a lot of intellectual pleasure in taking apart the sourceless binaries to see how they work, more so than trying to write user-friendly GUIs in Z80 asssembler :-)
See you next week!
*Rich*
Next Article
Previous Article
TWWMCA Archive Main Page