(C) 2003 Hank Wallace

After you digest this important guidance, see a following case analysis of a busted FBI/SAIC software project that cost the taxpayers 170 million dollars! If only they had read these unspoken secrets…

The sales manager walked into my office with a red face and steam coming out of his ears. “Those programmers! I’ve got the phones ringing of the hook for product and all they do is put me off until next week. When are they ever going to finish that software?” Being somewhat of a software-jock myself, I had mixed feelings and wanted to defend their position. But on the other hand, don’t they realize that the company is depending on them?

How many situations like this have you been in? Unfortunately, it is all too common these days as computers seem to multiply like rabbits, bringing more and more “benefits” to our lives. It’s just another case of those cursed blessings.

It is difficult to understand the world of programming if you are not a programmer, regardless of the level of talent required. To the uninitiated, it can be frustrating, especially when talking with a programmer about the status of a project and getting no concrete commitments. Fortunately it is not necessary to learn to program computers in order to understand something about programming any more than insanity is required to glimpse the mind of the psychopath, though some programmers would say that computer programming and insanity are closely related.

This article is for those who work with those who program computers.

Why software?

Why are we so determined to write software, anyway? It seems that every widget produced these days has a computer buried inside, even in a simple thing like a wristwatch. The driving force is cost, of course, the notion being that creating and changing software is less expensive than the same with hardware, and this is usually true. Many minor changes are made to a product after it is in production, and it is easier to integrate those changes by fiddling with the software rather than the hardware. The hardware can sometimes remain the same while software modifications make radical changes in form, fit, and function. Witness the PC.

Let’s look at some things that yield a little clearer view of programs and programmers, things which are not well understood even by software practitioners.

Why are software estimates hardly ever correct?

Estimating labor is a difficult task in any business. It requires extensive knowledge of the operations to be performed and the desired outcome. You can’t estimate how long it will take you to get to an unknown destination. Most of this knowledge comes from experience. For example, if I record how long it takes a clerk to enter 1000 orders, I can figure the average time per entry is the total time divided by 1000. So, if I have a goal of entering 100 orders per hour, I can calculate the number of clerks needed, assuming equivalent clerical capabilities. Sounds simple, but when you consider that every person works and learns at a different pace, it proves difficult to estimate exactly how many clerks are needed without waste and without bottlenecks.

The same uncertainty occurs when estimating a software project, only worse. The principle of leverage comes into play as I will illustrate. Say, you commission your software services group to write an application which allows order entry, work in process monitoring, and posting to the sales ledger. They estimate that it will take two months to write each of the three pieces of the program, based on their past experience; six months total. Note that information from the order entry section of the program feeds the other two, as in a paper based system.

The leverage comes into play when there are errors in the estimate. Let’s follow the progress of the project. The first piece of the program, order entry, is designed on schedule in two months. The second piece is also designed on schedule. However, the third piece, sales ledger posting, runs 30% over due to some bugs in the existing accounting package, which incidentally requires more information from the order entry section than was initially planned. This means that the order entry section must be redesigned, and these design changes ripple down to the work in process section, lengthening the development time of all three sections. So, a 30% overrun on one section has blossomed into a huge overrun on the whole project, otherwise known as a disaster!

This sort of scenario is extremely difficult to avoid, especially in today’s world of ‘integrated’ programs which are supposed to work together smoothly, but don’t. When estimating a programming project, it is impossible to forecast all the problems which may leverage themselves into huge schedule overruns by rippling down through previously written programs, or complicating the design of unwritten software. If you get a big program written in less than 130% of the labor or time estimate, it’s usually time for celebration. Most jobs that hit the time target invariably eject burned out husks of programmers who spent quite a few 80-hour weeks meeting the deadline.

I know one programmer who examines each aspect of the job in great detail, tallying time estimates for the component software modules, finally arriving at a total for the entire project. He then multiplies that precise figure by a slop factor of 3 which he has found yields true estimates of the actual time required for him to do the job, based on historical comparisons. His estimates are accurate and reasonable, though not rigorously derived.

Why does it take so long to write software?

Estimates aside, why does it take so long to write a seemingly simple program? It is usually easy to summarize the operation of a program in a couple sentences. For example: The computer must turn the heat on when the house gets cold, and turn the air conditioning on when the house gets hot. Piece of cake, right?

If you were to write the program just like that, you would find that about every ten minutes the heater would come on and the air conditioner would turn off, and vice-versa, because the air conditioner would prod the heater to action, and the heater would stimulate the air conditioner. “Well, that’s silly,” you say, “of course you don’t have them both active at the same time!” You and I know that, and it is assumed in the operational summary, but the computer does not know that. A better summary would be: This program must turn the heat on when the house gets cold in the winter, and turn the air conditioning on when the house gets hot in the summer, but never both at the same time. We might even add that the heating and cooling temperatures must be set at least five degrees apart, and short term changes (as from a door draft) should be ignored.

This is the point: It is not usually the specification which makes a program difficult to write, but rather the list of things that the program should NOT do, which is only implied in the specification.

Another reason for lengthy development cycles is that software is like any other craft. It takes longer to produce a high quality product than a low quality one. Note that I am not talking about mass-produced things like cars and soda cans. Software is not an assembly line item, but rather is crafted by an individual or group for a specific task. Compare the programmer to a cabinet maker. If you give a cabinet maker one day to build a bookshelf, you will get a plain rectangular box with shelves and one coat of varnish, still tacky. But give the same craftsman a month and you will get hand detailed inlay with sculptured edges and a rich, deep finish. Of course, you will also get a larger bill!

You need to picture the schedule for your software project as something that already exists in a parallel universe, but you don’t have access to it. You can, however, get a good idea of the schedule from the scope of the project and your past experience on similar projects. But be advised that scheduling tools are NOT connected to that parallel universe, so any adjustments you make to your Gantt chart will not be reflected in reality! Read that last sentence again. Your experience helps you predict the schedule, and your programmer’s growing experience helps to reduce the schedule, but otherwise it’s as fixed as the law of gravity. Picture it. Live it. Love it.

There is no such thing as rush, high quality software. It takes time to do a good job. Sure you can sometimes throw more people at a project and get it done faster, but chances are you won’t improve the quality of the result.

Why does it take so long to debug software?

Programmers make mistakes, called bugs in software parlance. Some of those mistakes are highly detrimental to the program, and some less so. I suppose quite a few undetected program bugs accompany our astronauts around the earth on every shuttle mission, but fortunately without serious negative consequences so far. There were documented bugs in the computer programs that landed our Astronauts on the moon, too.

A program must be tested to detect and correct the errors introduced when a program is written. Some of the bugs are simple, like typos. But some are deeper, like mistaken assumptions or design errors which mean that the very structure of the program is unsound. So, the debugging process has a twofold purpose. First, it uncovers inconsistencies between the program and the crucial design specification from which it was written. Second, it uncovers inadequacy in the design specification itself, a difficult task.

It is tough to test all the requirements of the design specification to be sure the program performs them well, and moreso for the implied things the program should not do, as explained above. The most thorough test checks each line of the program to be sure it performs its intended function. It may seem like you can just step through the program line by line, testing as you go, but it is not that easy because control of the computer in a program does not flow smoothly from beginning to end, as you read a book. Rather, control can jump from one part of a program to another so that certain parts of a program are used very infrequently, like the leap year exceptions in clock/calendar programs which are active only once every four years. Each line must be tested by simulating all possible conditions (such as leap years), and this is a grueling job.

It takes a long time to debug software because, for every hundred lines in a program, there are possibly thousands of program execution paths and combinations of activities that can occur. It is a problem of volume.

On one occasion, a small program I wrote (about 2000 lines) was ready for test and I advised my supervisor, a marketing and sales veteran. He asked about my debug strategy and I told him that I would test the program a little bit at a time, simulating the conditions under which the program would operate to isolate the bugs and verify the design. He asked why I did not just plug the program in and turn the product on. I explained that the chances of it working to any extent were practically zero, and he just looked at me with a puzzled expression. It took about two months to debug the program.

When is the software finished?

I have been involved in many software projects, big and small. In some cases, deadlines passed and it was necessary to ship the product before all the major bugs had been worked out of the software. In each of those cases, it cost the company much, much more to finish the software in the field than in the lab. MUCH MORE. More in travel, lodging, lost time due to lack of software development tools which could not be brought to the site, and in general a poor environment for debugging software. And boy did the managers realize this after the fact, but they could not have been convinced of this beforehand.

When a program is written and tested, most of the serious problems are found quickly, with the more stubborn and less obvious bugs yielding later. Expressed graphically, it looks like this:

As an example of an unfinished program, take a look at this screen shot from the FoxNews web site. See anything unusual? All the stock market numbers are shown as “NaN.00”. The text “NaN” means “not a number”, and is displayed when the computer language (likely Java or Javascript) cannot display an improperly formatted floating point number. This program was not tested with invalid input, and consequently displays this garbage which is meaningless to most readers, except for computer programmers who have made this same mistake! This program runs every day on millions of computers, but is unfinished.

Is programming such a specialized task?

I have been approached by managers who implied that anyone who knows how to use a word processor could learn to program. Perhaps. But sometimes I wish that programming were done with goofy symbols that only programmers understood so that the poor analogy of programs and prose would die.

Here is the difference. Programs are interpreted by machines. Prose is interpreted by humans. Humans have had years of education, and more importantly, experience with which to gauge the context of what they read. Every time you turn a computer on, it is as naked as the day it rolled off the assembly line. Imagine waking every day and having to be taught how to walk and talk all over again. This is what happens to a computer when it performs the process called booting just after you turn it on. Sure, there are some programs that learn from experience, but they are in the minority still. A computer is not an intelligent entity.

For example, if I took this article and swapped any two paragraphs, you could still read it, and though it would seem a bit disjoint, you could still make sense of it and even suggest that I fix the error. On the other hand, I can find two letters in any program that, when swapped, will totally disable it. The point is that most computers are not adaptive and have no cognitive skill at all. This puts a tremendous burden on the programmer for exacting detail to ensure that the program is fit and robust. This detail requirement makes programming a specialized task.

Is programming such a complex task?

Sometimes one of my clients will ask me about a program I wrote a month or two ago. I then proceed to query the client to refresh my memory as to what I wrote so that I may answer the question. I get some funny responses, but I just can’t remember all that detail forever. Even when writing a program, I sometimes forget how a previously written piece works, and I must study it again. This illustrates the next point: A program can get too complex for one person to comprehend all at one time.

There is a methodology that most programmers use to construct complex programs. It is called top-down design and works like this. The main objective of the program is stated. It is then decomposed into two or more sub-objectives. Each is likewise split into more manageable pieces until they are small enough to understand and program easily. This is the familiar divide-and-conquer approach. The programmer can easily understand each small piece of the program, but it is sometimes difficult to remember all the interactions after they are designed, written, and sewn together. Thus, a programmer must usually go back and re-learn how major portions of a program work before being able to find bugs or do modifications.

When such complexity occurs in business, it is customary to hire more employees to handle the workload. However, the programmer usually does not have that luxury, making some of them seem quite unknowledgeable even about their own programs. Yes, programming is a complex task.

What can you do to help create quality software?

Every program should be specified by a written document which is studied and evaluated by both management and the programming staff. It makes sense to invest some thought in such a costly process as the creation of software. I have seen a verbally specified project run 400% over schedule because of communication problems and misunderstandings. There is no question as to whether a written specification is desirable.

Sometimes the initial specification for a program is conservative, encompassing only basic features. The intent is to invest in the software development task slowly, expanding the software as experience demands. This is a noble goal. However, it presents some problems. If the initial specification does not have a vision toward the final system, the resulting software may not have the capability of being expanded easily, greatly increasing project costs and delays. It is as if the initial spec details an outhouse, but future modifications add a victorian parlor, eat-in kitchen, five bedrooms, and a jacuzzi. No matter what the changes, the central feature is still an outhouse, lowering the value of the whole. The solution is to specify as much of the system as possible, even loosely, and also not to hide any information from the programmers.

So you have little time to create such a specification? I assure you that writing down your wishes will help clarify them for both yourself and the programming staff, and help you get just what you want, no more, no less. You will not get a high quality product from the programming staff unless you make your desires perfectly clear in the software specification.

And remember, one small omission from a specification makes it a speciFICTION, so be thorough!

After all that, why can’t we get it right the first time?

Everyone dislikes nagging software bugs, but the answer to this question really has nothing special to do with software. When you learned to walk, did you hop up out of a crawl and take off on a stroll? Of course not. Were your parents peeved because it took you hundreds of tries before you could walk ten feet? No.

It’s like this: Anything worth doing requires iteration. To appreciate this point, try typing a short letter without using the backspace key on your computer or typewriter. That applies to programming, too. It sometimes takes hundreds of iterations over the course of weeks or months to put a program into usable form. And programmers loathe this fact most of all. There is nothing more demoralizing than to see your own program, your baby, freak out and spit garbage all over the computer screen.

2011 Update: Programmer Quality

I have noticed a trend over the last two decades that affects software quality and project cost. The trend is that software development systems and languages are built one upon another, layer upon layer, advancing in function. No one codes in C any more, especially regarding web oriented systems. They use Java, or a scripting language, or .NET. These are more meta languages floating on top of other systems. The Java language is useless without the huge built in object library. Dot net is a throwback to the 1970’s interpreted languages, all to avoid security holes created by Microsoft’s own geeks. The house of cards is getting taller, and it is possible for someone with virtually zero experience to write a web app in an hour. Whee!!!

The down side is that programmers have less capability as time goes on, simply because the programming languages include advanced features that do not have to be written. I see this clearly when working on systems that span the vertical space from hardware and low level drivers to high level applications. Younger programmers have no mental model of the machine that the code is running on. At one time, every programmer had to take some courses in assembly language programming where a mental model of the processor was required. Today, the processor model is an abstract virtual machine that has no relationship to the real world.

Thus, when programmers have to interact with the real world, they are at a loss. To understand this phenomenon, ask any programmer under 30 to convert decimal 1234 to hexadecimal with a handheld calculator. “Uhhhh… Hexawhat?”

This lack of concrete knowledge handicaps young programmers to the point that they are only useful when programming in a language for which they have been trained. Older programmers were taught programming, period, regardless of the platform or processor.

To get the most out of your software development process, be sure that each of your programmers has low and high level skills, including assembly language programming abilities. Be sure he/she can analyze an algorithm for performance, before it runs on the computer. Quiz them on basic mathematics and algorithms. The more general thinking your programmers are, the greater success you project will have.

Conclusion

It is difficult to understand computer programming if you are not a programmer because it combines a wide range of activities from bookkeeping-like chores to the design of large, complex data structures and algorithms, most of which are abstract and have no analog in the real world.

In summary…

• The purpose of software is to allow a piece of hardware (the computer) to have a more easily designed, versatile, changeable, and less expensive function.
• A software specification is mandatory for a quality software product. And YOU must participate to foster this goal.
• It is difficult estimating the magnitude of a software project in part because of the leverage effect where small problems snowball into schedule overruns. The abstract nature of the programming task also contributes to the difficulty.
• It takes more time to write high quality software than low quality software.
• Conditions unstated in the software specification (the DON’Ts) must be included in the computer program, sometimes making an aparently simple program quite complex.
• Programmers and software specification designers make mistakes (bugs), and it takes time to find and correct them.
• The complexity of a program can grow much faster than the simple size of the program (number of lines). This complicates the task of debugging and maintaining software.
• Programming is a somewhat specialized task because computers are specialized, dumb machines. When computers gain a level of human-like intelligence, the programming task may not be so specialized.
• A program is finished when your testing confirms that it works according to specification, independent of product shipment schedules.
• Anything worth doing requires that we try, evaluate our errors, and retry.

 An Unfortunate but True Illustration from Real Life

A recent article on Fox News (“Scrapped Computer Program Leaves FBI Wanting,” March 26, 2005, by Kelley Beaucar Vlahos) exposed the folly of the FBI and Science Applications International Corp. as they collaborated to funnel 170 MILLION dollars of taxpayer money down the rat hole of software development. This project is the antichrist project poster child illustrating every sin outlined in “The Unspoken Secrets of Software Development.” Thanks to the government and a bloated government contractor for giving me such a crystaline example of how to screw up a project.

This project is the FBI’s Virtual Case File, a nationwide application to allow sharing of case files and other information, in an effort to reduce investigation delays and help connect seemingly unrelated dots in the pursuit of criminals and terrorists. It’s a database that helps the FBI do its job. So far so good.

Being a huge undertaking, the project was four years in the making, though not complete. Now the FBI has admitted to Congress that the entire project is a debacle and the money wasted. Let’s look at the smouldering remains and see what we can learn. The italicized quotes are taken from the Fox News article.

“Our ability to handle a project like that was not what I thought it was,” Mueller told the House Appropriations Committee on March 8. “It’s my fault for not having put the appropriate persons in position to review that contract and assure it was on track.”

Can’t you picture the kickoff meeting at the FBI?

“Thanks for attending our Virtual Case File kickoff meeting here at the FBI. We’re very excited about this project as it will allow us to put all the bad guys in the slammer. Let me introduce our main players who will manage the project.”

“First, Bob Blunderbuss here has managed contracts for office supplies and toiletries for several years, and has taken a course in Visual Basic at the community college. He’ll be our point man for all system software requirements and contractual issues.”

“Next, Sally Hasanmba will be supervising the subcontractors we select. She has used computer software for a while now at here desk, and has actually seen a picture of a real case file, standard manilla, so is uniquely qualified to guide our subs regarding visual artwork for the splash screen on the Virtual Case File system.”

“Finally, Jason Thicklens has experience integrating commercial software with our existing 1850’s Cobol applications, and so he will be working to integrate this new state of the art package with our existing 1850’s Cobol applications.”

This is not too far from the truth, as I have been to several of these types of kickoff meetings. Reread the unspoken secret Is programming such a specialized task? to understand that YES, it is specialized, and YES you need the right people in the right places to get the job done right.

To manage a project, you need not only management experience, but real world experience doing work similar to that your subcontractors will be doing. If not, how will you know what questions to ask? How will you know you are getting a raw deal before you have spent $170 million dollars?

However, hats are off to the FBI for at least stepping up to the plate and admitting their error. What does the subcontractor say?

In its defense, SAIC contractors told Congress the project suffered from constantly shifting mission changes, which SAIC attempted to keep up with, as well as an aggressive timetable to get the final phase operating. It was further hindered by a lack of consistency in management oversight within the bureau, a fact the FBI fully acknowledged hobbled SAIC’s efforts.

There are several sins outlined here. First, “constantly shifting mission changes.” Reread Why are software estimates hardly ever correct? These people likely had a specification that would break your foot if dropped from waist height, but did it mean anything? Apparently not. Once it was written up and signed off, the FBI kept changing it. Perhaps even SAIC requested changes.

This all indicates that the FBI did not know what they wanted in the first place!

You could not dream up a more sure recipe for a money wasting project. Reread What can you do to help create quality software?

The next sin is the “aggressive timetable.” The people who are administering the project sometimes have little understanding of the work that must be done, especially with government contracts. The whole ball of wax is represented to them as a bunch of bars on a chart created with Microsoft Project. Not meeting the deadline? Just slide those little bars to the left and we’re back on schedule! Who cares that the work cannot be done in that time frame. Why ask the software jocks if this is possible? They are always angling for more time anyway.

Then we have a “lack of consistency in management oversight.” That’s a real shocker (not). How can people who don’t understand software manage a software project? With all deference to Mr. Blunderbuss at the Bureau, he probably is managing several failing projects at once!

Not only that, but (sin number four) the FBI acknowledged that it “hobbled SAIC’s efforts” through this lack of consistency. Why is there no consistency? Could it be related to a lack of vision, or goals, or a well-defined problem needing a well-defined solution? Could it be the FBI had several different committees or departments driving SAIC in ten different directions? And SAIC did not have the backbone to put a stop to this mess?

The sad fact is, what SAIC says is likely all true. And it is likely they knew this from the outset, and so did the FBI. You see, the FBI and SAIC have probably worked together on several projects, perhaps dozens, each suffering from the same mismanagement, but with smaller budgets, and they did not learn from their mistakes. Reread After all that, why can’t we get it right the first time?

Another source quoted in the Fox News article said

“Unfortunately, if you go straight from design to implementation, where then is the logical point where you say this isn’t really going to solve our problem?” he asked. “I think they put the cart before the horse.”

Yep. The FBI has no concept of software deployment, no concept of iteration toward a goal, no hint that there may be problems, that they should start small and scale up. Every commercial client I work for embraces this concept, but they would not if their funds were unlimited, as it appears the Federal Government’s are.

I don’t know about you, but I want a refund. I think that the government employee pension fund should be able to spare $170 mil, probably without noticing.

Please reread “The Unspoken Secrets of Software Development.” It’s brief and to the point. Whether your next project has a budget of $170 million or $170, you will benefit from that experience and a close examination of the FBI’s disaster project. Each of their sins can be repeated on a smaller scale by you in your next project. You are no more immune than they.

Author Biography

Hank Wallace is the owner of Atlantic Quality Design, Inc., a consulting firm located in Fincastle, Virginia. He has experience in many areas of embedded software and hardware development, and system design. See www.aqdi.com for more information.