A look at the software development lifecycle we all love to hate.

Transcript for episode 7 of the Test & Code Podcast

This transcript starts as an auto generated transcript.
PRs welcome if you want to help fix any errors.

“The waterfall model is a sequential development approach, in which development is seen flowing steadily downwards (like a waterfall) through several phases, typically: requirements analysis resulting in a software requirements specification, software design, implementation, testing, integration (if there are multiple subsystems), deployment (or installation), and maintenance.” - wikipedia

And that’s pretty much what people are talking about when referring to the waterfall model. However, that’s only the start of the story.

Do we really need all of those steps?

There are two steps common to all software developed: analysis, and coding. For all software developed, someone has to figure out what the problem is, and the code has to be written. Pretty hard to argue with that. I didn’t come up with that. That’s all Dr. Winston W. Royce there.

Hang on now, wasn’t Royce the dude that got us using waterfall in the first place? Yes. And no. It’s complicated. So hold that thought and we’ll get back to it.

The waterfall model has been used and modified and changed and rebelled against since before I started programming. And it seems to get referred to all the time when someone is promoting their favorite methodology. Although sometimes it goes by the name “classical development”.

Waterfall such an important character in the story of software development that we should get to know it a better. Do a little character development.

To look at waterfall in more depth:

  1. Let’s start with a few waterfall variants that I’ve come across.
  2. A happy path waterfall scenario as a reasonable process flow.
  3. The unhappy path. What are some problems with the waterfall model?
  4. What about OO? How does OOD and OOP fit in with waterfall?
  5. The agile movement as a rebellion against waterfall-like models.
  6. Waterwheel
  7. A fair look at the document that’s blamed for starting all of this, Dr. Royce’s “Managing the Development of Large Software Systems”.
  8. Comparing Royce’s final model to modern iterative models.

Waterfall Variants

Waterfall doesn’t always have the same steps. I’ve seen several versions of basically the same thing with slight variations.

I’m not going to include pictures of these because they are really boring pictures, and frankly, people looking at pictures instead of reading and understanding is what got us into this mess in the first place.

Here’s one:

  1. Requirements
  2. Analysis
  3. Design
  4. Development
  5. QA (Quality Assurance)

And another:

  1. Requirements
  2. Design
  3. Implementation
  4. Verification
  5. Maintenance

And here’s one unceremoniously known as Royce, Figure 2:

  1. System Requirements
  2. Software Requirements
  3. Analysis
  4. Program Design
  5. Coding
  6. Testing
  7. Operations

It’s the second drawing in the document that I’m going to talk about in a little bit. That’s why it’s “Figure 2”.

Here’s one often drawn in a V shape and called the V Model:

  1. Concept of Operation
  2. Requirements and Architecture
  3. Detailed Design
  4. Implementation
  5. Integration, Test, and Verification
  6. System Verification and Validation
  7. Operation and Maintenance

Why is it a V? If you put step 4, implementation, in the middle at the bottom of the V, you can line up the other 6 steps left and right. And it indicates what testing and post-development stages line up with which pre-development stages.

And it makes people feel better that it’s in a V shape bacuase it looks cool and official like. However, steps still progress with hardly any looping. So, whatever shape you draw it in, it’s waterfall.

A happy path waterfall scenario as a reasonable process flow.

Let’s try to give this a fair shake. Here’s the idea. Ad hoc software development is chaotic. And waterfall models try to bring order to the chaos.

Do they succeed? Not really. But they do produce a lot of documentation.

However, when you talk it through, it seems reasonable.

First, what does the customer need the software to do? That’s the collecting requirements phase.

Next, you can take those requirements and hand them to someone that knows what software is capable of doing and they can generate some functional requirements. That’s analysis.

I think. I’m actually not really sure of the exact border crossings between analysis, architecture, and design. But whatever, let’s go with it.

Now the analist hands off this list of requirements to an architect or designer or both or whatever and they rough out what all of this work is going to look like. They come up with a breakdown of all of the different parts of the software that need implemented. Then they come up with a design that will satisfy the requirements. These parts are architecture and design. Again, I’m not super clear on where the dividing line is.

And somewhere in the analysis and design phases you can come up with really how many coders and testers you need, and document writeres and whatever. This is probably done partly with managers and architects or designers or leads bartering and bluffing and playing schedule chicken and such.

Sounds messy. Like making sausage. But before you get coding too much you need to figure out how many people you need and how long it’s gonna take to finish.

And you need a design so the coders know what to write, the QA folks know what to test against, and the learning products people to write how-to guides, users manuals, and such.

And so then you hand off these designs to developers who code away merrily and eventually come up with a completed system. That’s coding (or development, or implementation, or the magic black box on product lifecycle charts between design and testing).

Next you get the quality assurance department involved to test the thing against the requirements and against the design. Maybe they get started writing test plans and such. But usually no actual testing is done until the software is mostly done.

Now, perhaps the learning products people got started early with the guides and such. But again, probably not. And even if they did, odds are they need changed. So that often comes after the software is mostly done. Maybe in parallel with QA.

Now you hand over your lovely bug free software to the customer. And the manuals. Don’t forget those. They are heavy and help the customer feel like they got their moneys worth. BTW, the more they spend, the more documentation you should write.

The customer is absolutely thrilled with your wonderful software that’s delivered on time, on budget, and is exactly what they need.

And if they are confused by anything (which they won’t be), they can just look something up in the manual because it’s easy to read and perfectly matches the behavior of the software.

Except that doesn’t really happen.

The unhappy path. What are some problems with the waterfall model?

See, this waterfall model is designed to go in order. That’s lovely, we like order.

But…. requirements are ambiguous, sometimes contradictory, and tend to change in the middle of the project..

And the problem might not actually have been analyzed correctly. The software might be solving the wrong problem. And the design might be ambiguous, contradictory, incomplete, and might not actually satisfy the requirements. And there might be coding errors. Correction. There will be coding errors. Actually, everyone involved in this process is human. So there will be errors in every step of this process.

That’s why we have testing. The testing will hopefully show what’s wrong, and allow us to fix it.

But the testing is at the end. Maybe months after the project started. And if testing shows that we really can’t satisfy all the requirements, we might have to go all the way back to the beginning, causing at least a 100% overrun in time and money.

Well. That sucks.

Oh yeah. I forgot to mention. Most of these phases produce reams and reams of documentation that most people involved never read, some people read only once, and really no one understands.

Some software projects produce more documentation than code.

One of the problems with this that some people noticed is that it was hard to map real world problems to software very well.

What about OO? How does OOA/OOD/OOP fit in with waterfall?

Some folks, I really don’t know who, noticed this mismatch and came up with object oriented analysis, object oriented design, and object oriented programming. OOA, OOD, and OOP. They look like acronyms, but they are silly to try to pronounce.

This is actually where I enter the story. I learned object oriented thinking, analysis, design, modeling, and programming.

And object oriented everything totally made sense to me. Analyze the problem in terms of the actual problem domain. Then try to describe and design a software solution using nouns and verbs that came from the problem domain.

This actually helps quite a bit to be able to describe a design in terms that customers can understand and a translation process that makes sense to developers.

Model the world as entities, as objects. And give them traits, behaviors, and most importantly, responsibilities. And describe how the object interact with each other using verbs.

One good thing is that it’s really easy to spot dumb requirements. (“So, the bank teller simultaneously handles transactions with up to 50 customers at a time. …. No wait. That 50 came from the possible customers in the bank at once, not how many are at the teller station at a time. They queue up, and the teller only has to interact with one customer at a time… “. Stuff like that.)

Surely with these OO tools we are better at finishing software on time that is useful. It’s gotta be better.

So, let’s splice this OO stuff into our software development lifecycle somewere and off we go..

Except, there were problems. It wasn’t fun. Teams would spend months in long meetings drawing pictures on white boards, deciding on class hierarchies, and arguing about which classes had which responsibilies.

And user scenarios were drawn with bounce diagrams and alternate swim lanes and lots of time fighting with drawing tools to get the stupid arrows to line up right without looking too crazy.

It sucked also.

Was it better than the pre OO waterfall?

No. It was waterfall with more pictures and even more documentation.

And people were even more afraid to propose changes, becuase that would mean you had to draw the stupid drawings over again.

So in the end we really thought of the drawings as a starting point for development, and didn’t bother with minor updates. And eventually by the end of the project, they didn’t really match the software all that well.

Oh yeah. Forgot to mention that the simple design drawings that I learned in school were being developed by lots of people, and then they got standardized in UML.

And UML is now such a crazy nightmare that it’s harder to read than code. (Ok. An asterisk here and a 1 over there means one to many. What’s about that other arrow over there with no numbers on it? And what’s the dashed arrow mean again?…)

Well, if OO isn’t the answer, then what is?

The agile movement as a rebellion against waterfall-like models.

Enter super Agile with a capital A, a cape, tights, and a pearly white smile.

A bunch of people were coming up with alternatives to waterfall. There’s lots of stuff people didn’t like, and they were coming up with quite a few iterative processes to counter the heavyweight in the corner, Mr. Evil Waterfall.

These newer lightweight, iterative development models were all coming up with some good ideas. But lots of companies weren’t taking them seriously. I think the “lightweight” moniker made some people freaked out and thought it was just designers and developers whining about the work. Suck it up, people. We’re professionals and we write software professionally. And people pay a lot of money to have us be able to fulfill our promises and deliver stuff on time and within budget.

And we need to be able to track and control things. If you don’t like it, go flip burgers or something. But stop whining.

Or maybe not.

But there was at least some indication that lightweight process weren’t being taken seriously.

But then that started changing. And more and more developers weren’t wanting to put up with the waterfall BS. And the world got faster. Software projects lasting 18 months or more and still being late and overbudget and being horrible to use just weren’t cutting it anymore.

And releases were expected to be faster.

Six month release cycles now seem really, really slow.

By the time the “Manifesto for Agile Software Development” showed up, Extreme Programming was getting looked at seriously.

And if XP did nothing else, it got almost everyone to pull automated testing into their development processes.

So, the agile movement did help. A lot.

I have quite a bit to say about many of the agile practices. But I’m not focusing on those right now. I’m focusing on waterfall.

Because it is still around.

Waterwheel

Waterfall isn’t part of agile practices though, is it?

Well….

A guy named Trek recently posted an article of what they called the waterwheel methodology

That’s when requirements are gathered. Then the requirements are dripped into the development process as user stories. The development team gets to feel good about their super cool agile development process. And then when they are done, they pass off the code to QA to check agains the requirements.

Hmmmm… requirements, then development, then QA. That’s still waterfall folks.

Anyway. Good job Trek for coming up with the waterwheel analogy. It’s damn funny.

A fair look at the document that's blamed for starting all of this, Dr. Royce's "Managing the Development of Large Software Systems".

So I think we should learn some lessons from all of this. And I think we should start by going back to the source.

Please, please, please, pretty please, with a cherry on top, keep with me to the end. I think it’s super interesting. And even if this is kinda dull to you, the entire world is paying the price of people in the 70’s and 80’s NOT actually reading the paper that started it all, and just looking at the picture.

So. Where did this come from?

There’s a paper by Dr. Winston W. Royce titled “Managing the Development of Large Software Systems” that appeared in Proceedings, IEEE WESCON in August 1970.

It is often cited as being the first description of the waterfall model.

I’ve ran across many references to this paper for many years, but just read it for myself recently. It’s easy to find. I encourage you to go read it as well. For such an influential paper, it’s pretty short. It’s only about 4 1/2 pages of text with a handful of pages worth of diagrams. It’s a quick read with some great insights into the problems with developing large software systems.

And even though Dr Royce doesn’t use the term waterfall in the paper, he seems to be the first person to describe the model.

Reading this paper with an open mind provides a wonderful starting point for the discussion of other methodologies.

So let’s get into it.

Royce states right off the bat that “there are only two essential steps common to all computer program developments, regardless of size or complexity. There is first an analysis step, followed second by a coding step…”. Remember that, from earlier. Yeah. That was this paper.

Further, this simplification of just analysis and coding are all that is needed for small projects operated by those who built it. That’s refreshingly honest. How many scripts and utilities have you written for you own use or your team? I know I’ve written a ton. Any tests for these? specification documents? requirements analysis? users guide? maintenance plan? No. Not usually any of those. Maybe a bit of planning when redesigning a build system or something. But usually pretty light on process. Because I’m the user. Or one of the main users. And the other users are in the same workspace. I’m testing it by using it. It doesn’t have to handle corner case input. It has to handle the stuff I throw at it and that’s it. And if it ever isn’t enough, I update it and expand it.

But this doesn’t work for large systems.

What’s different about larger systems?

One thing is that there are more people. And there is way more code than any one person can understand at one time. And large expensive systems aren’t usually written for the developers. They are written for someone else. So the developers aren’t the primary users of the software, if they use it at all. And there are specializations.

Some people are good at talking with customers. There are some people with domain experience that can think like a customer. And there are people good at designing large systems. And some people who are great developers, possibly some of them not great communicators. And there are new developers. In fact, probably a wide range of experience and expertise. And people mess up if we don’t write things down.

So the requirements get written down. And architecture, design, coding style guides, etc. has to be communicated and understood by a large group of people. Without clear communication of a shared vision for the software, a good understanding of how it’s going to fix the customer problems, etc. it’s tough to get a bunch of people moving in the same direction. And documentation is often leaned on for communication. And wiki’s and emails and whatnot. But chaos can happen easier with lots of people on a large software project.

And that’s what linear models like those described by Royce were trying to tackle. Bring order to the chaos.

Royce clearly didn’t come up with the waterfall model. He is describing it as what he has experienced as a current state of normal in 1970, and he’s pointing out problems with it and proposing solutions.

Analysis and coding. That isn’t enough for large systems.

Interestingly, Rice comments that the customer value still only comes from analysis and coding. The extra steps are there to try to ensure working software and improve predictability in cost and schedule. He states that the prime function of management is to sell the need for these extra steps to customers and developers. Customers don’t want to pay for that extra work, and developers don’t want to do all that extra work. But it’s necessary to control things and make sure you don’t end up in the weeds somewhere. I found this description of the main role of management amusing. And a little uncomfortable.

Now, let’s look at an example waterfall and how it would look in practice of an organization.

Here’s Royce’s Figure 2 model again:

  1. System Requirements
  2. Software Requirements
  3. Analysis
  4. Program Design
  5. Coding
  6. Testing
  7. Operations

We complain that one reason that waterfall models fail is because they are completely linear. But Royce observed that the steps are not without loops. He notes that the best projects regularly iterate between two adjacent steps. He also notes that many problems that cause larger regression in steps only have to go back at most 2 steps. For instance, test failures often have to go back to design, but not all the way back to requirements.

Analysis of problems with the waterfall model

Royce’s observations about the problems with this system are spot on.
First and foremost is the problem that testing is at the end.

Yep. I was blown away when I read that. Right there on the second page of the most common reference of waterfall is this: “… the implementation described … is risky and invites failures.” The testing phase which occurs at the end of the development cycle is the first event where we can tell if the software fits within it’s constraints and solves the problem it’s supposed to. Any problem found in testing involves doing some of the previous steps over with the possiblity of having to start everything over, causing a 100% overrun in schedule and cost.

You’ve got this massive risk of screwing up and not finding out that you’ve screwed up until the end. How do you reduce that risk? The rest of the paper gives Royce’s recommendations.

He’s got 5 recommendations.

  1. Preliminary design before analysis and coding.
  2. Current and complete documentation.
  3. Spend 1/4 to 1/3 of the time doing a pilot implementation.
  4. More testing, and more test related activities earlier.
  5. Involve the customer more.

Okay…. This really doesn’t sound that crazy. Why is this guy blamed for waterfall?

Let’s run through his recommendations in more detail.

Recommendations to reduce the risk inherent in testing at the end.

1. Preliminary Design

His first recommendation is to add a preliminary program design step after the requirements are mostly settled and solid. This isn’t to design all of the details, but a rough sketch of the final product.

This preliminary design will cover system constraints, functional expectations, and behavioral expectations. He also mentions designing the database and the processes needed. The early database design would inherently tell everyone later in the process the pieces of data that are considered important, how they are related to eachother, etc. The breakdown of processes is a rough split of functionality responsibility into different computing blocks. The preliminary design is also an opportunity to start all of the documentation for the system, including operating procedures.

Many of the constraint concerns in 1970 aren’t really too much of a concern anymore for many software projects. Royce’s constraints were things like computing resources, memory needs, processor time, etc. This seems weird when viewed from our perspective. But remember, in 1970, computing time was more expensive than people time, for the most part. The best analogy to modern computing is probably something like microcontroller embedded programming with real time constraints, or something like that.

However, many modern systems have other constraints that would benefit from thinking about earlier. Say a web service that needs to be able to handle some number of users making some number of requests per minute, with response times needing to be well under a second each in order for the system to not feel sluggish. Or consider mobile applications that run fine on newer mobile phones, but should also work on older, slower models without frustrating users.

Our systems still have constraints. But they aren’t always the same constraints Royce and others in 1970 had to deal with.

2. Documentation

The second recommendation Royce had was documenting the design. Royce recommended a freaking crazy amount of documentation, even to the point of mentioning a possible 1500 page product description that also needs to be easy to read. This is clearly nuts.

However, his intent was to try to uncover all of the hidden corners and not leave anything left implied. And this documentation included operating procedures, which implies user level behavior documentation.

Here are the documents in his final system diagram that he’s recommending:

  1. Requirements doc.
  2. Design (preliminary design that evolves into the final design.)
  3. Interface design. (that’s pretty cool)
  4. Test plan, developed during program design, and updated as necessary during testing. (way cool)
  5. Operating instructions, developed after testing. (hmmm. maybe too late)

Really, the reason I’m not including the diagram is that I want you to go at least skim this paper. Skip to the end if you only want to see the picture I’m talking about.

Much of the goal of all of this documentation is to control downstream developers and make sure they don’t muck things up because they didn’t have clear instructions.

However, massive documentation backfires. It is possible to write 1500 pages of design. But very few people will really read it and understand it. Also, documentation is a roadblock to redesigning, refactoring, and allowing flexibility in requirements and feature sets. They discourage change. This is the opposite of agility. Who wants to propose a change if it also includes updating all of that documetaion.

Well. Changes happen anyway, and usually the documentation just doesn’t get updated. But then QA departments don’t really know what to test against.

And what if a customer changes the requirements.

In fact, changes are so painful, that many large projects have change request procedures built into the system and the contracts.

This is to discourage customers from changing requirements by making it difficult and expensive to do so.

3. Pilot implementation

The third recommendation from Royce is to just go ahead and plan to do all of the steps twice. Do two complete iterations through the entire process.

However, for the first time through, you’re not going to do everything. The pilot implementation will be planned for about 1/4 or 1/3 of the entire time for the project, and will go through mini versions of all of the steps.

The pilot preliminary design will go ahead and identify the critical must have features, the MVP perhaps, or critical path features (the hard stuff). The rest of the pilot steps can then build the whole system as a skeleton system.

The idea of a pilot implementation shows up in different forms in later methodologies. It shows up as tracer bullets in Pragmatic Programmer, except that you’re firing all the tracer bullets at once. It also probably was influential in iterative methodologies such as XP and Scrum.

A short pilot phase requires a different set of people to be involved than in normally paced development. “… a very special kind of broad competence is required on the part of the personnel involved. They must have an intuitive feel for analysis, coding, and program design. They must quickly sense the trouble spots in the design, model them, model their alternatives, forget the straightforward aspects of the design which aren’t worth studying at this early point, and finally arrive at an error free program.” That’s Royce again.

This idea of shorter iterations being better implemented with a small team of cross discipline intuitive people who can analyze, design, and code was mostly ignored for quite a long time. But it shows up again 20-30 years later with the agile development movement.

4. Testing

The forth recommendation from Royce is to beef up testing. If you do all of the testing related activities at the end, it doesn’t work. If your schedule is slipping, you’ll cut the testing phase short. And there’s this massive risk that you have to start over.

So what is Royce recommending for testing changes.

  1. Test plans are put in place during the program design phase before coding.
  2. The pilot iteration test phase will test critical parts of the system early.
  3. Visual inspection should be used for all code. We call this code reviews now.
  4. Pay attention to code coverage during testing.

Do much of this before the final QA phase. The QA phase should be the final checkout of the system, not the start of testing.

Interesting. I didn’t know these ideas were that old. Unless I’m really reading it wrong.

Variants of these recommendations show up in agile methods.

5. Customer involvement

The fifth recommendation is to involve the customer early and often. Royce describes this mostly as additional document sign off steps.
However, he does also mention bringing in customer feedback for functionality and cost trade-offs. This is good.

In my experience, even with waterfall-like or scrum-like projects, I’ve never had an end customer involved in the project. However, I have been on projects where we have a domain expert, or customer proxy; someone who can work in house and make decisions on behalf of a customer. I’ve even heard of people actually being hired from customer-representative companies partly for this purpose.

Even though that customer proxy or domain expert is not what Royce mentions, it’s still cool that he recommended getting customer involvement at a significantly more frequent pace than what he saw as normal before his paper.

Scary big diagram

Royce concludes with a final version of the waterfall diagram with all of his recommended modifications, extra steps, extra documents, and pilot project development.

His goals were admirable, and he has some great recommendations and observations about large scale software development.

However, even with (and partly because of) his modifications, his final version of the software lifecycle diagram has scared the crap out of developers for the last 45 years.

Nobody wants to work like that as a developer. I think fewer and fewer customers want to pay for all that.

Remember that he states that really the only thing adding value is analysis and coding. Everything else is put in place for the sake of control and predictability. These aren’t bad goals. Other than the rather crazy amount of documentation, the paper isn’t all that bad of advice.

But the picture still makes me want to run away.

Comparing Royce's final model to modern iterative models.

Royce might have been off his rocker, but he’s not completely off the porch.

It’s interesting to compare his recommendations and his final model with practices that end up in so called “agile” practices.

1. Preliminary design before analysis and coding.

Ok. This is still a bit weird. But I think his intent is to get designers closer to the code to have more of a say in the earlier processes.

2. Current and complete documentation.

Yeah. That’s the place where he goes completely off the rails. But he didn’t have instant messenger and wiki’s to keep this sort of thing either.

3. Spend 1/4 to 1/3 of the time doing a pilot implementation.

This is really splitting every project into two iterations. Where he mostly differs with recent iterative practices is the split of making the second iteration so big. Modern iterative practices say just keep the iterations the same. Or even make the subsequent iterations shorter than the first.

This is a great start. This is conceding that testing activities earlier is better.

5. Involve the customer more.

Seems like pre-XP to me. He’s recommending that the customer make many of the tradeoff decisions and do more sign off at several stages, where XP wants the customer to move in (sleeping bag and all).

Don’t take my word for it

Please do give the paper a read.
You shouldn’t just take my opinions on the paper. go read it and form you own.