Why is automated regression testing important in an Agile environment? An Agile development approach can dramatically improve the quality of software, but it requires a very different approach to software testing. Here’s a previous blog post with more on why an Agile testing approach makes so much sense:
What’s the Impact of Agile on Regression Testing?
Because the testing approach is so different, regression testing; and, in particular automated regression testing becomes a lot more important:
In a traditional plan-driven project management approach (Waterfall), all testing is typically done at the very end of the project
At that point-in-time, all software development is normally complete and the software is stabilized
In an Agile environment, software testing is done concurrently with development throughout the project
As a result, software is still changing as it is being tested
The difference is shown below:
A big challenge this creates is this:
In a Waterfall-style development process, the software should be stabilized once it goes into testing
In an Agile environment, the software is continuously changing throughout the project. During each sprint, in addition to testing new functionality, it is essential to do regression testing. The purpose of regression testing is to verify that some new changes haven’t inadvertently broken something that was previously tested
In a large, complex project, the number of features requiring regression testing will grow significantly as the project is in progress. That makes it almost impossible to do manual regression testing. Attempting to do manual regression testing would slow down the progress of the project significantly if it can be done at all.
For these reasons, it is important to do regression testing to ensure the quality and integrity of the software in an Agile environment. And, for a large, complex software project, automated regression testing becomes essential.
How many times have you heard people compare Agile versus Waterfall?
It happens a lot,
I do it myself, and
I keep hearing presentations that talk about how Agile has displaced Waterfall
But, if you really think about it, I don’t think that’s a very meaningful comparison and it’s out-of-date. The result of this confusion is people tend too see these two alternatives as binary and mutually-exclusive choices and they try to force-fit a project to one of these extremes:
There are many myths, misconceptions, and stereotypes behind the typical “Agile” versus “Waterfall” comparison. I hope this article will help people see these two alternatives in a fresh new perspective as complementary to each other rather than competitive.
What’s Wrong With The Typical Agile versus Waterfall Comparison?
“Agile” and “Waterfall” Are Not Individual Discrete Methodologies
When people talk about “Agile versus Waterfall”, it sounds like there are two individual discrete methodologies that can be directly compared:
One called “Agile” and
One called “Waterfall”
The truth is that neither “Agile” or “Waterfall” is an individual discrete and well-defined methodology. It would be more accurate to recognize that each one is a style of methodology with more than one different methodology associated with each style. For example:
What is the “Agile” Methodology?
“Agile” is not a single methodology. Agile methodologies include Scrum, Kanban, and other less commonly-used methodologies. Scrum is so widely-used, that when many people say “Agile”, they really mean “Scrum”; however, it’s not really accurate to say that “Agile = Scrum”. In addition, many people consider Scrum to be a general framework and not a specific methodology. Here’s some more detail on that:
The common denominator of these Agile methodologies is that they are flexible and adaptive. That makes them well-suited to an uncertain environment where an empirical process control approach is needed. I think the word “adaptive” is a much better and more accurate term to use to describe these methodologies because it defines a style of methodology and doesn’t convey the idea of a specific methodology at all.
What is the “Waterfall” Methodology?
“Waterfall” is also not a single methodology. When people talk about “Waterfall”, they are not necessarily referring to the pure original phase-driven approach defined by Dr. Winston Royce in the 1970’s. “Waterfall” has evolved significantly since those days and there are many variations that might still be considered to be “Waterfall”. For example, the Rational Unified Process (RUP) and its many variants probably would be considered “Waterfall”.
Again, when people use the word “Waterfall”, they are not generally talking about a specific methodology; they’re talking about a style of methodology. You will find more detail on that in this article:
The common denominator of these methodologies is that they are “plan-driven”. They emphasize trying to achieve predictability over the costs and schedule of a project plan. They do that by nailing down the requirements in detail upfront and controlling any changes once the project is in progress
It’s Not a Binary and Mutually-exclusive Choice
Another serious flaw in the typical “Agile” versus “Waterfall” comparison is that it implies that there is a binary and mutually-exclusive choice between these two extremes and that is also not the case.
These two approaches are not mutually-exclusive. If you recognize that these are really styles of methodologies and not really individual methodologies, there are many ways to blend the two styles together:
Its not a matter of blending a Scrum methodology with a true Waterfall methodology;
It’s a matter of blending a plan-driven style of methodology with a more adaptive style of methodology in the right proportions to fit a given situation.
Agile versus Waterfall – An Example of Sloppy Terminology
A lot of this is just sloppy use of terminology; but, in many cases, there’s also an implication that Agile is good and Waterfall is bad. Here is an example to illustrate what I mean by sloppy use of terminology when people talk about “Waterfall”:
From the 2011 Chaos Report: “Agile Succeeds three times more often than Waterfall” The report goes so far as to say, “The agile process is the universal remedy for software development project failure.”
What do they mean by “Waterfall”? (Are they talking about a specific methodology – like the Waterfall that was defined by Winston Royce in the 1970’s or are they talking about a broader range of plan-driven methodologies?
How did they define how “success” was measured and how do you compare success from one project to another? Are the metrics for success really the same across all projects to make that comparison?
How can anyone possibly say that “The agile process is the universal remedy for software development project failure”?
This is an example of some of the opinionated polarization that has existed for a long time. Saying that “Agile is better than Waterfall” is like saying “A car is better than a boat”. They both have advantages and disadvantages depending on the environment that you’re in.
A More Accurate and Meaningful Comparison
A more meaningful and more objective comparison is between an “adaptive” style of project management and a “plan-driven” style of project management.
“Plan-driven project management” is a style of project management where the requirements and a detailed plan for completing the project can be defined prior to implementing the project
In contrast, an “adaptive” style of project management starts the implementation of a project with a less well-defined plan of how the project will be implemented. Then, the requirements and plan for the project are expected to evolve as the project progresses
Neither of these is an absolute, rigid extreme. In reality, no project is ever totally plan-driven or totally adaptive; you won’t find many projects:
That start out with an absolutely rigid plan that is not expected to change at all, or
That have no plan whatsoever of how the project will be done
There is a broad range of alternative approaches between those two extremes as shown in the diagram below:
Fit the Methodology to the Nature of the Problem
A good rule to follow is don’t force-fit any project to any methodology that doesn’t fit the nature of the project. A methodology does not provide a “cookbook” approach for solving any problem you might have:
Rather than force-fitting a project to some particular methodology (whatever it might be) and taking a “cookbook” approach to applying that methodology mechanically.
The right approach is to go in the other direction and fit the methodology to the nature of the problem you’re trying to solve. It takes more skill to do that but it definitely can be done.
The impact of all the myths, misconceptions, and stereotypes behind the typical Agile versus Waterfall comparison is significant:
Don’t Throw Out the Baby With the Bath Water
By categorizing all plan-driven approaches as “Waterfall”, people tend to dismiss any form of plan-driven approach and regard any kind of upfront planning as inconsistent with an Agile project. That’s not the case.
Get Past the Polarization
There has been a lot of polarization between the traditional project management community and the agile community. There is a perception that project managers and project management are associated with the Waterfall approach. As a result, some people believe that project management skills are not needed at all because the Waterfall approach is an out-of-date approach for many projects.
There is actually a lot of project management going on in an Agile approach although you may not recognize it as “project management” for several reasons:
It’s not the traditional kind of “project management” that is so heavily stereotyped. It’s a different kind of project management with an emphasis on maximizing business results rather than controlling costs and schedules.
You also won’t typically find someone called a “Project Manager” at the team level in an Agile project because the functions that would normally be performed by a project manager have been distributed among other functions on the Agile team.
We need to get past many of the myths, stereotypes, and misconceptions that exist in this area. We need to see these two approaches in a fresh new perspective as complementary rather than competitive to each other.
Quality and reliability standards today for software are higher than ever and new approaches to software testing are essential to meet those goals. An Agile development approach provides a excellent approach to meet that challenge.
How Is Traditional Software Testing Done?
In a traditional software testing approach, a separate and independent QA organization normally does the testing:
The development team typically turns over software to the QA organization for testing at the end of the project
That normally happens after all development is complete.
That method of testing is very consistent with a “Waterfall” style of development. A Waterfall project typically consists of phases that might look something like this:
That’s been the traditional way that software development and testing has been done for a long time.
How Is Agile Testing Different?
Agile takes a fundamentally different approach to the whole development process (not just testing).
First of all, Agile does not break up a project into phases as the Waterfall model does
It breaks up a project by chunks of functionality and uses a much more integrated approach to develop and test each “chunk” of functionality
There are two major differences that have a big impact on how testing is done:
Incremental Development Approach
Integral Testing Approach
We will discuss each of these in the following sections.
1. Incremental Development Approach
Instead of developing and testing the complete functionality of the project as one big effort:
The functionality is broken up into small increments called “user stories” and
The work to be done to develop those user stories is broken up into sprints. Each of those sprints are typically 2-4 weeks long
The work in each sprint includes testing as well as development
Ideally, at the end of each sprint, fully-tested software for that portion of functionality should be ready for release.
2. Integral Testing Approach
Instead of testing being done by a separate and independent QA organization:
The testing function is an integral part of the Agile team, and
As soon as software is sufficiently complete, it is tested
It is worthwhile to note that integration of testing with development does not necessarily mean that developers do their own testing. Within an Agile development team:
There is value in having people who are specialized and trained in testing
Having developers test their own code is generally not a very good idea
What Are the Advantages of the Agile Approach?
There are several significant advantages to the Agile testing approach.
1. Proactive Rather than Reactive Approach
Agile testing is a more proactive approach to eliminate defects at the source. A reactive approach to find and fix defects later can require a lot more rework.
2. Immediate Feedback to Developers
An Agile testing approach provides immediate feedback to the development team. If difficult software bugs go unresolved, they can compound themselves and that can make it much more difficult find and resolve the defects.
3. Quality is Not Someone Else’s Responsibility
Agile testing makes the quality of the product an integral part of the development process. The development team producing the product owns responsibility for the quality of the product they produce. It is not someone else’s responsibility.
In the traditional Waterfall-style development process, it is not uncommon for the development team to throw code “over the fence” to QA. They put the responsibility on QA to find any bugs.
The Roots in Total Quality Management (TQM)
In order to understand why this approach makes so much sense, it’s useful to understand the deeper roots that influenced it:
Agile is heavily based on the principles in Total Quality Management (TQM)
TQM was originally developed for manufacturing by Dr. W. Edwards Deming
The quality of the Japanese automobile industry was improved significantly as a direct result of TQM,
It is relatively easy to extend the TQM principles to software development.
TQM emphasized a major shift in thinking from
A reactive “Quality Control” approach that was heavily based on inspection to
A more proactive “Quality Assurance” approach to go upstream in the process to eliminate defects at the source
That makes sense for a number of reasons:
Resources required to perform the inspection are costly
Defects that are found after the work is already done can result in costly and unnecessary scrap and rework
Any inspection approach is based on sampling is not a thorough approach for finding defects. Some defects will likely slip through when you use sampling resulting in relatively low levels of quality
It is easy to see how these same factors apply to a software environment. Many software testing processes have been based on an old-fashioned quality control approach. which is based heavily on inspection.
I often see questions on “How do you choose the best methodology for a project?”
Is There a Best Methodology for a Project?
I don’t think that there is one single “best” methodology for a project that works for all projects. A lot of people make the mistake of force-fitting a project to some standard methodology.
Traditional Plan-driven Approach (“Waterfall”)
Some project managers will try to use a traditional plan-driven methodology for a project (what many people loosely call “Waterfall”):
It may be the only project management approach that they know.
It has also been widely-accepted as the only way to do project management
Force-fitting a Project to a Methodology
Many other people have the misconception that there is a binary and mutually-exclusive choice between “Agile” and “Waterfall”. As a result, they will attempt to force-fit a project to one of those extremes
Fit the Methodology to the Project
The “best” approach is to go in the other direction and fit the methodology to the nature of the project.
That takes more skill but it definitely can be done
You should also recognize that there is not a binary and mutually-exclusive choice between “Agile” and “Waterfall”
There is a whole spectrum of different approaches ranging from heavily plan-driven at one extreme to heavily adaptive (or Agile) at the other extreme
Factors to Consider
There are a number of factors that influence the selection of the best approach for a particular project:
1. Level of Uncertainty
First and probably the most significant factor in choosing an approach, is the level of uncertainty in the project. A project with a high level of uncertainty would be best-suited for a more adaptive (Agile) approach. Attempting to force-fit such a project to a traditional plan-driven project management approach could be disastrous.
It would force you to make a lot of assumptions to try to resolve the uncertainty; and, in many cases, those assumptions may be wrong and require a lot of re-planning and possible re-work
The emphasis on planning and control in a traditional plan-driven project is not conducive to changes. That will make it difficult to maximize the value of the solution in an uncertain environment
2. Need for Creativity and Innovation
Next, in today’s competitive environment, creativity and innovation can be very important to create very competitive business solutions. An approach with a heavy emphasis on planning and control can stifle creativity and innovation.
3. Customer Relationship
Finally, managing customer expectations is probably one of the most critical aspects of any project. If the results of a project are not consistent with customer expectations, the project will likely not be viewed as successful no matter how good you think it is. The nature of the customer relationship can range between:
Contractual Style of Relationship
A contractual style of relationship is where there are very definite and well-defined customer expectations that must be met. In this type of relationship, the customer may not take any responsibility for the success of the project. The customer defines requirements and expects the project team to do whatever is necessary to meet those requirements. In this style of relationship, there is limited participation by the customer in the project
Naturally, the contractual style of relationship is well-suited to a project with a relatively low level of uncertainty. It must be possible to define the customer’s requirements in some level of detail prior to the start of the project. It would be much more difficult to make a contractual-style relationship work in a project with a high level of uncertainty.
Collaborative Style of Relationship
A collaborative style of relationship is where there is a shared responsibility for the success of the project. In this environment both the project team and the customer take an active role in defining the direction of the project as it is in progress
What’s the Right Style of Relationship?
Of course, the customer has to be amenable to whatever type of relationship you choose. If the project has a high level of uncertainty, that would lean towards more of a collaborative relationship; however:
The customer has to be open to that kind of relationship for it to be successful.
This is a big problem in many companies where it is difficult to break down organizational boundaries between organizations
It requires establishing truly collaborative relationships based on a spirit of shared responsibility, trust, and partnership.
3. Project Team Capabilities
The final major factor in selecting a project approach is, of course, the capabilities of the project team.
An Agile approach requires a lot of training and skill and a hybrid Agile approach can require even more training and skill.
Naturally, it does not make any sense to choose an approach that the team is not capable of implementing.
Why Is Choosing the Best Methodology for a Project Important?
There are two major factors that require us to broaden the way we think about “project management” today:
Solutions tend to be much more complex and the level of uncertainty is much higher
Competitive pressures frequently require much higher levels of creativity and innovation
For those reasons, force-fitting all projects to a standardized plan-driven approach is not necessarily the best approach for all projects.
Choosing the best methodology for a project can be a difficult thing to do. It is not necessarily a simple matter of choosing Agile or Waterfall. You have to fit the methodology to the nature of the project. A project should be focused on producing value for the customer and its very important to understand what “value” means to the customer:
First of all, meeting cost and schedule goals is only one component of value, and it may not be the most important component of value to the customer
Second, there have been many projects that have met their cost and schedule goals but failed to deliver an acceptable level of business value
Finally, “Value” can be a very elusive goal but, in any case, it’s what the customer thinks that “value” is that counts
There is no “best” process. Saying that “Agile is better than Waterfall” is like saying “A car is better than a boat”. Both have advantages and disadvantages depending on the environment you’re in:
When Does an Agile Approach Work Best?
When does an Agile approach work best? An Agile model tends to work best in projects where:
There is a relatively high level of uncertainty and a flexible and adaptive approach is needed to resolve the uncertainty as the project is in progress
Creativity and innovation are needed to maximize the business value of the solution
When Does a Plan-driven Approach Work Best?
When does a plan-driven approach work best? A traditional plan-driven model (what many people loosely call “Waterfall”) tends to work best where:
There is a relatively low level of uncertainty, the solution is well-defined, and some level of planning and control are needed to maximize predictability of costs and schedules
The organization is not really well-prepared to implement an Agile approach and/or the project team is not trained in Agile
I’m sure that you could find at least 100 definitions of what “Agile” is on the Internet. It’s like the old fable of “The Blind Men Feeling the Elephant”:
“It was six men of Indostan To learning much inclined, Who went to see the Elephant (Though all of them were blind), That each by observation Might satisfy his mind”
Each man came away with a different opinion depending on which part of the elephant he touched:
“Each in his own opinion concludes that the elephant is like a wall, snake, spear, tree, fan or rope, depending upon where they had touched”
Different Views of Agile
The way people define Agile is somewhat similar. Depending on which part you touch, you may come away with a different impression of what Agile is:
Many people will define Agile in terms of how it is done. For example, many people will say that it is defined by the Agile Manifesto. Here are a couple of examples of that:
“Agile is a set of methods and frameworks that embody the principles and values of the Agile Manifesto”
“Agile is a term used to describe approaches to software development emphasizing incremental delivery, team collaboration, continual planning, and continual learning. The term “Agile” was coined in 2001 in the Agile Manifesto.”
Some people will say that it is just a mindset or way of thinking. Here’s an example of that.
“Being ‘Agile Is a mindset. It’s about finding the right thing to build, faster (and not just building things faster)”
Some people will define it by comparing it to “Waterfall” to tell you what it is not. Here’s an example of that:
“Agile is a time boxed, iterative approach to software delivery that builds software incrementally from the start of the project, instead of trying to deliver it all at once near the end.”
None of those definitions are incorrect, but it’s like “feeling the elephant”; they don’t really get to the real essence of what the “elephant” is, in my opinion.
How Do You Define Agile?
Personally, I like the definition that is published by the Agile Alliance:
“The ability to create and respond to change in order to succeed in an uncertain and turbulent environment.”
I think that is a better definition because it gets to what I consider the real essence of what Agile is.
“Agile is best suited for situations that have some level of uncertainty where creativity and innovation are important to maximize the business value of the solution as opposed to other situations with lower levels of uncertainty where planning and control to achieve predictability are more important.” (My own definition)
What Problems Does Agile Solve?
We should acknowledge that Agile is not a solution to every problem you might have. One way to define it is in terms of what problems it is useful for.
If you accept that Agile is not a solution to every problem you might have, the first thing you would want to know is “what problems is it useful for?”
You don’t need to get too far into the mechanics of how to do it until you’ve determined that it is a useful solution to the problem you’re trying to solve.
There is an interesting observation you might draw from this – people get very immersed in the mechanics of how to do Agile and that’s how they define it. Isn’t it more important to know what problems it’s useful for solving before you get into the details of how to use it?
“Agile” means a lot of different things to different people. To some people:
It means just developing software faster, or
It means creating a more people-oriented project environment,
To others, it means making the project management process a lot more efficient by streamlining the whole process and eliminating unnecessary documentation
Those are only a few different connotations – there are many, many more. In addition, there are also many more stereotypes, myths, and misconceptions about what Agile is.
All of those things are potential outcomes of an Agile process but that’s not the fundamental essence of what an Agile process is all about in my opinion. The fundamental essence of an Agile process is adaptivity.
What’s Wrong With the Typical Project Management Approach?
Many project management processes, as we know them today, were designed around what is called a “traditional plan-driven project management” model (what many people loosely call “Waterfall“).
In this model, achieving predictability over the outcome of a project and the costs and schedule associated with achieving that outcome is very important
Therefore, it is also very important to have clearly-defined requirements as well as an adequate level of planning to be able to somewhat accurately predict the outcome, costs, and schedule of the project
That’s the predominant way that project management has worked since the 1950’s and 1960’s. A project was considered successful if it delivered what the requirements for the project within the defined budget and schedule.
That kind of predictability can be important. For large investments, it allows a company to:
Attempt to calculate with some level of certainty what the return on their investment is likely to be from a project, and
Make a go/no-go decision as to whether the project should be funded or not based on that information.
The primary problem with that approach is that it requires developing a fairly detailed plan for the project upfront. That is very difficult, if not impossible to do in projects with a very high level of uncertainty.
Why Is Adaptivity Important?
We live in a different world today from the world that existed in the 1950’s and 1960’s when formalized project management approaches were first defined.
Higher Levels of Uncertainty
There is a much higher level of uncertainty:
Technology is rapidly changing,
Solutions are much more complex, and
The business environment that we operate in is dynamic and constantly changing.
In that kind of environment,
Developing a detailed plan for a project with a lot of uncertainty upfront will typically require you to make a lot of assumptions.
And, many times those assumptions will be wrong and may require significant re-planning and possible rework later.
Rather than force-fitting a project that has a high level of uncertainty to a traditional plan-driven approach;
it’s much better to fit the methodology to the nature of the project and that’s where a more adaptive approach really makes sense.
That doesn’t mean that you don’t do any upfront planning; it means that you use a level of planning that is appropriate to the level of uncertainty in the project:
Traditional Plan-driven Approach
Adaptive (Agile) Approach
Atempt to develop a detailed set of requirements and a detailed project plan for the project upfront
Limit the amount of upfront planning based on the level of uncertainty in the project and use a “rolling-wave” planning approach to further define the requirements and plan for the project as the project is in place
Need for Creativity and Innovation
Another important factor in today’s environment is that there is a greater need for creativity and innovation to develop truly leading-edge products. An over-emphasis on planning and control can stifle creativity and innovation.
What’s an Example of a Project Requiring an Adaptive Approach?
A Simple Example
I use an example in my Agile Project Management training that is somewhat extreme but it gets the point across. The example I use is:
Suppose you were given the task to find a cure for cancer and you were asked to outline:
What the solution will be,
How long it will take to develop it, and
What the total cost of the research will be to develop the solution
In that situation, it would be ridiculous to attempt to develop a detailed project plan with accurate cost and schedule estimates – there is just way too many uncertainties to resolve
So what would you do? Give up and do nothing? That doesn’t make sense either
It’s important to recognize that we do know some things about finding a cure for cancer based on years of research that have gone into that area.
However, there are still way too many unknowns to develop a detailed project plan for a solution
What you would do is take advantage of what is known as much as possible and then take an iterative, trial-and-error approach to find a solution
Thomas Edison and the Light Bulb
That’s the way Edison invented the light bulb…here’s a quote from Edison on that subject:
“I speak without exaggeration when I say that I have constructed three thousand different theories in connection with the electric light, each one of them reasonable and apparently to be true. Yet only in two cases did my experiments prove the truth of my theory. My chief difficulty, as perhaps you know, was in constructing the carbon filament, the incandescence of which is the source of the light.”
(1890 Interview in Harper’s Magazine)
In a 1910 autobiography of Edison, Edison’s friend and associate Walter S. Mallory is quoted as asking
“Isn’t it a shame that with the tremendous amount of work you have done you haven’t been able to get any results?” The book goes on to say that “
Edison turned on him like a flash, and with a smile replied:
“Results! Why, man, I have gotten lots of results! I know several thousand things that won’t work!”
What Makes This Kind of Project Different?
There are two things that make this kind of project fundamentally different:
The level of uncertainty is very high and makes it impractical or impossible to develop a detailed plan for the project upfront
Creativity and innovation required for finding a good solution are far more important than predictability
How Does an Agile Approach Solve This Problem?
An Agile process is built on an “Empirical” Process Control model. The word “empirical” means “based on observation”. In the context of an Agile development process, “Empirical” means that during the course of a project, both the product as well as the process to produce the product are continuously refined as the project is in progress. The goal is to produce the right product and to optimize the value of the product being produced.
Why Is This Important to Project Managers?
You might ask “Why is this important to project managers?”
Isn’t Agile something that only applies to software development? (That’s a common misconception)
The truth is that all projects have some level of uncertainty associated with them
If you try to force-fit all projects to a traditional plan-driven project management approach, its just not going to work in many cases. Imagine, for example,
Trying to develop the next generation of the iPhone or any other new and innovative product
In that kind of project, creativity and innovation is just as important, if not more important, than predictability.
In this new environment, a project manager who only knows how to do a traditional plan-driven approach to project management will be at a serious disadvantage. What makes this even more difficult is that:
That this is not a binary and mutually-exclusive choice between “Agile” and “Waterfall” as many people seem to think.
It’s a matter of figuring out how to blend a traditional plan-driven approach with an adaptive (Agile) approach in the right proportions to fit a given situation.
Agile will have a profound impact on the project management profession that will cause us to rethink many things we have taken for granted for a long time about what “project management” is.
Dr Royce described a model that consists of a sequence of phases. In this model, the outputs of one phase flow into the next phase like a “Waterfall”:
The process was called ‘Waterfall’ because the results of one phase flow into the next phase like a waterfall.
What Was Life Like Prior to ‘Waterfall’?
it is useful to understand what life was like prior to Waterfall and what problems it tried to solve. What existed at that time was a lot of poorly-organized development efforts with little or no structure, discipline, and planning. Some of the major problems with the that approach were:
Coordinating Work of Large Development Teams
As projects grew in terms of scope and complexity with potentially much larger numbers of developers, it became apparent that a more planned and structured approach was essential to coordinate the work of large development teams
Cost and Schedule Overruns
The other major problem was that there was very limited predictability over the costs and schedules of software projects:
There were many and frequent very significant cost and schedule overruns, and
Business sponsors demanded some level of predictability
How Did the Waterfall Process Improve These Problems?
When the Waterfall approach was originally defined, it was a big improvement to go from practically no methodology at all to a very well-defined process. The new Waterfall process provided:
A “road map” to:
Coordinate the work of multiple developers as well as
Integrate the work with any other essential resources outside of the immediate development teams, and
A mechanism to gain control over the scope of software projects in order to get more predictability of project costs and schedules
Many younger people today don’t appreciate that history and just criticize Waterfall as being bad without understanding the problems it was intended to solve.
The “Pendulum Effect”
As with many things, there was a “pendulum” effect when the Waterfall approach was initially implemented. There was somewhat of an over-correction in many cases in going from no methodology to a very well-defined methodology. The pendulum swung in many projects from almost no control and discipline to very rigid control and discipline.
It Became Very Rigid and Inflexible
The initial implementation of the Waterfall process had a number of problems that even Dr. Royce recognized in 1970 when he first defined the process. Some of the most serious problems were:
The common practice when the Waterfall process was originally defined in 1970 was a very document-intensive and over-controlled process
You couldn’t exit a phase until all the documentation required to show that the work required for that phase had been completed, reviewed, and approved
The ultimate user of the software didn’t normally even see the software until all of the development and testing was complete and by that point in time; it was very difficult, if not impossible, to go back and make any significant changes
The emphasis on control of scope made the process very inflexible to any changes that might be needed to meet user needs and business goals in an uncertain environment
What Was the Impact?
As a result,
There have been many situations where the project may have met cost and schedule goals but failed to provide a sufficient level of business value
Another major problem was that a heavy emphasis on documentation and other overhead required for reviews and approvals made the whole process bureaucratic and not very cost efficient
It’s important to note that many of the problems associated with “Waterfall” are a result of how it is implemented and not necessarily inherent problems in the methodology itself.
Why Is the Agile versus Waterfall Comparison So Misleading?
A big reason why the typical Agile versus Waterfall comparison is so misleading is that the words “Agile” and “Waterfall” are so loosely-used:
How is the Word “Waterfall” Loosely-used?
Before Agile came into widespread use, many variations on the original Waterfall model were developed to create a more adaptive approach to solve some of these problems.
More iterative processes such as the Rational Unified Process (RUP) and a number of variants became widely-used in the 1990’s and early 2000’s
There has been a proliferation of a broad range of many different development models such as the Spiral model
Some of those have only a very limited resemblance to the original ‘Waterfall’ model as it was defined in 1970.
In spite of this evolution, people still loosely characterize all of those methodologies as ‘Waterfall’ as if it was one specific, unique and well-defined methodology called ‘Waterfall’ and that is not really the case
The common denominator of all the methodologies that people loosely call ‘Waterfall’ is that
They emphasize some level of upfront planning and control.
The goal is to try to achieve predictability over project scope, costs, and schedules.
For that reason, I think the word “plan-driven” is a more accurate and objective description of what people really mean when they say ‘Waterfall’.
How Is the Word “Agile” Loosely-Used?
The word ‘Agile’ is also loosely used. We all know that ‘Agile’ is not a specific methodology although many people equate ‘Agile’ with Scrum:
Scrum is not really a specific methodology, it is really a framework that is intended to be adaptable to a broad range of situations
Agile is not really equivalent to Scrum. There are other Agile methodologies such as Kanban
The common denominator of methodologies that people call ‘Agile’ is that:
They are flexible and adaptive and
Emphasize creativity and innovation in an uncertain environment
Emphasizing planning and control to achieve predictability with lower levels of certainty.
For that reason, I prefer to use the word “adaptive” instead of the word ‘Agile’ when comparing it to ‘Waterfall’ (plan-driven).
When people in the Agile community compare ‘Agile’ and ‘Waterfall’, it’s usually in the context of
Agile is good and Waterfall is bad and that’s really not accurate and objective.
Saying “Agile is better than Waterfall” is like saying “A car is better than a boat” – both have advantages and disadvantages depending on the environment that you are in.
The words “Agile” and “Waterfall” are very loosely used in practice and that causes a lot of confusion. They are used as if both “Agile” and “Waterfall” are unique, individual methodologies and that is not the case.
The word “Waterfall”, in particular, is very loosely-used.
When people use the word “Waterfall”, they’re not necessarily talking about the real “Waterfall model that was defined by Dr Winston Royce in the 1970’s
They’re really talking about any plan-driven methodology that is not completely agile.
A better way to think about “Agile” versus “Waterfall” is in terms of “adaptive” versus “plan-driven”. That’s a much more accurate and objective way of making that comparison.
I recently responded to a question on an online discussion that asked “Are there companies that use Agile and Six Sigma?”. This raises an interesting question of “Are Agile and Six Sigma really complementary to each other?”. How would you go about blending the two approaches?
Agile and Six Sigma – Potentially Conflicting Approaches
There are numerous approaches that might conflict with each other such as:
Agile and Six SIgma,
Lean and Agile, and
Agile and Waterfall
Those approaches have different objectives. If you pursued these approaches individually and independently of each other, the objectives of each approach might be somewhat contradictory. However, if you do it intelligently, it is very possible to blend these approaches in the right proportions.
That requires a lot more skill and
It requires a different kind of thinking to see them as complementary rather than competitive approaches
The Fundamental Problem
There is a significant fundamental problem that must be overcome to see all of these approaches in a different perspective:
Companies and individuals get enamored with a methodology like Agile or Six Sigma and see it as a “silver bullet” solution to any problem that they might have
They attempt to mechanically force-fit their business to one of those methodologies without fully understanding the principles behind it
An Example With Six Sigma
When I published my first book in 2003, Six Sigma was very hot and everyone wanted to “jump on the Six Sigma bandwagon”. At that time, I researched a number of companies that were doing Six Sigma and other process improvement methodologies. What I saw was this:
In companies that seemed to do Six Sigma successfully:
It wasn’t even obvious that it was Six Sigma and they might not have even called it “Six Sigma”:
The implementation wasn’t limited to Six Sigma
They understood the principles behind Six Sigma, and
Might have blended Six Sigma with other process improvement methodologies, and
It was very well-integrated with their business:
It was just a tool that was part of their business
Rather than a program that was superimposed on their business
In other companies, I saw a much more superficial implementation of Six Sigma that didn’t last in many cases:
There was a lot of emphasis on the “mechanics” of doing Six Sigma,
There was a lot of “hoopla” about the ceremonies associated with Six Sigma. (green belts, black belts, etc.), and
They openly advertised that they were using Six Sigma to promote themselves
Does that sound familiar? I think a similar thing is going on with Agile today.
The Key Factor for Success
What I learned from that some of the key factor for success are:
Don’t get overly enamored with any methodology (Six Sigma or anything else):
Don’t think of it as a “silver bullet” for any problem you might have.
Be objective and recognize that any methodology has advantages and limitations depending on the problem you’re trying to solve,
Adapt the methodology to fit the problem and the business environment rather than force-fitting your business to some predefined methodology, and
Go beyond simply doing a mechanical implementation of any methodology (Agile or Six Sigma) and understand the principles behind it at a deeper level
Are Agile and Six Sigma Really Complementary To Each Other?
On the surface, Six Sigma and Agile would tend to pull you in different directions:
Agile emphasizes creativity and innovation as well as flexibility and adaptivity to maximize the business value of the solution
Six Sigma emphasizes process standardization and control of a process to minimize process variation
The key to seeing these approaches as complementary rather than competitive is to understand the fundamental principles behind the approach at a deeper level rather than getting lost in the “mechanics” of the approach.
The essence of Six Sigma is attempting to standardize processes and reduce variation in processes. If you became obsessive about pursuing that goal, it would also not be very consistent with being Agile. However, there is absolutely nothing wrong with attempting to standardize processes to some extent as long as it is also done intelligently and in balance with other objectives.
The importance of “Systems Thinking”
A fundamental skill for doing this successfully is “Systems Thinking”. Systems thinking is essential for seeing these seemingly contradictory approaches in a much broader context. It enables you to see how these objectives can interact in complementary ways rather than being competitive. Here’s an article with more detail on “Systems Thinking”:
It is very possible to blend together different approaches that are seemingly in conflict with each other as long as it is done intelligently. It requires:
Understanding the fundamental principles behind each approach rather than getting lost in the mechanics,
Using a systems thinking” approach to see these seemingly contradictory approaches in a different perspective. Systems thinking enables you to see how they might actually be complementary to each other rather than competitive, and
Learning to fit the methodology to the problem rather than force-fitting a problem to any given methodology
This is exactly the approach behind the Agile Project Management online training courses I’ve developed.
It’s apparent to me that a lot of people have gotten so heavily focused on the mechanics of how Agile is implemented that they’ve lost sight of the big picture of what the real essence of Agile is all about.
The term “Agile” has taken on a number of different meanings today that are largely based on how it is implemented
For many people, “Agile” has become synonymous with Scrum and if you’re not doing Scrum and doing it “by the book”, you’re not really Agile at all
I think it is useful to step back and take a look at “What is the real essence of Agile”?
What is the Real Essence of Agile?
The real essence of “Agile”, in my opinion, is that:
It puts an emphasis on being adaptive to customer and business needs in order to maximize the value of the solution; rather than
Following a rigidly-defined plan with an emphasis on managing costs and schedules of delivering the solution.
For that reason, I like to use the terms “adaptive” and “plan-driven” rather than terms like “Agile” and “Waterfall”.
What’s the Truth About “Agile” vs “Waterfall”?
There’s a lot of myths, stereotypes, and misconceptions about “Agile” and “Waterfall” that we need to get past:
Are Agile and Waterfall Distinct and Unique Methodologies?
The terms “Agile” and “Waterfall” make it sound like you’re comparing two specific methodologies:
One called “Agile” and
One called “Waterfall”
and that’s not really accurate:
“Agile” is not really a specific methodology or framework like Scrum;
It is much broader than that – it is a way of thinking defined by the Agile Manifesto
In It a Binary and Mutually-Exclusive Choice?
The “Agile versus Waterfall” kind of thinking leads people:
To think that there is a binary and mutually-exclusive choice between those two approaches and
That causes people to try to force-fit a project to one of those extremes.
The right approach is to go in the other direction and fit the methodology to the nature of the problem and sometimes that requires a blending the two approaches in the right proportions to fit the problem
Choosing the Best Approach
Here are some guidelines for choosing the best approach.
When Does Agile Work Best?
Agile works best in situations that have a high-level of uncertainty where it isn’t practical or possible to define the solution to the problem upfront. In that kind of situation, it is much more effective to use an adaptive approach to incrementally and iteratively define the solution in more detail as the project is in progress rather than attempting to define detailed requirements for the solution upfront.
The example I like to use to illustrate this is finding a cure for cancer.
If you set out to define a project to find a cure for cancer, it would be ridiculous to try to define a detailed plan upfront; there’s just far too much uncertainty.
The only approach that is likely to work is an iterative, trial-and-error approach to find a solution.
Agile is based on what is called an “Empirical Process Control” model. The word “Empirical” means “based on observation” and that means that in an Agile project, both the requirements for the solution AND the process for developing the solution are continuously refined as the project is in progress.
When Does a Plan-driven Approach Work Best?
That kind of empirical process control model approach is naturally not the most efficient approach if there is a low level of uncertainty and it is possible to easily define detailed requirements for the solution prior to the start of the project. In that situation, a trial-and-error approach really isn’t necessary and a plan-driven approach is much more appropriate and much more efficient.
The example I like to use to illustrate this scenario is building a bridge across a river. If you were defining a project to construct a bridge across a river, it would be ridiculous to say “we’ll take an adaptive approach, build the first span of the bridge, and decide how we will build the rest of the bridge after we’ve completed the first span. That wouldn’t make any sense.
This kind of situation calls for a plan-driven approach that is based on a defined process control model.
The key advantage of a defined process model is that it is predictable and repeatable and it is probably more efficient for projects with a low level of uncertainty.
If you designed a process for building a bridge and it has proven successful once, the same process or a variation of that process is likely to work in another similar situation with somewhat predictable results.
What if it is Between Those Extremes?
Very few real world situations are as extreme and clear-cut as the ones I’ve used of finding a cure for cancer and building a bridge across a river.
Most real-world situations fall somewhere between those two extremes.
There is some level of uncertainty but it’s not complete uncertainty.
You rarely start any project without at least some idea of what the solution is going to look like although the solution may be progressively refined in more detail as the project is in progress. There’s a range of approaches that look something like this:
In this kind of situation, you have to tailor the approach to fit the nature of the project:
One of the biggest factors to consider is the level of uncertainty associated with the solution.
That requires more skill but it definitely can be done
It requires knowledge of a broader range of methodologies (both plan-driven and adaptive (Agile))
It also requires a deeper knowledge of the principles behind the methodologies in order to know how to tailor them to fit a given situation.
You can’t just force-fit a situation to some predefined methodology (whatever it might be) and do it mechanically.
I’ve participated in several discussions and presentations lately where the subject of Lean and Agile came up and I think the relationship of the two is very interesting. If you pursued each of those approaches to the extreme and tried maximize what you would get out of both, they would tend to pull you in different directions:
Lean emphasizes reducing “waste” and Agile emphasizes flexibility and adaptivity to meet customer needs.
Those two approaches are not totally compatible with each other; however, they are not necessarily incompatible either. It just requires some skill to blend them together in the right proportions to fit a given situation.
Here’s an example, Michael Nir recently made a presentation at an Agile Boston meeting about “The Agile PMO” which was based on his book of the same name. Michael indicated that a key potential role of an Agile PMO is to reduce waste in an organization and that goal is very consistent with Lean. An example of that could be under-utilization of people in the organization.
How Does Lean Reduce Waste?
Michael indicated that a key potential role of an Agile PMO is to reduce waste in an organization and that goal is very consistent with Lean. For example,
Under-utilization of people in the organization,
Under-utilization of resources, or
Less than optimum utilization of resources
could certainly be a major source of waste in an organization. There are a number of ways that a PMO can reduce waste:
Utilization of Specialized Resources
If specialized resources that are not dedicated to project teams (such as DBA’s) are not well-planned and coordinated across teams:
Project teams may be idle waiting for these specialized resources, or
The specialized resources might not be fully-utilized waiting for work from project teams
Project Portfolio Management
If a project portfolio is not well-managed, allocation of resources to project teams may not be not well-aligned with company business goals and priorities
Project Management of Individual Projects
If individual projects are not well-managed and are allowed to go off track, the allocation of resources to projects may not be optimized to maximize the business results for the company
Development Process Definition and Training
The development process is not well-defined,
Tools aren’t adequate to support the process, and/or
Project teams are not well-trained to execute the process
the execution of the process will not be consistent across teams and may not be as efficient and effective as it could be
In all of those areas, a PMO might add value by reducing waste but how far do you go with that?
Can You Reduce Waste to Zero?
Carried to an extreme, a focus on simply reducing waste could easily become dysfunctional. Michael mentioned that waste in some organizations could be as high as 95%. What would happen if you attempted to reduce waste to 0%?
First, reducing waste to 0% is probably an unrealistic and impossible goal. No business is totally predictable where everything is known in advance to enable perfect prioritization, planning, and scheduling of resources
Second, putting too much emphasis on reducing waste would would mean superimposing a level of control and standardization on projects. That could easily be inconsistent with achieving the flexibility and adaptivity required by an Agile approach
What’s the Right Answer?
Given that conflict, what’s the right answer? This is not necessarily an easy problem to solve. It will take some skill to figure out the right blend of:
Focusing on lean and reducing waste and
Preserving the flexibility and adaptivity required by an Agile approach.
There clearly seems to be an optimum point between the two extremes of focusing on those two extremes individually. A PMO could probably perform a value-added role in helping an organization find that optimum point.
Finding that optimum point is yet another example of the need for “systems thinking”. Here’s a previous post I wrote on that subject:
People many times like to over-simplify what is really much more complex and reduce it to a simple, binary choice between two extremes:
“Agile” versus “Waterfall” is one example of that and
“Lean” versus “Agile” is another example.
On the surface, Lean and Agile might appear to be in conflict with each other. If you pursued each approach individually and mechanically without really understanding the principles behind each at a deeper level, they could easily be in conflict.
On the other hand, if you take take a systems-thinking approach to understand these seemingly disparate approaches at a deeper level. you will begin to develop a fresh new perspective to see them as complementary to each other rather than competitive.
Michael made a key point that it is a matter of focusing on value versus control and he’s absolutely right. Here are some ways a PMO could add value:
Better defining processes and tools,
Providing training to people, and
Doing some level of project portfolio management and resource planning of people
Each of those can potentially add value; however, it does take some skill to determine the optimum point beyond which it stops producing value and starts to become dysfunctional.