Category Archives: Agile versus Waterfall

How Do You Choose the Best Methodology for a Project?

I often see questions on “How do you choose the best methodology for a project?”

choose the best methodology for a project

Is There a Best Methodology for a Project?

I don’t think that there is one single “best” methodology for a project that works for all projects.  A lot of people make the mistake of force-fitting a project to some standard methodology.

Traditional Plan-driven Approach (“Waterfall”)

Some project managers will try to use a traditional plan-driven methodology for a project (what many people loosely call “Waterfall”):

  • It may be the only project management approach that they know.
  • It has also been widely-accepted as the only way to do project management

Force-fitting a Project to a Methodology

Many other people have the misconception that there is a binary and mutually-exclusive choice between “Agile” and “Waterfall”. As a result, they will attempt to force-fit a project to one of those extremes

Fit the Methodology to the Project

The “best” approach is to go in the other direction and fit the methodology to the nature of the project. 

  • That takes more skill but it definitely can be done
  • You should also recognize that there is not a binary and mutually-exclusive choice between “Agile” and “Waterfall”
  • There is a whole spectrum of different approaches ranging from heavily plan-driven at one extreme to heavily adaptive (or Agile) at the other extreme
Range of Agility

Factors to Consider

There are a number of factors that influence the selection of the best approach for a particular project:

1. Level of Uncertainty

First and probably the most significant factor in choosing an approach, is the level of uncertainty in the project. A project with a high level of uncertainty would be best-suited for a more adaptive (Agile) approach. Attempting to force-fit such a project to a traditional plan-driven project management approach could be disastrous.

  • It would force you to make a lot of assumptions to try to resolve the uncertainty; and, in many cases, those assumptions may be wrong and require a lot of re-planning and possible re-work
  • The emphasis on planning and control in a traditional plan-driven project is not conducive to changes. That will make it difficult to maximize the value of the solution in an uncertain environment

2. Need for Creativity and Innovation

Next, in today’s competitive environment, creativity and innovation can be very important to create very competitive business solutions. An approach with a heavy emphasis on planning and control can stifle creativity and innovation.

3. Customer Relationship

Finally, managing customer expectations is probably one of the most critical aspects of any project.  If the results of a project are not consistent with customer expectations, the project will likely not be viewed as successful no matter how good you think it is.  The nature of the customer relationship can range between:

Contractual Style of Relationship

A contractual style of relationship is where there are very definite and well-defined customer expectations that must be met.  In this type of relationship, the customer may not take any responsibility for the success of the project.  The customer defines requirements and expects the project team to do whatever is necessary to meet those requirements. In this style of relationship, there is limited participation by the customer in the project

Naturally,  the contractual style of relationship is well-suited to a project with a relatively low level of uncertainty. It must be possible to define the customer’s requirements in some level of detail prior to the start of the project.  It would be much more difficult to make a contractual-style relationship work in a project with a high level of uncertainty.

Collaborative Style of Relationship

A collaborative style of relationship is where there is a shared responsibility for the success of the project. In this environment both the project team and the customer take an active role in defining the direction of the project as it is in progress

What’s the Right Style of Relationship?

Of course, the customer has to be amenable to whatever type of relationship you choose.  If the project has a high level of uncertainty, that would lean towards more of a collaborative relationship; however:

  • The customer has to be open to that kind of relationship for it to be successful. 
  • This is a big problem in many companies where it is difficult to break down organizational boundaries between organizations
  • It requires establishing truly collaborative relationships based on a spirit of shared responsibility, trust, and partnership.

3. Project Team Capabilities

The final major factor in selecting a project approach is, of course, the capabilities of the project team.   

  • An Agile approach requires a lot of training and skill and a hybrid Agile approach  can require even more training and skill. 
  • Naturally, it does not make any sense to choose an approach that the team is not capable of implementing.

Why Is Choosing the Best Methodology for a Project Important?

There are two major factors that require us to broaden the way we think about “project management” today:

  • Solutions tend to be much more complex and the level of uncertainty is much higher
  • Competitive pressures frequently require much higher levels of creativity and innovation

For those reasons, force-fitting all projects to a standardized plan-driven approach is not necessarily the best approach for all projects.

Overall Summary

Choosing the best methodology for a project can be a difficult thing to do. It is not necessarily a simple matter of choosing Agile or Waterfall. You have to fit the methodology to the nature of the project. A project should be focused on producing value for the customer and its very important to understand what “value” means to the customer:

  • First of all, meeting cost and schedule goals is only one component of value, and it may not be the most important component of value to the customer
  • Second, there have been many projects that have met their cost and schedule goals but failed to deliver an acceptable level of business value
  • Finally, “Value” can be a very elusive goal but, in any case, it’s what the customer thinks that “value” is that counts

There is no “best” process. Saying that “Agile is better than Waterfall” is like saying “A car is better than a boat”. Both have advantages and disadvantages depending on the environment you’re in:

When Does an Agile Approach Work Best?

When does an Agile approach work best? An Agile model tends to work best in projects where:

  • There is a relatively high level of uncertainty and a flexible and adaptive approach is needed to resolve the uncertainty as the project is in progress
  • Creativity and innovation are needed to maximize the business value of the solution

When Does a Plan-driven Approach Work Best?

When does a plan-driven approach work best? A traditional plan-driven model (what many people loosely call “Waterfall”)  tends to work best where:

  • There is a relatively low level of uncertainty, the solution is well-defined, and some level of planning and control are needed to maximize predictability of costs and schedules
  • The organization is not really  well-prepared to implement an Agile approach and/or the project team is not trained in Agile

Additional Resources

You will find much more detail on this in my Online Agile Project Management Training.

In addition, you may also be interested in the following articles related articles:

What Does Waterfall Mean? How Do People Use That Word?

What does ‘Waterfall’ really mean? Have you ever thought about that?

  • The word “Waterfall” is often used in comparison to ‘Agile’ but do people know what they really mean when they compare ‘Agile’ to ‘Waterfall’?
  • I think the word ‘Waterfall’ is one of the most loosely-used words in the English language (the word ‘Agile’ is not far behind).

When people talk about ‘Agile’ and ‘Waterfall’,

  • it sounds like they’re comparing two very specific and well-defined methodologies that are binary and mutually-exclusive opposites of each other
  • However, when you dig into what the words ‘Waterfall’ and ‘Agile’ really mean, you quickly discover that’s a very inaccurate and misleading comparison
What Does Waterfall Mean?

What Does Waterfall Mean?

Strictly speaking, the word ‘Waterfall’ was originally defined in 1970 by Dr. Winston Royce in his very famous paper:

Dr. Winston Royce’s 1970 Waterfall Paper

Dr Royce described a model that consists of a sequence of phases. In this model, the outputs of one phase flow into the next phase like a “Waterfall”:

What Is Waterfall

The process was called ‘Waterfall’ because the results of one phase flow into the next phase like a waterfall.

What Was Life Like Prior to ‘Waterfall’?

it is useful to understand what life was like prior to Waterfall and what problems it tried to solve. What existed at that time was a lot of poorly-organized development efforts with little or no structure, discipline, and planning. Some of the major problems with the that approach were:

Coordinating Work of Large Development Teams

As projects grew in terms of scope and complexity with potentially much larger numbers of developers, it became apparent that a more planned and structured approach was essential to coordinate the work of large development teams

Cost and Schedule Overruns

The other major problem was that there was very limited predictability over the costs and schedules of software projects:

  • There were many and frequent very significant cost and schedule overruns, and
  • Business sponsors demanded some level of predictability

How Did the Waterfall Process Improve These Problems?

When the Waterfall approach was originally defined, it was a big improvement to go from practically no methodology at all to a very well-defined process. The new Waterfall process provided:

  • A “road map” to:
    • Coordinate the work of multiple developers as well as
    • Integrate the work with any other essential resources outside of the immediate development teams, and
  • A mechanism to gain control over the scope of software projects in order to get more predictability of project costs and schedules

Many younger people today don’t appreciate that history and just criticize Waterfall as being bad without understanding the problems it was intended to solve.

The “Pendulum Effect”

As with many things, there was a “pendulum” effect when the Waterfall approach was initially implemented. There was somewhat of an over-correction in many cases in going from no methodology to a very well-defined methodology. The pendulum swung in many projects from almost no control and discipline to very rigid control and discipline.

It Became Very Rigid and Inflexible

The initial implementation of the Waterfall process had a number of problems that even Dr. Royce recognized in 1970 when he first defined the process. Some of the most serious problems were:

  • The common practice when the Waterfall process was originally defined in 1970 was a very document-intensive and over-controlled process
  • You couldn’t exit a phase until all the documentation required to show that the work required for that phase had been completed, reviewed, and approved
  • The ultimate user of the software didn’t normally even see the software until all of the development and testing was complete and by that point in time; it was very difficult, if not impossible, to go back and make any significant changes
  • The emphasis on control of scope made the process very inflexible to any changes that might be needed to meet user needs and business goals in an uncertain environment

What Was the Impact?

As a result,

  • There have been many situations where the project may have met cost and schedule goals but failed to provide a sufficient level of business value
  • Another major problem was that a heavy emphasis on documentation and other overhead required for reviews and approvals made the whole process bureaucratic and not very cost efficient

It’s important to note that many of the problems associated with “Waterfall” are a result of how it is implemented and not necessarily inherent problems in the methodology itself.

Why Is the Agile versus Waterfall Comparison So Misleading?

A big reason why the typical Agile versus Waterfall comparison is so misleading is that the words “Agile” and “Waterfall” are so loosely-used:

How is the Word “Waterfall” Loosely-used?

Before Agile came into widespread use, many variations on the original Waterfall model were developed to create a more adaptive approach to solve some of these problems.

  • More iterative processes such as the Rational Unified Process (RUP) and a number of variants became widely-used in the 1990’s and early 2000’s
  • There has been a proliferation of a broad range of many different development models such as the Spiral model
  • Some of those have only a very limited resemblance to the original ‘Waterfall’ model as it was defined in 1970.

In spite of this evolution, people still loosely characterize all of those methodologies as ‘Waterfall’ as if it was one specific, unique and well-defined methodology called ‘Waterfall’ and that is not really the case

The common denominator of all the methodologies that people loosely call ‘Waterfall’ is that

  • They emphasize some level of upfront planning and control.
  • The goal is to try to achieve predictability over project scope, costs, and schedules.

For that reason, I think the word “plan-driven” is a more accurate and objective description of what people really mean when they say ‘Waterfall’.

How Is the Word “Agile” Loosely-Used?

The word ‘Agile’ is also loosely used. We all know that ‘Agile’ is not a specific methodology although many people equate ‘Agile’ with Scrum:

  • Scrum is not really a specific methodology, it is really a framework that is intended to be adaptable to a broad range of situations
  • Agile is not really equivalent to Scrum. There are other Agile methodologies such as Kanban

The common denominator of methodologies that people call ‘Agile’ is that:

  • They are flexible and adaptive and
  • Emphasize creativity and innovation in an uncertain environment

rather than:

  • Emphasizing planning and control to achieve predictability with lower levels of certainty.

For that reason, I prefer to use the word “adaptive” instead of the word ‘Agile’ when comparing it to ‘Waterfall’ (plan-driven).

Overall Summary

When people in the Agile community compare ‘Agile’ and ‘Waterfall’, it’s usually in the context of

  • Agile is good and Waterfall is bad and that’s really not accurate and objective.
  • Saying “Agile is better than Waterfall” is like saying “A car is better than a boat” – both have advantages and disadvantages depending on the environment that you are in.

The words “Agile” and “Waterfall” are very loosely used in practice and that causes a lot of confusion. They are used as if both “Agile” and “Waterfall” are unique, individual methodologies and that is not the case.

The word “Waterfall”, in particular, is very loosely-used.

  • When people use the word “Waterfall”, they’re not necessarily talking about the real “Waterfall model that was defined by Dr Winston Royce in the 1970’s
  • They’re really talking about any plan-driven methodology that is not completely agile.

A better way to think about “Agile” versus “Waterfall” is in terms of “adaptive” versus “plan-driven”. That’s a much more accurate and objective way of making that comparison.

Additional Resources

You will find much more detail on this in my Online Agile Project Management Training.

What is the Real Essence of Agile? What Are the Real Advantages?

It’s apparent to me that a lot of people have gotten so heavily focused on the mechanics of how Agile is implemented that they’ve lost sight of the big picture of what the real essence of Agile is all about. 

  • The term “Agile” has taken on a number of different meanings today that are largely based on how it is implemented
  • For many people, “Agile” has become synonymous with Scrum and if you’re not doing Scrum and doing it “by the book”, you’re not really Agile at all

I think it is useful to step back and take a look at “What is the real essence of Agile”?

What is the Real Essence of Agile?

What is the Real Essence of Agile?

The real essence of “Agile”, in my opinion, is that:

  • It puts an emphasis on being adaptive to customer and business needs in order to maximize the value of the solution; rather than
  • Following a rigidly-defined plan with an emphasis on managing costs and schedules of delivering the solution.

For that reason, I like to use the terms “adaptive” and “plan-driven” rather than terms like “Agile” and “Waterfall”.

What’s the Truth About “Agile” vs “Waterfall”?

There’s a lot of myths, stereotypes, and misconceptions about “Agile” and “Waterfall” that we need to get past:

Are Agile and Waterfall Distinct and Unique Methodologies?

The terms “Agile” and “Waterfall” make it sound like you’re comparing two specific methodologies:

  • One called “Agile” and
  • One called “Waterfall”

and that’s not really accurate: 

  • “Agile” is not really a specific methodology or framework like Scrum;
  • It is much broader than that – it is a way of thinking defined by the Agile Manifesto

In It a Binary and Mutually-Exclusive Choice?

The “Agile versus Waterfall” kind of thinking leads people:

  • To think that there is a binary and mutually-exclusive choice between those two approaches and
  • That causes people  to try to force-fit a project to one of those extremes. 
  • The right approach is to go in the other direction and fit the methodology to the nature of the problem and sometimes that requires a blending the two approaches in the right proportions to fit the problem

Choosing the Best Approach

Here are some guidelines for choosing the best approach.

When Does Agile Work Best?

Agile works best in situations that have a high-level of uncertainty where it isn’t practical or possible to define the solution to the problem upfront.  In that kind of situation, it is much more effective to use an adaptive approach to incrementally and iteratively define the solution in more detail as the project is in progress rather than attempting to define detailed requirements for the solution upfront.

The example I like to use to illustrate this is finding a cure for cancer.  

  • If you set out to define a project to find a cure for cancer, it would be ridiculous to try to define a detailed plan upfront; there’s just far too much uncertainty. 
  • The only approach that is likely to work is an iterative, trial-and-error approach to find a solution.

Agile is based on what is called an “Empirical Process Control” model.  The word “Empirical” means “based on observation” and that means that in an Agile project, both the requirements for the solution AND the process for developing the solution are continuously refined as the project is in progress.

When Does a Plan-driven Approach Work Best?

That kind of empirical process control model approach is naturally not the most efficient approach if there is a low level of uncertainty and it is possible to easily define detailed requirements for the solution prior to the start of the project. In that situation, a trial-and-error approach really isn’t necessary and a plan-driven approach is much more appropriate and much more efficient.

The example I like to use to illustrate this scenario is building a bridge across a river. If you were defining a project to construct a bridge across a river, it would be ridiculous to say “we’ll take an adaptive approach, build the first span of the bridge, and decide how we will build the rest of the bridge after we’ve completed the first span. That wouldn’t make any sense.

This kind of situation calls for a plan-driven approach that is based on a defined process control model.

  • The key advantage of a defined process model is that it is predictable and repeatable and it is probably more efficient for projects with a low level of uncertainty.
  • If you designed a process for building a bridge and it has proven successful once, the same process or a variation of that process is likely to work in another similar situation with somewhat predictable results.

What if it is Between Those Extremes?

Very few real world situations are as extreme and clear-cut as the ones I’ve used of finding a cure for cancer and building a bridge across a river.

  • Most real-world situations fall somewhere between those two extremes.
  • There is some level of uncertainty but it’s not complete uncertainty.
  • You rarely start any project without at least some idea of what the solution is going to look like although the solution may be progressively refined in more detail as the project is in progress. There’s a range of approaches that look something like this:
Range of Agility

In this kind of situation, you have to tailor the approach to fit the nature of the project:

  • One of the biggest factors to consider is the level of uncertainty associated with the solution.
  • That requires more skill but it definitely can be done
  • It requires knowledge of a broader range of methodologies (both plan-driven and adaptive (Agile))
  • It also requires a deeper knowledge of the principles behind the methodologies in order to know how to tailor them to fit a given situation.
  • You can’t just force-fit a situation to some predefined methodology (whatever it might be) and do it mechanically.

Additional Resources

You will find much more detail on this in my Online Agile Project Management Training.