Tag Archives: Agile Versus Waterfall

My Methodology is Better than Your Methodology

There are a lot of people who get caught up in what I call “methodology wars” where they are intent on their position that my methodology is better than your methodology and whatever methodology they advocate is better at solving any problem you can possibly imagine than any other methodology. You can see this in the many “agile versus waterfall” discussions and other discussions where SAFe, Kanban, or some other methodology/framework is positioned as a “silver bullet” for any problem you might have. They also tend to ignore all other methodologies as obsolete or irrelevant.

The truth is that all methodologies and frameworks have strengths and weaknesses depending on the situation and the right approach is to fit the methodology to the situation rather than force-fitting a problem to some pre-defined methodology. Sometimes that may require customizing a methodology to fit the problem and/or using a combination of elements from different methodologies. It’s a lot more difficult to do that, but it can be done – it requires:

  • Knowledge of a broader range of methodologies and frameworks,
  • Ability to see the strengths and weaknesses of those methodologies objectively, and
  • A deeper understanding of the principles behind those methodologies to know how they might be combined to fit a given situation

Here’s an example – I just finished adding some material on “Lean Software Development” to my online training courses on Agile Project Management.  Lean is not widely-used as a standalone methodology and it clearly didn’t win the “methodology wars” but the principles behind lean are the foundation for all Agile methodologies including Scrum.  If you look at the principles behind lean, they may appear to be at odds with other Agile methodologies:

  • Lean heavily emphasizes eliminating waste in a process to improve efficiency, while
  • Agile is more heavily-focused on taking a flexible and adaptive approach to meet customer needs and is less concerned about just eliminating waste

If you pursued each of these goals in isolation and to an extreme; they might be in conflict with each other, but if they are blended together in the right proportions to fit a given situation, they can be very complementary rather than competitive.

Here’s another example – many people seem to believe that all forms of traditional project management are obsolete and irrelevant and have been totally replaced by Agile.  On the surface; if you look at traditional, plan-driven project management and Agile, they may appear to be at odds with each other; and if each approach is pursued in isolation and to an extreme, they probably will be in conflict but that shouldn’t preclude blending the principles behind the two approaches together in the right proportions to fit a given situation.

This kind of thinking is commonly called “Systems thinking” – it requires seeing a problem in a holistic context and understanding the dynamics of the problem at a deeper level rather than mechanically imposing a predefined solution on a given problem.  This is the kind of approach I’ve tried to help students develop in all of my online Agile Project Management training courses.

Learn the Truth About “Agile” versus “Waterfall”

Background

How many times have you heard people compare “Agile versus Waterfall”? It happens a lot, I do it myself, and I keep hearing presentations that talk about how Agile has displaced “Waterfall”. But, if you really think about it, I don’t think that’s a very meaningful comparison and it’s out-of-date. Learn the Truth About “Agile” versus “Waterfall” – True “Waterfall”, as a methodology, died a long time ago for most projects outside of some specialized areas like construction; yet people continue to make that comparison.

The problem is that the word “Waterfall” is used very loosely and indiscriminately. In many cases, when people use the word “Waterfall”, they’re not using it to refer to the specific “Waterfall” methodology that was originally defined by Winston Royce in the 1970’s, they’re using it loosely to refer to a general style of project management that emphasizes some level of planning, predictability, and control over agility. Here’s an example – iterative methodologies such as the Rational Unified Process (RUP) became very popular in the 1990’s and early 2000’s and many people would consider that to be “Waterfall” just because they are somewhat plan-driven, but they don’t really fit the full definition of “Waterfall” at all.

An Example of Sloppy Terminology

Don’t get me wrong – I think Agile has huge benefits. I just want people to objectively understand the benefits of Agile versus Waterfall and the sloppy use of terminology to compare the two is often misleading and confusing. Here is an example I’ve taken from a real world source that is considered fairly credible to illustrate what I mean by sloppy use of technology when people talk about “Waterfall”:

Blue Line

From the 2011 Chaos Report: “Agile Succeeds three times more often than Waterfall”
The report goes so far as to say, “The agile process is the universal remedy for software development project failure.”

What do they mean by “Waterfall”? (Are they talking about a specific methodology – like the Waterfall that was defined by Winston Royce in the 1970’s or are they talking about a broader range of plan-driven methodologies?

How did they define how “success” was measured?

How can anyone possibly say that “The agile process is the universal remedy for software development project failure”?)

Blue Line

Saying that “Agile is better than Waterfall” is like saying “A car is better than a boat”. They both have advantages and disadvantages depending on the environment that you’re in. When people use the word “Waterfall” like this, I’m tempted to ask, “Which aspect of ‘Waterfall’ are you referring to?”

  • Are you referring to the phase gate approach where a project is broken up into phases and there is a phase gate for approval to transition between phases? I don’t think that approach has been widely practiced for years for software development projects and even Winston Royce himself had reservations about it
  • Are you referring to an over-reliance on documentation? That is a more legitimate comparison because Winston Royce did come out very strongly in support of a lot of documentation, but that shouldn’t imply that an Agile project has no documentation whatsoever.
  • Are you referring to the tendency to plan an entire project upfront before starting the project and then manage changes to the project requirements through change control? That also might be a legitimate comparison, but it also shouldn’t be meant to imply that an Agile project shouldn’t do any planning upfront.
  • Are you referring to the practice of attempting to complete all of the project requirements all at once? Long before Agile became well-known, iterative approaches like the Rational Unified Process (RUP) provided a way to solve that problem and break up a project into iterations.

A More Meaningful Comparison

A more meaningful and more objective comparison is between an “adaptive” approach to project management and a “plan-driven” approach to project management. “Plan-driven project management” is a style of project management that is applied to projects where the requirements and plan for completing the project can be defined to some extent prior to implementing the project. In contrast, an “adaptive” style of project management starts the implementation of a project with a less well-defined plan of how the project will be implemented and the requirements and plan for the project are expected to evolve as the project progresses.

No project is ever totally plan-driven or totally adaptive; you won’t find many projects that start out with an absolutely rigid plan that is not expected to change at all, and you won’t find many projects that have no plan whatsoever of how the project will be done. There is a broad range of alternative approaches between those two extremes as shown in the diagram below:

Increasing Agility and Adaptivity

It is a matter of choosing the right level of upfront planning to be applied to a project based on the level of uncertainty and other factors in the project and it takes some skill to do that.

There is nothing inherently wrong with either of these approaches (adaptive or plan-driven). They both have advantages and disadvantages for a given project and they should be seen more as complementary approaches rather than competitive. Instead, many people see “Agile” and “Waterfall” as binary and mutually-exclusive choices and that causes people to try to force-fit a project to one of those extremes rather than selecting and tailoring the approach to fit the project. For example,

  • If I were to set out to try to find a cure for cancer and I attempted to apply a highly plan-driven approach to that project, the results would probably be very dismal
  • Similarly, if I tried to use an agile approach for building a bridge across a river, the results would be equally dismal

Why does that happen? It takes much more skill to fit a methodology (or combination of methodologies) to a project – you have to know a broader range of methodologies and you have to understand the principles behind the methodologies at a deeper level to know how to tailor the methodology and blend different methodologies together to fit a given situation. Some people are primarily skilled in one particular methodology and tend to implement that methodology mechanically “by the book”. It’s like being a carpenter and the only tool in your tool bag is a hammer.

Overall Summary

The impact of misusing the word “Waterfall” is significant:

  • It causes people to “throw out the baby with the bath water”. By misusing the word “Waterfall” and categorizing all plan-driven approaches as “Waterfall”, people tend to dismiss any form of plan-driven approach and to regard any kind of upfront planning as inconsistent with an Agile project.
  • It has caused a lot of polarization between the traditional project management community and the agile community. The perception is that project managers are associated with the Waterfall approach and, as a result, project management skills are not needed because the Waterfall approach is an out-of-date approach for many projects.

The true Waterfall approach has been obsolete for many projects for a long time (the exception being some selected industries like construction where it is still very useful and relevant). So, I don’t think comparing Agile to Waterfall is very meaningful any more, but its very difficult to get people to stop thinking in those terms because it has been so prevalent for such a long time.

The difference between a highly adaptive project and a highly plan-driven project is how much of that planning is done upfront in the project rather than being deferred till later. It’s not a black-and-white decision to have a totally plan-driven approach or a totally adaptive approach and it requires some skill and judgment to determine what level of upfront planning makes sense in a given project. When people present this decision as “Agile versus Waterfall” it distorts what the real decision is and makes it look like a binary, all-or-nothing choice and that’s not the case.

Additional Resources

I’ve created a free online training course that provides more information on this topic:

Learn the Truth About Agile versus Waterfall

Managed Agile Development Framework

I’ve seen many people ask a question like “should I use Agile or Waterfall for a project? That presumes that this is a binary, all-or-nothing choice that you have to choose one or the other and not both. It excludes the possibility that there is a hybrid approach that provides the benefits of both approaches. The Managed Agile Development Framework is an example of a hybrid approach that is very easy to implement

A few years ago I was responsible for managing a large government project that required meeting some defined cost and schedule milestones but the customer wanted to take an Agile approach to defining the requirements. In response to that project, I developed an approach which I call “The Managed Agile Development” framework that would satisfy those two seemingly inconsistent goals.

  • We were able to successfully build a partnership with the government client in which we did a very professional job of managing overall contractual requirements at the “macro-level”, and
  • Within that “macro level” envelope, we were still able to implement a fairly Agile development approach at the “micro-level”

Managed Agile Development Framework

I’m providing a brief description of how it works here (refer to my book for more details). There are two layers in the framework as shown in the diagram above. The “Macro” layer is plan-driven; but it can be as “thick” or “thin” as you want it to be. The “Micro” layer can be any Agile development approach such as Scrum.

  • The macro-level framework is a plan-driven approach, designed to provide a sufficient level of control and predictability for the overall project. It defines the outer envelope (scope and high-level requirements) that the project operates within
  • Within that outer envelope, the micro-level framework utilizes a more flexible and iterative approach based on an Agile Scrum approach that is designed to be adaptive to user needs

Naturally, there are tradeoffs between the level of agility and flexibility to adapt to change at the “micro-level” and the level of predictability and control at the “macro-level”. It is important that both the client or business sponsor and the development team need to agree on those tradeoffs. The framework provides a mechanism for making those tradeoffs by making the “macro-level” as “thick” or “thin” as you want to fit a given situation.

  • Increasing the level of predictability and control requires beefing up the macro-level and providing more detailed requirements at that level and implementing at least a limited amount of change control
  • To increase the level of agility, you can simply eliminate the macro-level altogether or limit it to only very high-level requirements
  • Other elements of the framework can be easily customized or eliminated depending on the scope and complexity of the project and other factors

A question that often comes up is “How do you handle change control?”. The answer to that question is that you have to design enough slack into the milestones at the “macro” level to allow detailed elaboration of requirements to take place in the “micro” level. However, when there is a significant enough change in the “micro” level that would impact achievement of the requirements in the “macro” level, that should trigger a change to the “macro” level milestones. This general approach can be used on almost any project.

Check out my online training courses to learn more about developing a hybrid Agile approach and some case studies that show how it has been used successfully.

The Agile Project Management Pendulum

The original Agile movement started out as a revolution against the traditional Waterfall methodology which was viewed as very cumbersome, bureaucratic, and inflexible. The need for that revolution was absolutely correct – an Agile approach does offer many advantages where a more adaptive approach is needed; particularly in environments where the requirements are very uncertain and subject to change. However, as in many other revolutions, there’s often a tendency for The Agile Project Management Pendulum to swing too far in the opposite direction to make a correction.

Agile Project Management Pendulum

In particular, a lot of polarization has developed between some people in the Agile community and people in the more traditional project management community.

  • There are many agilists who are entrenched in their perspective that the only way to be “agile” is strictly “by the book” and that there is no need for project management at all – they see project management as a role rather than a set of principles that can be adapted to a broad range of different environments, just as the agile principles can also be applied to a broad range of different environments
  • There are some project managers who are equally entrenched in thinking that traditional, plan-driven, control-oriented approaches are the only way to do project management and have not learned how to integrate an Agile approach into their overall toolkit

The pendulum has begun to swing back towards the middle a bit and there’s less polarization today than there was several years ago, but some of that bias still does exist on both sides of this fence. Some of the progress that has been made over the past few years has been:

  • PMI has recognized the need for integrating an Agile approach with a traditional project management approach and has begun moving in that direction with the PMI-ACP (Agile Certified Practitioner) certification. Although it is a step in the right direction, it doesn’t go far enough, in my opinion. It doesn’t really address the larger question of how a project manager would go about blending together Agile and traditional plan-driven principles and practices in a real-world situation and what role a Project Manager would play in an Agile project to use this knowledge.Is the PMI-ACP certification PMI’s answer to the Agile CSM certification? That would imply that the goal of the PMI-ACP exam would be to compete with CSM and train project managers for the Scrum Master role and I don’t believe that makes sense at all. The only way it makes sense, in my opinion, is for a Project Manager to take on a higher-level role in larger projects that require blending together some traditional plan-driven and Agile principles and practices in the right proportions to fit the situation, but that role is somewhat undefined at this point and also not necessarily widely understood and accepted.

     

  • As Agile begins to be utilized for larger and more complex, enterprise-level projects; there is an increased recognition in the Agile community that an Agile development process like Scrum that works very well at the team level doesn’t necessarily scale very well without some kind of overall management framework and several different frameworks have been developed to fill this need.
    1. The Scaled Agile Framework developed by Dean Leffingwell is an example of a relatively complete approach that incorporates higher levels of project and program management as well as project portfolio management into an overall framework that is fairly Agile from top-to-bottom; however, it is not easy to implement and would typically require a very major transformation for a company to adopt that kind of approach.
    2. For companies who want to integrate an Agile development approach at the team level into a more traditional management framework, the Managed Agile Development approach defined in my latest book and Scott Ambler’s Disciplined Agile Delivery framework are both alternatives that can be used in a more traditional management environment.

It’s time to get past the polarization that has existed in the past and begin to see Agile and plan-driven approaches as complementary to each other, rather than competitive. It’s not an “either-or”, “black-and-white” alternative to adopt an Agile or Waterfall approach as some people have portrayed it; it’s more of a continuous spectrum of alternatives offering different levels of control and adaptivity as needed to fit a given situation. That’s the challenge I’ve tried to take on in my two books on this subject.