The Value of Estimation in an Agile Project

I participated in a discussion recently on the subject of estimation in an Agile Project – the individual who started the discussion indicated that his team was not very good at estimating and asked whether it was important for them to become more proficient in estimating the level of effort required. I think it is very important and it is a skill that is often neglected in Agile development projects. Here’s why I think it is important:

  • At a project level, there is a need for some kind of planning to estimate the scope of the effort and to set expectations of how long it is going to take to finish the project. Very few projects are given a “blank check” to get something done without some kind of expectations associated with the cost and schedule of the project and it’s irresponsible to not set and manage those expectations. I have seen Agile projects where the project has gone on-and-on for an extended period of time without a plan for when it would finish; and, in one case, the project was so large that it couldn’t possibly be done by a single Agile team and that wasn’t discovered until well into the project when the project had to be re-planned and estimated.
  • At a more tactical level within a project, there is an ongoing need for the Product Owner to evaluate the value produced by stories against the level of effort required to develop that capability to ensure that the work is being prioritized properly to maximize the value the project produces. If you only know the business value to be produced without an idea of the level of effort associated with it, it is very difficult to make a good decision about that.
  • There is also a need to accurately size the level of effort that can be taken into a sprint so that it can be completed successfully. It can be demoralizing for a team to never finish a sprint successfully because they weren’t able to accurately estimate the level of effort required.

There is no question that estimation is a difficult thing to do in an Agile environment, but the importance of doing estimation is not well-understood and developers sometimes resist making estimates. The important thing is to define the approach for doing estimation in the context of the project you’re operating in. Some projects may have very uncertain requirements and may be very difficult to estimate and that may be considered OK but that doesn’t have to be the norm for all Agile projects. It is not an all-or-nothing decision to be totally adaptive with no plan or estimates at all or to be rigidly plan-driven. There are plenty of alternatives between those two extremes.

What Are the Symptoms of Methodology Myopia?

Have you ever met someone who has “Methodology Myopia”? The symptoms of this problem are that the person thinks that there is one particular methodology (whatever it might be – Agile, Waterfall, or something else) that is a universal solution for any kind of project you might have. This “one size fits all” approach many times results in people force-fitting a project to a methodology whether it fits or not and practicing that methodology rigidly without necessarily tailoring it to fit the situation. I believe that a better approach is to fit the methodology to the project and sometimes that might require blending elements of different methodologies together to fit the situation.

Why does this problem exist?

  • Sometimes people only know one methodology. For example, for many years, project managers were heavily schooled in the Waterfall approach and there was no need to learn any other methodology.
  • It takes a lot more skill to know how to tailor a methodology and/or mix-and-match elements of different methodologies to fit a situation. It requires knowledge of a broader range of methodologies and a deeper understanding of the principles behind the methodology rather than just the mechanics of how it is implemented.
  • Sometimes people get so over-zealous about a particular methodology that they lose their objectivity and forget that any methodology has limitations and needs to be applied intelligently and tailored as necessary to fit a situation. For example, there are some agilists who are so zealous about Scrum that they completely reject any kind of more traditional plan-driven approach as obsolete and no longer relevant.
  • Consultants often fan the flames of this fire by building the frenzy about a particular methodology or approach being the best thing possible because they are selling their services based on a particular approach or methodology.

In my book, I use the analogy of a Project Manager as a “Cook” versus a Project Manager as a “Chef” (credit for this original idea is due to Bob Wysocki). A “Cook” knows how to prepare a limited number of recipes and do it by the book – a “Chef” knows how to prepare a much broader range of recipes with different and more exotic ingredients and also knows how to create entirely new recipes when required. This is a major challenge for the Project Management profession today…we need more Project Managers who are “Chefs”, who know how fit a methodology or combination of methodologies to a given situation. Many times this will require learning how to blend seemingly disparate approaches like Agile and plan-driven methodologies in the right proportions to fit the situation. Any Project Manager who only knows how to do a traditional plan-driven project management approach based on Waterfall may be severely limited in their career options in the future.

This is a difficult challenge to take on because I don’t believe anyone has all the answers of how to go about blending Agile and traditional plan-driven principles and practices together in the right proportions to fit a given situation. That takes a lot of skill, there is no “canned” solution, and the knowledge of how to do that as well as what works and what doesn’t work in different environments is constantly evolving at this time. However, even though this is a difficult challenge to take on, it’s important to recognize that this problem does exists and begin to take action to resolve it.

For many Project Managers, this requires developing a very broad and deep knowledge of both Agile and traditional plan-driven project management approaches as well as some actual experience in to blending them together that goes well beyond what is covered in the PMI-ACP certification. And, for both Project Managers and agilists, it requires getting past many of the stereotypes, myths, and misconceptions that have polarized these two communities to begin to see how these seemingly disparate approaches can be complementary to each other rather than competitive with each other.

The Agile Project Management Pendulum

The original Agile movement started out as a revolution against the traditional Waterfall methodology which was viewed as very cumbersome, bureaucratic, and inflexible. The need for that revolution was absolutely correct – an Agile approach does offer many advantages where a more adaptive approach is needed; particularly in environments where the requirements are very uncertain and subject to change. However, as in many other revolutions, there’s often a tendency for The Agile Project Management Pendulum to swing too far in the opposite direction to make a correction.

Agile Project Management Pendulum

In particular, a lot of polarization has developed between some people in the Agile community and people in the more traditional project management community.

  • There are many agilists who are entrenched in their perspective that the only way to be “agile” is strictly “by the book” and that there is no need for project management at all – they see project management as a role rather than a set of principles that can be adapted to a broad range of different environments, just as the agile principles can also be applied to a broad range of different environments
  • There are some project managers who are equally entrenched in thinking that traditional, plan-driven, control-oriented approaches are the only way to do project management and have not learned how to integrate an Agile approach into their overall toolkit

The pendulum has begun to swing back towards the middle a bit and there’s less polarization today than there was several years ago, but some of that bias still does exist on both sides of this fence. Some of the progress that has been made over the past few years has been:

  • PMI has recognized the need for integrating an Agile approach with a traditional project management approach and has begun moving in that direction with the PMI-ACP (Agile Certified Practitioner) certification. Although it is a step in the right direction, it doesn’t go far enough, in my opinion. It doesn’t really address the larger question of how a project manager would go about blending together Agile and traditional plan-driven principles and practices in a real-world situation and what role a Project Manager would play in an Agile project to use this knowledge.Is the PMI-ACP certification PMI’s answer to the Agile CSM certification? That would imply that the goal of the PMI-ACP exam would be to compete with CSM and train project managers for the Scrum Master role and I don’t believe that makes sense at all. The only way it makes sense, in my opinion, is for a Project Manager to take on a higher-level role in larger projects that require blending together some traditional plan-driven and Agile principles and practices in the right proportions to fit the situation, but that role is somewhat undefined at this point and also not necessarily widely understood and accepted.

     

  • As Agile begins to be utilized for larger and more complex, enterprise-level projects; there is an increased recognition in the Agile community that an Agile development process like Scrum that works very well at the team level doesn’t necessarily scale very well without some kind of overall management framework and several different frameworks have been developed to fill this need.
    1. The Scaled Agile Framework developed by Dean Leffingwell is an example of a relatively complete approach that incorporates higher levels of project and program management as well as project portfolio management into an overall framework that is fairly Agile from top-to-bottom; however, it is not easy to implement and would typically require a very major transformation for a company to adopt that kind of approach.
    2. For companies who want to integrate an Agile development approach at the team level into a more traditional management framework, the Managed Agile Development approach defined in my latest book and Scott Ambler’s Disciplined Agile Delivery framework are both alternatives that can be used in a more traditional management environment.

It’s time to get past the polarization that has existed in the past and begin to see Agile and plan-driven approaches as complementary to each other, rather than competitive. It’s not an “either-or”, “black-and-white” alternative to adopt an Agile or Waterfall approach as some people have portrayed it; it’s more of a continuous spectrum of alternatives offering different levels of control and adaptivity as needed to fit a given situation. That’s the challenge I’ve tried to take on in my two books on this subject.

Agile Product Backlog Grooming “Iceberg”

An understanding of the Agile Product Backlog Grooming “Iceberg” is important. A typical Agile approach of Product Backlog grooming is sometimes focused primarily on limiting the backlog to only about 2-3 sprints ahead and doing grooming only as needed to get stories ready for development. That is a big difference between a “pure” Agile approach and more of a hybrid approach that blends an Agile development approach within a somewhat plan-driven envelope. If your project can be totally adaptive and doesn’t require a plan, you can probably live with that approach and completely define the Product Backlog “on the fly” as the project progresses. However, in many situations (particularly in large enterprise-level projects), there is a need to have a plan (either at the release level or project level) for the project. It’s very difficult, if not impossible, to limit the backlog too much and also be able to have some kind of plan of where the project is going.

For that kind of environment, I like the idea of thinking of the Product Backlog as an “iceberg” as shown in the diagram below (I believe this iceberg Idea was originally created by Mike Cohn):

Product Backlot Grooming Iceberg

The “iceberg” idea is that as you pull items off of the top of the iceberg, other stories that are at lower levels in the iceberg move up to take their place. The process of developing stories and moving them up the iceberg goes on continuously to feed the development process. A Kanban process with different stages of definition for stories fits this process very well. (Thank you to Michael Hurst for suggesting that) Here are a couple of suggested guidelines for this process:

  • You never want the development team waiting for stories to go into development. The number of stories that are fully groomed and ready to go into development should always be far enough in advance to keep the queue from going empty and to prevent developers becoming idle waiting for stories to develop.
  • There are logical stages that the grooming process for the rest of the backlog goes through and those stages should align with whatever planning objectives you need to meet. For example, if there is a need for release-level planning or project-level planning to forecast a plan and an estimated schedule, the Backlog needs to be sufficiently-defined and defined far-enough in advance to support that objective.
  • Product Backlog grooming can be a very time-consuming process and the time for grooming of what’s coming next needs to be built into the current sprint or it may not get done in time to keep up with the development effort.
  • Using people on the team efficiently to do backlog grooming can be a challenge. The developers who are going to take responsibility for the development of the stories need to be involved in the final stages of grooming prior to taking the stories into development so that there is a common understanding of the intent for the stories. However, it might be a significant drain on the team if the entire development team became deeply engaged in the very early stages of backlog grooming – a large portion of that effort can be done by the Product Owner and/or a Business Analyst working with the Product Owner supplemented by inputs from the development team as needed.

People, Process, and Tools in an Agile Project

The inter-relationship of people, process, and tools in an Agile project is important to understand. On several occasions, I’ve been brought in to rescue a project that was in trouble. In many cases, there is an expectation that, as a Project Manager, I will re-plan the project, get it on the right track, and perhaps also”ram-rod” the effort to get it moving through to completion if necessary. That’s the typical expectation of what a traditional Project Manager might do. I think the Agile environment is different…I’ve certainly seen a need to re-plan projects and get them on the right track and it often takes some strong leadership skills to do that; however, a more critical role in many cases in an Agile project is working on developing the right people, process, and tools to make the Agile project teams more effective and self-sufficient. The irony is that if you do that successfully, you might practically eliminate the need for a Project Manager and put yourself out of a job, but that’s the right thing to do. I think that’s a key difference in an Agile project – in a traditional project, the role of the Project Manager might normally be expected to continue all the way to the end in order to push the project on to completion.

I’ve seen several Agile projects that got into trouble because they didn’t have an adequate amount of upfront planning and, as a result didn’t have the right people, process, and tools to do the project successfully – this happens frequently in larger, enterprise-level projects because many people don’t understand or appreciate what it takes to scale an Agile development approach to an enterprise level. Here are a few examples of typical problem areas I’ve seen:

  • People – It takes a lot more skill to organize a large, enterprise-level Agile project because the number and different types of people involved are typically a lot more numerous and complex. It can be a real challenge to figure out how to organize all of the people (both inside and outside of the project teams) to get the effort done successfully. For example, I’ve seen projects get in trouble because there was no Project Governance model defined to provide overall direction to the project. In many cases at an enterprise level, its not as simple as having a single Product Owner to provide direction and without a plan for how all of the stakeholders and decision-makers will participate in the project as it moves forward, it’s very easy for a project to go off-course and get in trouble.
  • Process – In many cases, there’s also not a clear understanding of what it takes to scale an Agile process like Scrum to enterprise levels. There are typically layers of management needed above the team level and it has to be well-integrated with the company’s primary management structure. It may not be as simple as layering a Scrum-of-Scrums approach on top of a couple of Agile teams. There is also often a need to define a hybrid process that blends together some amount of traditional plan-driven project management at a higher level with an Agile development approach like Scrum at the team level.
  • Tools – Tools also can be a major problem area. The tools that people many times use for small, simple Agile projects, just don’t necessarily scale well to larger, more complex, enterprise-level projects. For example, people might use physical story cards and white-boards for tracking progress of small, single-team Agile projects but those methods start to fall short very rapidly as you to try to scale the effort to larger, enterprise-level projects with multiple teams that need to coordinate with each other and may not be collocated. Managing that type of situation typically requires more sophisticated, on-line electronic tools.

I’ve been in situations where clients might not have the patience to address some of these more systemic issues involving people, process, and tools. In some cases, there might be an expectation that the Project Manager will just somehow “ram-rod” the project to get it moving and get it back on the right track but taking that kind of approach and ignoring the more systemic factors associated with people, process, and tools is not likely to be very successful. That’s equivalent to just putting pressure on a broken process to make it work better. A more effective approach in most cases is to fix the broken process.

What’s the Difference Between Anarchy and Self-Organizing Teams?

What’s the difference between anarchy and self-organizing teams? The Merriam-Webster dictionary defines “Anarchy” as:

  • a state of lawlessness or political disorder due to the absence of governmental authority
  • a utopian society of individuals who enjoy complete freedom without government
  • absence or denial of any authority or established order

Wikipedia defines “self-organization” as follows:

Self-organization is a process where some form of global order or coordination arises out of the local interactions between the components of an initially disordered system. This process is spontaneous: it is not directed or controlled by any agent or subsystem inside or outside of the system; however, the laws followed by the process and its initial conditions may have been chosen or caused by an agent.”

However, that doesn’t mean that self-organizing teams are given a blank check to do whatever they want to do without any higher-level direction and it also doesn’t mean that self-organizing teams should not be held accountable for their actions. The team needs to earn the right to be self-organizing by proving that they are responsible about making and keeping commitments – that’s an essential part of self-organization and one of the key things that distinguishes self-organization from “anarchy”. It would be irresponsible for any manager to rely on a team to be self-organizing if the team doesn’t hold themselves accountable for meeting commitments.

Reliance on self-organizing teams is a very powerful and very important concept in Agile and, if it is done correctly, it can have a huge impact on the whole company. As an example, Valpak, is one of the successful case studies in my book and one of the significant differences Agile has made on their organization is due to self-organizing teams:

  • Prior to Agile, the company had to do a fair amount of management (as most companies do) to resolve individual and team performance issues
  • After Agile, those issues almost totally went away because the teams became self-organizing. If there was someone with a performance issue on the team, the team did not tolerate it and resolved it themselves.

The successful implementation of Agile and self-organizing teams at Valpak allowed their senior management team to focus on higher-level strategic issues much more than they had ever been able to do before because they were freed from managing so many day-to-day issues.

What are the attributes of a good, self-organizing team?

  • They understand and take responsibility for fulfilling the higher-level business direction of the company that they work for but no one has to micro-manage them to get it done.
  • No one needs to pressure them into making commitments – they make commitments eagerly, voluntarily, and responsibly.
  • They’re very clear and unambiguous about the commitments that they make
  • They take commitments seriously and have the integrity and self-discipline to take accountability for their actions and follow-through to make sure that any commitments they make are consistently met

Developing a highly effective, self-organizing team is not an easy thing to do and it can take time but its a very powerful component of Agile. There is also typically a balance between letting self-organization go too far and becoming an end in itself or turning into anarchy. Most companies do not exist totally for the benefit of the workers (that’s what socialism is and we all know how successful that is) so overall business direction and leadership is still important. What is needed; however, is visionary leadership, not micro-management.

What is Agile Functional Decomposition?

What is Agile functional decomposition? (and why is it important?) Investopedia defines “functional decomposition” as follows:

“A method of business analysis that dissects a complex business process to show its individual elements. Functional decomposition is used to facilitate the understanding and management of large and/or complex processes and can be used to help solve problems…Basically, functional decomposition takes something complicated and simplifies it. The individual elements of the process and their hierarchical relationship to each other are commonly displayed in a diagram called a functional decomposition diagram.”

Why is that relevant to Agile? Agile is about maximizing the business value that a project produces. On small, simple projects, you might be able to discover what that business value is easily based on direct face-to-face discussion with the Product Owner and individual stakeholders. On large enterprise-level projects, that may not be so easy to do. As an example, I was involved in a large project with about 500 user stories and we had to do some prioritization to move things out to a future release. It’s very difficult to do that on a project of that size without understanding the relationships among the stories if they’re not organized into some kind of functional hierarchy.

Using Agile functional decomposition to organize stories into epics and themes makes it possible to keep all of the stories well-aligned with producing the higher-level business value that the project is intended to produce and makes it a lot easier to effectively manage the Product Backlog. It also provides a capability for traceability – you can look at the top-level functionality and then look down into the functional decomposition to verify that the lower-level functionality is really complete and sufficient to support the higher-level functionality.

Functional decomposition has been around for years but many people don’t think it is relevant to Agile projects – it’s one of those traditional, plan-driven practices that many people seem to assume is now obsolete and no longer relevant with Agile, but I don’t believe that is the case. You don’t have to completely throw away all the traditional plan-driven practices you may have learned in order to practice Agile and some of those practices become especially important as you begin to try to scale Agile to an enterprise level…Functional Decomposition is an example.

The Importance of Systems Thinking and Agile

Understanding the importance of systems thinking and Agile is critical in order to gain a deeper understanding of Agile. Wikipedia defines “Systems Thinking as follows:

“Systems thinking is the process of understanding how things, regarded as systems, influence one another within a whole. In nature, systems thinking examples include ecosystems in which various elements such as air, water, movement, plants, and animals work together to survive or perish. In organizations, systems consist of people, structures, and processes that work together to make an organization “healthy” or “unhealthy”.”

Why is “Systems Thinking” important? It allows you to see things in an entirely different perspective:

  • You see the “whole” rather than the “pieces” and understand their relationship. In an agile implementation you see the business as a large ecosystem and see the development process as only one component of that ecosystem and you begin to better understand how the two are interrelated to each other.
  • Within an Agile development process, you begin to better understand how all the components of that process work together to make the overall process more effective and instead of following the process rigidly and mechanically, you see it as a much more dynamic process where each component of the process may need to be adjusted to fit the situation.

“Binary Thinking” is the antithesis of Systems Thinking. Instead of seeing the real complexity that is inherent in many situations, people who engage in binary thinking are sometimes looking for a simple, cause-effect explanation for something that isn’t really very simple at all. They:

  • Tend to see the Agile values and principles in “black-and-white” terms as absolute statements rather than relative statements that need to be interpreted in the context of the situation as they were intended to be.
  • See the relationship of Agile and more traditional plan-driven approaches as “either-or”, mutually exclusive choices (Either you’re Agile or you’re not) and they may see these approaches as competitive with each other rather than seeing them as potentially complementary.

That sort of narrow thinking has led to many stereotypes, myths, and misconceptions about what Agile is and also about what traditional Project Management is. We need to rethink what Agile is as well as rethink what traditional Project Management is to see them in a new light as potentially complementary rather than competitive approaches and “systems thinking” is the key to that. It is also the key to becoming a “learning organization”. Wikipedia defines a “learning organization as follows:

“A learning organization is the term given to a company that facilitates the learning of its members and continuously transforms itself. Learning organizations develop as a result of the pressures facing modern organizations and enables them to remain competitive in the business environment. A learning organization has five main features; systems thinking, personal mastery, mental models, shared vision and team learning. The Learning organization concept was coined through the work and research of Peter Senge and his colleagues (Senge, 1994). It encourages organizations to shift to a more interconnected way of thinking.”

Adopting a “systems thinking” approach and becoming a learning organization are two of the most important aspects of enterprise-level agility.

The Importance of Agile Project Governance

Do the words “Agile Project Governance” sound like they go together? On first glance, your answer might be “No” – the idea of superimposing a Project Governance Model on an Agile project sounds like mixing oil and vinegar, but I don’t believe that is necessarily the case especially when you attempt to scale an Agile project to large, complex enterprise-level projects.

Much of the wisdom about Agile has been about how to optimize the performance of an individual Agile team. There are hundreds of books about almost every aspect of optimizing Agile team performance that you can imagine but the idea of scaling Agile to enterprise-level projects is a road that is far less traveled and requires a lot of new thinking. In many cases, it requires blending together some plan-driven principles and practices with Agile principles and practices to create an overall governance model for large, complex programs consisting of multiple teams.

The predominant thinking in many cases seems to be that you have to shift the entire culture of a corporation to adapt it to an Agile development approach. My experience has been that is not necessarily the best approach in many situations. The culture of a company needs to be built around what makes sense for that company to be most successful in whatever business environment that it operates in and that is not always well-aligned with becoming totally Agile. An Agile culture works best in companies who are focused on product innovation such as companies who produce and sell software products like Intuit Quickbooks or where some form of product innovation is closely related to their primary business goals like Amazon.com.

In companies where product innovation is not the most important factor in the success of the business, it becomes a lot more difficult to convert a company to an Agile development approach because it may not make sense to force the company to adopt a culture that is totally Agile. Consider, for example, the following case studies from my new book:

  • Harvard Pilgrim Healthcare (HPHC) – HPHC’s business success is driven largely by customer satisfaction (they have been ranked number one in healthcare industry surveys for over nine years) and operational excellence (being able to process millions of healthcare transactions efficiently at a very low cost. They were faced with a requirement to do a massive redesign of all their operational systems that involved over 100 Agile teams over a five year period. Obviously, they can’t afford any disruption of customer service in that process and there are a lot of moving parts that need to be carefully coordinated to avoid any disruption of service to their customers.
  • General Dynamics, UK is a large defense contractor in Europe. Obviously, there business has to be geared around government contracting so there are limits on how Agile they can become, yet they were able to blend together some level of traditional plan-driven project/program management practices with an Agile development approach to achieve the right balance to fit their business.
  • Valpak is a very Agile company and fast-paced product innovation is much more critical to their business success. Valpak is the company that periodically sends you those blue envelopes stuffed with coupons. They have a huge processing facility in Tampa where paper comes in one end and millions of coupons stuffed in envelopes ready for mailing come out the other end and the whole operation is very highly automated. It is a very impressive operation and Valpak has been very successful in transforming their entire business using Dean Leffingwell’s Scaled Agile Framework.

The key point is that each of these three businesses is different and trying to use a “one size fits all” approach to force each business to adapt to a “textbook” agile development model would not have been the best solution. The right approach is to go in the other direction and fit the methodology (or combination of methodologies) to fit the business environment. It requires a lot more skill to do that but fortunately some frameworks and models are evolving to make that a lot easier to do. Dean Leffingwell’s Scaled Agile Framework
is only one example. My new book, Managed Agile Development – Making Agile Work for Your Business provides a lot more guidance on this topic.

What is an Agile Business Analyst? How is the Role Different?

Many software development projects are moving rapidly to an Agile development approach which has dramatically impacted the role of a project manager in an Agile environment.  Agile also has a significant impact on the role of a Business Analyst.  In this article, we will explore “What is an Agile Business Analyst?” and “How is the role different?”.

What is an Agile Business Analyst?

Traditional Business Analyst Role

The traditional role of a Business Analyst is defined as follows:

“The business analyst’s primary objective is helping businesses implement technology solutions in a cost-effective way by determining the requirements of a project or program, and communicating them clearly to stakeholders, facilitators and partners.” Business Analyst Job Description, Villanova Univ.

That’s the predominant role that has been played by Business Analysts for many years.  In some cases, it has boiled down to simply talking to users about what their needs are and writing requirements documents and it has become a very documentation-intensive role.  However, the real value-added that a Business Analyst should provide is in helping to define innovative solutions to business problems and not just simply creating requirements documents.  In a traditional plan-driven environment, documentation often takes on a life of its own and we lose sight of the fact that the goal is to create business value and documentation is only a by-product.

What Are the Skills Needed by a Business Analyst?

It’s useful to step back and look at the skills that a good Business Analyst needs to really provide value-added beyond simply writing requirements documents.   A Business Analyst has to have an understanding of:

  1. The Business Domain that they work in (finance, healthcare, manufacturing, or whatever else it might be). That means that the BA understands how that business operates, what the major business processes are, the metrics that are important to the business, and how to improve the processes from a business perspective
  2. Process Improvement Tools and Techniques – They need to understand how to use process improvement tools and techniques such as Six Sigma and others to analyze an existing business process and identify and define opportunities for improvement
  3. An IT Solutions perspective – they need to know how to translate that understanding of the business process improvement needs into IT solution requirements to create effective and innovative business solutions.
  4. A Project Process Understanding – Finally, they need to understand how an overall project development process works and how their role fits into that process so that they can work most effectively as a member of a development team.

These fundamental skills have not changed over the years and a good Business Analyst is strong in all of these areas; however, in some cases the role of a BA has degenerated into simply writing requirements documents.

What Is Wrong With the Traditional Business Analyst Role?

The problems with the Traditional Business Analyst role are essentially the same as a Traditional Project Management role.

  • Level of Uncertainty – In today’s environment, there is a much higher level of uncertainty associated with defining software solutions.  The traditional approach of documenting detailed requirements for projects prior to the start of the project just doesn’t work well with high levels of uncertainty.  You would likely wind up wasting a lot of time trying to define detailed requirements upfront prior to the start of the project based on speculation and those requirements might then go through an enormous amount of change as the project is in progress.
  • Emphasis on Creativity and Innovation – In the past, the primary emphasis in many projects has been on planning and control to achieve predictability of project costs and schedules.  Competitive pressures today have created a need to put more emphasis on creativity and innovation to maximize the business value of the solution.  An over-emphasis on planning and control can destroy creativity and innovation.  Can you imagine trying to develop the next generation of a very innovative product like a new iPhone using a traditional, plan-driven approach based on developing detailed requirements for the product upfront?

How Do Agile Requirements Work?

To understand the role of a Business Analyst in an Agile environment, we first need to understand how an Agile requirements process works.  There are a lot of misconceptions about Agile – some people may think that there are no documented requirements and the project is started by the developers writing code.   That is not the case.  The approach for developing requirements in an Agile project should be directly related to the level of uncertainty in the project:

  • In a project with a high level of uncertainty, it would be foolish to try to develop detailed requirements prior to the start of the project and the requirements might be limited to an overall vision and high-level requirements that the project must meet.  More detailed requirements would be further elaborated as the project is in progress based on direct feedback and inputs from the users.
  • If there is a lower level of uncertainty, the project might go further towards developing more defined requirements particularly if there is a need for some level of predictability of the project costs and schedule.  However, an Agile environment would still emphasize some level of flexibility and adaptivity to maximize the business value of the solution as the project is in progress.

There are several key differences in an Agile approach:

  1. Less Emphasis on Documentation – Agile requirements are considerably simplified and are typically written in terms of “user stories”.  A “user story” is a brief and very succinct definition of a business need without much detail about how it will be implemented,  A “user story” is considered to be a “placeholder for conversation” and it is understood that further definition of how it will be implemented will be determined based on direct, face-to-face communications with the users as the user stories are being implemented.
  2. Changes Are Encouraged – An Agile approach is designed around flexibility and adaptivity to maximize the value of the solution.  For that reason, changes to requirements as the project is in progress are encouraged.
  3. Direct Communication – There is a lot more direct communication between the business users and the project team.  First, there is a Product Owner who represents the business who directly participates in the project to provide overall business direction and priorities.  Second, the developers on the project team are expected to communicate directly with the business users to further elaborate and define requirements.

In this kind of environment, I think it should be obvious that the role of a Business Analyst should be more than just a “middle-man” to talk to the users and document requirements.  That would inhibit direct communications with the project team and probably also limit the flexibility and adaptivity that is so important to an Agile approach.

What is an Agile Business Analyst?

The role of a Business Analyst in an Agile environment is not well-defined just as there is no defined role for a Project Manager at a team-level in an Agile environment.  On small, simple Agile projects there may not be a need for either of these two roles but that is frequently not the case on large, complex enterprise-level projects.

The role of a BA is often neglected – it is assumed that the Product Owner plays that role but it can be difficult for a Product Owner to perform that role without some assistance on very large complex projects. I recently had to create a job description for hiring a BA on an agile project – here’s what I came up with as the primary skills needed:

  1. Analyzing a broadly-defined area and using functional decomposition to define high-level epics and stories to create a well-organized, value-driven framework to provide the required business value in the Product Backlog.  If the stories and epics follow a logical functional hierarchy it provides a mechanism for better understanding the relationship of the stories and epics to each other and for satisfying overall business goals.
  2. Writing individual stories that are very clear and concise and are easy to understand and implement by the development team.Writing effective user stories is a skill that is often taken for granted. What is often overlooked in good stories is the “why” or the “so that” clause that expresses the business value the story is intended to provide. A good BA can provide that perspective that is difficult for a developer to provide.
  3. Identifying related user stories and epics, grouping them into themes as necessary and documenting the interrelationship and associated business process flows as necessary.The interrelationship of user stories and epics should be well-understood and the implementation of stories across different functional areas may require some planning and coordination so that they are consistently implemented. This overall framework can provide a mechanism for easily identifying any inconsistencies and/or missing functionality.
  4. On large projects, there may be a need to integrate the needs of related projects as well as the needs of a number of different stakeholders to produce an overall solution.

The key point is that if a Business Analyst is involved at all in an Agile project, he/she should add value in some way and should not be just an intermediary between the development team and the business users and stakeholders.

Blending Agile and Traditional Project Management