Agile and Waterfall, not so much two ways to manage but more a spectrum of practice

Like many technology and business leaders today you may be considering using Agile practices within your enterprise as a way to realize productivity gains for your technology projects. The first thing you will likely ask is: what needs to change?  You might be thinking that you need to change everything over from your current practices and that may not be true.  Your creative employees might already be using a form of Agile practice today but they don’t refer to it that way.  You might believe that your project teams are using a strict Waterfall process for planning and execution of your important enterprise initiatives, but you might be surprised to find out how agile your team currently works.

Neither Waterfall nor Agile are actually management practices, they are ideals.  Waterfall assumes that one may clearly map out a sequence of tasks for a project and that all will perform in perfect steps, without deviation or change, from beginning to close; we can call this ideal Strict Waterfall. Agile assumes that following a defined plan will only hinder the delivery team because it prevents them from exploiting value added opportunities that could not have been envisioned at the start of the project; we can call this ideal Pure Agile.  Pure Agile and Strict Waterfall are not practical management practices; they are ideals; real-life management practices fall somewhere between these ideals in a kind of spectrum of practice between the two extremes.  So your team is not using Waterfall or Agile, because they are managing work in the real world and the processes they are using fall somewhere between these two ideals.  In order to execute a successful Agile transformation, you do need to understand what you are doing today and where you want to be.

It is helpful to visualize Strict Waterfall and Pure Agile as opposite sides of a spectrum of techniques that can be used to manage projects. [1]  Using what works is good project management practice, and it is Agile as well.

Figure 1 - The Agile Spectrum


Figure 1 is loosely based on an article by Allan Kelly titled “The Agile Spectrum,” it is an examination of the meaning of so called Pure Agile and Strict Waterfall and how real-world projects are actually managed.  Allan’s conclusion was that real-world projects are never managed to a strict method of Waterfall or Agile, rather there are trade-offs.  Figuring out how your teams currently manage themselves will allow you to visualize where on the spectrum your team is starting from should you be considering an Agile transition.   Knowing your baseline will help you measure where you end up and if you are ultimately successful in your transition.

Agile processes are not difficult to understand, in fact there are videos on the internet that can explain the core concepts of Agile in a few minutes. [2]  A recommended video presented by Hamid Shojaee of Axosoft, Intro to Agile Scrum in under 10 minutes [3] is a very concise and fluid description of the Scrum process (a method on the Agile side of the spectrum) and a great introduction to the method.  By the time you finish watching the video Hamid gives you an introduction to the product backlog, where it comes from and how it is organized; the organization of product features using a prioritized list and creating a release plan from this prioritized list.  It is very quick, easy to follow and only focuses on a high level description of Scrum.  The thing to keep in mind about Scrum however, is that there is a defined process to its use, it is not Pure Agile.  For example, shifting tasks in the middle of an iteration, or Sprint, is frowned upon by Scrum Practitioners.

In order to understand Waterfall, we have to go back to a time when most projects (that we thought of as projects) were huge in scope (like building a bridge).  When you are running such a monumental project it might be easy to consider the “small” changes required to deliver on milestones as incidental and not really a deviation from the plan. Winston Royce, who is actually credited with coming up with the Waterfall method [4] was not really an advocate of Waterfall. [5]  He found the method deceptive in its simplicity and fraught with risk for the development team as they struggle to catalog all of the complex requirements and bring predictability to an inherently unpredictable process.  In his paper “MANAGING THE DEVELOPMENT OF LARGE SOFTWARE SYSTEMS,” Royce explains the ideal project in terms of a sequence, or, waterfall of activities that culminate in the project close.  From this simple view of project process, he says that while this is the desirable way for a project to be managed it is risky to assume that in reality your project will execute this perfectly. His paper has a number of drawings of backtracking processes to take into account issues, design changes and error corrections.  In fact, what Royce actually describes in not Waterfall at all but something that more closely resembles the cyclical processes of Agile methods (probably because he was discussing software projects and not bridge building).

Perhaps with a different perspective of what Waterfall and Agile really are you can visualize where your enterprise practices actually fall on the spectrum and where you have to take it in an Agile Transformation.

A few years ago I worked for a company that had severe delivery issues and needed a reorganization and transformation of their project management processes.  As I worked with the delivery teams and senior management it became obvious that software teams were not focused and frequently became sidetracked on different projects in the midst of execution.  Customer service had a great deal of influence over project management with the effect that important delivery tasks were sidetracked to perform fixes over the phone, rather than scheduling fixes during appropriate project planning.  Senior management would interrupt project teams with new business that was deemed of higher importance to in-flight work.

What I found was that what my enterprise needed was the discipline of an Agile management method to get them on track to consistently deliver customer value.  In this case I did not move my Agile team farther toward the Agile side of the spectrum but more toward the Waterfall side and introduced Scrum as a method of remaining Agile while still applying some discipline to the delivery methodology.

Integrating aspects of Agile into existing process or adapting your process to be more Agile is completely acceptable as long as the changes promote value to the workflow.  Agile is not, in and of itself, a defined process but a general framework. Unlike Waterfall, it is difficult if not impossible to create an exact model of how a Pure Agile process is accomplished in a strictly procedural manner; [6] Agile is not intended to work that way.  You can however, apply rigorous procedure around an Agile team in order to contain the extent (or scope) of the development work.  Scrum is an example of an Agile methodology that applies a rigorous structure to support and focus your Agile teams.  Change is inherent in the Agile process and it may be comfortable (or in some cases a business necessity) to limit the level of adaption the team provides to the end-user (client).  For example, if the vendor and the client have a fix-price contract, with a defined Statement of Work, then it might be necessary to work with the client during execution to define not only new requirements, but how requirements from the original SOW must be adjusted in order to balance out the budget limits of the contract or perhaps summarily increase the price of the contract during project execution.

There are three main issues for senior management to consider on a move to Agile practice:

1. Risks and mitigation in new process adoption.
2. Motivation (why go Agile? What do you want to accomplish?)
3. How Agile do you want to be (on the Agile Spectrum)?

There are always inherent challenges in moving to a new way of doing things.  The results of the transition may be temporary confusion, resistance, stress or anxiety, so it is best to determine up front the end-goal of Agile adoption (i.e. motivation).  There may be good reasons for the move to Agile but it is always best to agree on what those reasons are up front so that you have some determinant way to measure success.

In the same manner as the decision to implement an Agile framework it is also important to determine what processes and methods work for your team and the wider enterprise when moving to Agile.  Just because Scrum is a popular method for managing Agile projects does not mean it is necessarily the method that will work best in your enterprise.  Agile is more of a framework than a method and many Agile principles can be used within traditional management structures and still produce added value.

Example Agile Transition
In this example the project manager transitions a project with a major bank in the 4th year of a four year project (it is essentially 75% done when the move to Agile is undertaken) and the motivation for this was to create a renewed sense of urgency with the delivery team, to increase throughput and to shorten interim releases.[7]

Risks of switching to Agile framework:

• Some team members will embrace change others do not
• Confusion around methods, roles.
• Causing stress, anxiety and resistance
• Loss of momentum, team become focused on short-term goals, more scope-creep

Lessons learned

• More upfront communication of the process; create understanding of what will happen, process and roles, reports to deliver, how they roll up to over-all summary of progress.
• Lots of coaching and change agents involved throughout
• Watch for tendency to slide back to old methods (That’s not my job mentality must be confronted)
• Use a scorecard to assess suitability of team to adopt agile, what are the gaps; apply more coaching and training
• Maintain focus on the program goals.
• And make it fun (as possible; have team time, BBQs, etc.)

In this example we see an Agile adoption undertaken while the development team continues to produce work.  The switch from Waterfall to Agile practices happened over a period of months and started as more Waterfall in practice moving gradually, as the team adopted more and more goal oriented practices, toward an Agile method.

Let’s consider the motivation for this change:  create a sense of urgency and increase throughput with shorter release cycles.  In addition, the team considered the risks of the switch, which included capacity for change, confusion of roles, loss of momentum and scope creep on projects that suddenly allow the team to make decisions on important deliverables rather than escalating to senior management.

How do we know they accomplished their goals?  Let’s look at the outcomes reported:

• Promote task ownership (moving away from: that’s not my job mentality)
• More communication and discussion, create understanding
• Focus on goals (as opposed to completing tasks)
• Coach and mentor team through change

The outcomes listed in the example fall pretty closely in line with the goals of Agile as described in the Agile Manifesto [8]:

• People over processes and tools
• Deliver value over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

So it seems that they did accomplish their goal of moving the team toward a more Agile form of management practice mid-project and that it was fairly successful.

The example transition above was accomplished within the framework of an on-going project managed in a Waterfall method and then transitioning to Agile.  They had defined their goals upfront and considered their reasons for going Agile and considered the risks of the transition.  What the example illustrates is that Agile practices can be mixed with Waterfall practices and still add value without being “strictly” Agile in nature.  The value add to your portfolio is achieved when you have found the right place along the Agile Spectrum for the management of your enterprise.

Equally as important is to consider the goals of your transformation and the risks of what you are planning to do.  Is your value add worth the risks?  You should know your risks, rewards and goals going in or even if you do manage to achieve the goal of a more Agile enterprise the whole thing might be perceived as a failure due to poor examination and communication of risk.  As always we have to manage the expectations of our stakeholders.

Similarly, if you examine what your team is doing today you may be surprised to find out how Agile your team is already.  You may think they are using a fairly strict Waterfall practice, but, like Winston Royce, they may already have considered and encountered the risks inherent in trying to manage to “strict” Waterfall and adapted their practices to that reality.  If you support them in their lessons learned during an Agile Transition you may create added value to your enterprise by adopting strategies that work in practice for your people.

------------------------------------------------------------------------

[1] Kelly, Allan, The Agile Spectrum, (c) Allan Kelly 2013 allan@allankelly.net.
[2] See Scrum 101 series by Sally Elatta, © 2011 Agile Videos All rights reserved
[3] Presented by Hamid Shojaee, Axosoft, LLC.
[4] Royce, Winston, MANAGING THE DEVELOPMENT OF LARGE SOFTWARE SYSTEMS, Reprinted from Proceedings, IEEE WESCON, August 1970, pages 1-9. Copyright © 1970 by The Institute of Electrical and Electronics Engineers Inc. Originally published by TRW.
[5] “I believe in this concept, but the implementation described above is risky and invites failure.”  [Royce, Winston, LARGE SOFTWARE SYSTEMS, p. 2.]
[6] “It is not possible to exactly define Agile Methods.” Larman, Craig, ©copyright 2004 by Pearson Education, Inc., Agile & Iterative Development: A Managers Guide. p. 25.
[7] Porfilio, Tony, speaking at PMI-SOC Monthly Meeting Sink or Swim? Changing Methodology Mid-project from Waterfall to Agile; Should You Do It? Audio, Presentation, April 24, 2014, Novotel, North York, Ontario.
[8] http://www.agilemanifesto.org/