Waterfall or Agile for the Modern World?

10.07.2018
10 min.
title

Has everyone forgotten about Waterfall? This methodology once ruled the world of software development, back when teams were stable for years, if not decades. Now, when teams are changing continuously, it seems to become obsolete and neglected.

The need for flexibility is addressed in the Agile methodology. Since this concept was first articulated in 2001, it has turned into the default approach to software development. In its 9th Global Project Management Survey, PMI calculated that 71% of organizations use Agile on their projects sometimes, often or always.

How often organizations manage projects with Agile

After the rigidity of Waterfall, the Agile development model came as a blessing. It shifted the focus from processes to teams, from time-consuming documentation to the working product, from contract negotiation to cooperation. And of course, its drastic difference was in welcoming change and uncertainty, which allowed software developers to incorporate tweaks to their products at any stage of the project cycle.

These overwhelming differences between the two approaches to software development divided the IT world into two camps of those advocating the traditional plan-driven Waterfall and those hailing the quirky contemporary Agile. Both camps have their points, but as we’ll see further, neither methodology escaped its drawbacks. So let’s take a cold-eyed look at how Waterfall and Agile determine the success of IT projects.

Why Change Matters

With its extreme focus on ongoing changes to the working product, Agile is an easy target. This often makes the model’s antagonists conclude that Agile-led developers just don’t know what they’re doing and prefer to figure out everything as they go in an endless loop of trial and error.

In fact, ever-changing project requirements can spring from reality checks. First, there can be disparities between the software sponsor’s vision and the actual deliverable. Second, testing the limits of technologies is likely to require amendments to the architecture or other software components. Third, playing with the product design takes more than one review and feedback session. It’s also important to consider fluctuating market conditions, especially when we talk about products for sale, like SaaS and mobile games, for example.

For all these reasons, just sticking to the initial plan may not always work. There’s a long way from an idea to a finished product, and changes en route are inevitable.

Unlike Agile, the Waterfall software development model requires that all the project stages—requirements gathering, design, implementation, verification, maintenance—don’t overlap, strictly following one another. In such a structure, any changes to a completed solution come at a price, since any reworks will affect the entire system. To avoid these massive losses in time and money, Waterfall-led teams rather prefer to minimize change, sometimes at the expense of end users’ satisfaction or the overall operability.

That said, the approach to change is a major stumbling block when you try to compare the two methodologies, yet there are some other predicaments. Now it’s time to dive deeper and see how both Waterfall and Agile deal with critical project success factors.

The Rise and Fall of Waterfall

Of the two methodologies, Waterfall is the older. It originated in the manufacturing and construction industries where sequencing of actions is the most effective way to achieve a result. It was in the 1970s that the methodology was formalized for software development practices, and then adopted by the United States Department of Defense as the model of choice for working with IT contractors.

With its roots in predictable and regulated domains such as manufacturing and public sector, Waterfall shows its efficiency where there’s a high degree of upfront clarity about requirements and scope. In other words, where stability reigns, waterfall works wonders.

Obviously, not all industries are affected by uncertainty—some global industries don’t depend on fluctuations in consumers’ tastes and operate on predictable markets. Examples include such industries as oil and gas, mining, space, etc. In projects for such industries, there is more time to understand what is needed, and the changes wouldn’t be major. The project of sending a satellite off to space, for example, will have the same goals in two years as it does now.

Waterfall is also traditionally perceived as a methodology for strictly regulated industries like government and healthcare. However, even within these industries needs vary.

In healthcare, for example, the project of creating databases for medical clinics will most definitely have stable requirements because the industry doesn’t change in essence. In that case, Waterfall fits the mold. But if a medical institution were to develop a mobile app for its clients to look up services and book appointments, Agile would be the path for the developers to take. The nature of the product can therefore affect the choice of methodology.

Waterfall is a straightforward methodology where predictability is its key attribute, but sometimes even that can go wrong. One of the Australian government-funded IT projects was terminated in 2017 due to a failure to deliver the product—the gov.au website for government agencies—in line with the users’ requirements. The project methodology was Waterfall, which apparently accounted for the team’s inability to incorporate user feedback into the product.

As a result, the responsibility for further government projects was passed to the Digital Transformation Agency (DTA). In the wake of the failure, the agency’s interim Nerida O'Loughlin revealed their plans to replace “old-school” Waterfall with more Agile practices: “Is that the best way to deliver these projects? Are we not better to chunk it down, put smaller parcels out to market, prototype, test—very much aligned with the agile methodologies?”

There are other reasons, though, why Waterfall is no longer favored like before.

Why Waterfall is Eclipsed Now

One of the reasons why Waterfall was overshadowed by Agile is the dynamic nature of modern IT teams. In the past, they were stable for many years, which is perfect for Waterfall. Also, there were fewer software development projects and fewer programmers in the world in general. Teams were not yet dispersed—often their locations were limited to the same room—which allowed for easy face-to-face communication.

All that made the scope of work much more manageable, and Waterfall thrived in those conditions. But times have changed.

The global trend of employee mobility makes projects less predictable. International teams are the new norm, while employees move increasingly between their teams from one project to another. In addition to that, people change jobs quickly and freely; they are less tied down by social benefits and pension funds so they move from company to company, or go freelancing.

So now typically projects are carried out by teams not strongly experienced in working together because team members change all the time. Getting into a Waterfall-led team would mean to disturb a rigid system with pre-defined roles and responsibilities, where knowledge transfer would eat up project time not planned upfront. So when it comes to project assessment in Waterfall, it’s getting difficult to make correct predictions taking into account such volatile conditions.

Another critical drawback of Waterfall is the lack of a working prototype until late in the project. So Waterfall makes it impossible for the project stakeholder to get a sense of the result in advance. This means there can be no amendments to the initial requirements and scope, which again sends us back to the major shortcoming of this methodology: corrections to the previous work are next to impossible. If some stage goes wrong, the mistake will be inherited by the succeeding stages until the testing phase kicks in.

All considered, the Waterfall software development model is still good for smaller predictable projects where each team member gets the requirements and expectations from the very beginning. In that case, it becomes easier to onboard new team members.

However, custom software development has been leaning toward Agile as the ultimate approach that came to replace Waterfall in many instances. Now, 17 years after its inception, we can reassess the effectiveness of the Agile framework to see if it proved its worth, and whether there are downsides that hinder its project management potential.

Agile: A Doubtful Alternative?

It’s the axiom that Agile welcomes change and uncertainty. Agile is more flexible than Waterfall and allows for less specific requirements while project risks are typically managed better. Therefore, longer, more complex object-oriented projects work better using Agile.

Using Agile project management in SDLC

Back when Waterfall ruled, information wasn’t traveling as fast and the markets were not that volatile as today. Now, depending on the industry, weekly and monthly disruptive changes are casual, so tweaks and alterations to the development process are inevitable. As a result, the end product can turn out to be absolutely different from what was planned. In these conditions, flexibility becomes paramount, especially for longer projects, and Agile is perfect for that.

However, the way Agile addresses the disadvantages of Waterfall has its own flipside:

Cultural Clashes and Communication Pitfalls

Being rather like a journey with a potentially unknown destination, where the route is defined as you go, Agile requires a cultural shift within teams. Not every programmer is a good fit for an Agile project as this requires a very particular mindset, so cultural clashes may happen.

That said, mutual understanding and trust between team members is a must. Communication flows are critical, as much as allocating time for daily and weekly meetings. In Agile, messages should be conveyed in a straightforward way, and preferably via face-to-face discussions, which is not always possible, especially in distributed teams.

Faster Deliverables, Lack of Documentation

Agile projects may be fast-paced indeed, but this created a common misconception that Agile projects are generally shorter than Waterfall ones. It’s not really true—it’s just that in Agile workable chunks of code are delivered much earlier in the development cycle while you’d need to wait till the final stage in the case of Waterfall. Both Agile and Waterfall projects can last for years depending on the project complexity and dynamic. Such phases as maintenance and support can be ongoing, and last as long as the software is in operation.

Yet the misconception that Agile projects are only short-term can lead to the lack of documentation produced along the way. Further into the project, this could impede knowledge transfer—especially at the handover stage when the IT vendor’s part is completed and it’s time for the software owners to take charge.

Undefined Success Criteria

The fluidity of Agile is reflected in its lack of quantitative measurements of a project’s success.

Waterfall has always been squeezed in between the metrics of time, budget, and scope, but Agile is less definitive about all the three parameters, as well as doesn’t provide any benchmarks except for how workable a product release is.

It’s also difficult to decide when an Agile project is really over but by relying on subjective perception. Potentially, iterations can be endless, and there’s always enough room for perfection. Could it be otherwise with so many changes creeping in all the time?

Call for Optimization

This makes it clear that Agile as we know it calls for augmentation or, rather, optimization.

Agile appeared as a reaction to the drawbacks of Waterfall. But over 15 years later after the compiling of the Agile Manifesto, the ideas of the limited group of its founders are bound to fall under scrutiny as they are tested by hundreds, if not thousands of teams worldwide. Some CIOs even went on to call Agile a passing fad that will go out of fashion as software development teams recognize the framework’s inefficiency.

Of course, there’s little ground for such loud statements. But some extra effort is required indeed. First, to set up effective communication models for distributed teams. Second, to instill a sense of responsibility for the project success in each of the team members. This way, Agile-led teams will be able to optimize their workflows and cooperation patterns—something that the Agile Manifesto originally implied.

Hybrid Approach as a Compromise

The same PMI survey cited above reports that hybrid approaches—a blend of Waterfall and Agile—are applied in 20% of cases at surveyed organizations. The graph below shows that it’s just 1% fewer than ‘pure’ Agile.

Such hybrid approaches come up to compensate for the drawbacks of both development models. Inevitably, classic Waterfall projects are starting to incorporate more flexibility, while Agile-led teams are working to provide more clarity and predictability for project stakeholders. In both cases, we get a methodological blend that helps to reduce managerial anxiety that comes from low predictability of a project outcome.

One good example of such a combination is suggested by Deloitte consultants. They advocate the use of simulations or visual prototypes in Waterfall projects, especially those for government. Delivered and shared early in the design phase, such prototypes work well for checking if all the project stakeholders are on the same page. They help to identify possible mishaps or misunderstanding of the requirements upfront and avoid expensive reworks if something goes awry.

So, Agile or Waterfall?

Now when both methodologies hit their maturity it’s clear that neither Agile nor Waterfall is perfect.

To make them work, project managers need to identify their priorities first to choose the right option. Agile is ideal for volatile environments where classic criteria of fixed time, scope, and budget are not applicable. At the same time, it is clear that Waterfall is here to stay, serving well for projects with a high degree of predictability and independence from market changes.

Managers also need to recognize that flaws don’t usually come from a pure choice of a methodology but rather from how the team applies its principles in practice.

That’s why it’s reasonable to look at the team’s experience with the chosen methodology, at how team members fit together, whether they worked with each before and have good “chemistry.” This may help to ensure that project management overcomes potential pitfalls that result from any methodological limitations.