Agile Without Improvement

September 20, 2014

Agile development is a buzzword. Companies are going "agile" and are taking The Agile Manifesto and contorting it into something else. They take a philosophy and make it to fit their existing process, while ignoring the most fundamental part - continuous improvement.

The rise of agile…

The whole reason that people want to use agile development is they hate "waterfall" development. What is waterfall? It's a process by which you create a big specification for a project, you estimate it all out, design as much of it as you can, then you build for six months to a year until it's done.

When you finish a project, time has passed, the world changed, and what was a brilliant idea a year or two ago is now old hat. Nobody wants to buy an old hat.

So, as a project progresses, the people in charge of the product decide that there is some new feature that you have to implement. Guess what? Now your original timeline is off and your budget is busted. By adding this feature in mid-project, it's even more expensive to build because the design and specification has to change.

Imagine you are building a house that was designed to have a two car garage. After you pour the footings and the concrete driveway, you decide you want a three car garage because your neighbor has one. Great, now you have to rip everything out and put up a three car garage. That's a pretty expensive change to most projects. You could have saved $5,000 - $10,000 just by making the right decision up front.

Now, imagine you do that dozens of times over the course of a waterfall project. Just think how much money you are throwing away with the poor planning. It's absurd. It is no wonder that waterfall projects are rarely on time or on budget.

The only way to way to hit your time and budget estimates are to ignore any and all feature requests during the project. This puts you at the risk of your finished product being behind the curve.

The alternative to ignoring feature requests is to ship late, over budget, and still behind the curve. That's worse, but that's how many waterfall projects go.

As a side note, just because a team doesn't call their process waterfall, doesn't mean that it isn't. I've seen "agile" teams still running waterfall, just with stand up meetings and story points.

So, the rise of agile is also the fall of waterfall like processes which fail to adapt quickly to change.

What is agile?

Agile is not one thing, it's a philosophy that is interpreted in different ways by different teams. How you see agile changes depending on project management failures a team previously experienced.

There is a set of principles called The Agile Manifesto. It is a clever document that I believe captures a shared notion of a better software development process. Yet, on it's own it doesn't prescribe a process.

I believe they did this on purpose.

The first line of The Agile Manifesto says,

We are uncovering better ways of developing software by doing it and helping others do it.

So, at the moment of its creation, The Agile Manifesto was about finding better ways of developing software through practice and collaboration. The values that come after that are what they learned so far.

Keep that in the back of your mind because it is important to why agile has "failed" for so many teams, just like waterfall did.

The values themselves are as follows:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Understanding where to place value in agile is important. All together it's about responding quicker to change through collaboration to make working software.

Responsiveness. Collaboration. Working Software.

Those ideas sit at the center of agile.

And that should be great right? It's certainly better than the giant process that is waterfall. Let's throw all that stuff out the window right?

Well no. Processes and tools, documentation, contract negotiation, and following a plan are still key to getting working software. Just remember those ideas should not overshadow the ideas of responsiveness, collaboration (both internal and external), and working software.

When done properly, agile should be able to provide working software more often. It should give better estimates of budgets and time. And it should allow the team to collaborate with the customer to build the right solution to the problem they are solving. At least, that was the idea when they came up with this manifesto.

The fall of agile…

So, what went wrong? Why is there a continuous stream of "agile has failed us" themed blog posts that pop up on the internet every few weeks?

Part of this is at this point in the hype cycle people are realizing that agile isn't perfect. Whatever agile system they've adopted didn't solve all their problems. Meet the new boss, it's the same as the old boss. That kind of thing.

The other part is a lot of companies or teams take on a prescribed agile process like scrum and fail to adapt it to the team that they have. No process is perfect out of the box. Every team is different.

The way this works is teams take agile and throw away the parts they don't like for the sake of those parts that seem annoying or difficult. Or, teams will take and agile process like scrum and adhere to what they read in a book and never change it at all.

Both problems are the same problem. The team or the management doesn't want to change or grow. Agile fails when teams fail to change and improve.

When an agile team fails to embrace change, they just do a variation on waterfall with a shorter iteration cycle and daily stand ups.

I've seen teams who with an "agile" process still have six month or year long release cycles. Even though the two week iterations are happening, they aren't releasing working software. They keep adding features as the projects move forward. Things always take longer and are over budget.

It's a waterfall in agile's clothing.

The eventual fallout of that kind of process is predictable as it is with any failed process. There are teams that blame agile for their failures. The process sucks. Management doesn't get it. Let's create a new manifesto that is anti-agile! Let's have even less process! THIS TIME IT WILL BE DIFFERENT!!!

No it won't.

A team that doesn't want to change or improve won't. If you can't respond to change, you aren't agile.

Who needs a retrospective anyway?

The irony of those who say that scrum or kanban is a failed process is that they are both correct and incorrect. What they are saying is that those processes weren't correct for their team at the time they were implemented. They were incorrect that the process is objectively broken.

I've read many blog posts that go to great lengths to explain why their current process is broken. Apparently that is proof that the process doesn't work at all for anyone ever. What every blog post fails to mention is any notion of a retrospective, learning, or changing the process to fit their team without throwing the whole thing away.

Retrospectives are the one mechanism I've seen that allows a team to grow and improve their process. Yet, it's the most often ignored part of the process by both developers and management.

I think there are reasons for this.

  • Nobody wants to be told they are doing a bad job.
  • Nobody likes meetings.
  • Sharing your feelings about other people can be uncomfortable or difficult.
  • Telling your boss that they are screwing up the process is challenging.
  • When you have deadlines, a retrospective doesn't feel like a necessary thing to hit that deadline, so you skip it.
  • Managers don't like their people telling them what to do.
  • Workers don't want to be told what to do.
  • And so on and so forth.

People don't value the retrospective, so they just don't do it.

The idea of meeting to discover what went right, what went wrong, and what to do better seems like the simplest mechanism to learn as a team. It's so simple teams skip that step.

Without a mechanism to improve your process on a regular basis, your process will get behind and eventually it will go stale.

Lean manufacturing

In school, I studied operations management alongside computer science. One of the most important ideas in operations management is the notion of lean manufacturing as done by Toyota and other companies.

The biggest idea in lean manufacturing is kaizen, which is Japanese for continuous improvement.

It's a simple idea, but it's the idea that made Toyota the best car manufacturer in the world. Continuously making small improvements compounds your improvements over time. Small improvements are faster and cheaper than big improvements, so you get the return faster and it compounds quicker.

Which would you rather have? 1% improvement every week or 50% improvement every year? I would take 1% improvement every week.

If you improve 1% every week, over the course of the year you improve about 68% total. That's 18% overall better, and 35% more than the 50%.

Apply that to your business over the course of years and you have dramatically better returns. Apply that to a production process and you have dramatically better production.

The important word is not improvement, it's the continuous nature of improvement. For example, a company that practices kaizen can take an existing part of their manufacturing process and re-engineer it over the course of a week or two to be 75% more efficient. The next week they might do that to another part of the process. They might take that one line and improve the rest of their plants with the same process.

Each week or each month they make their process a bit better, cheaper, or faster. What started as one thing over time becomes something completely different and better, one little step at a time. Improvements compound and over time something mediocre becomes world class.

Kaizen software development

What would happen if we took a software development process and applied kaizen to it? Well, I've been a part of a team that did that and here is what happened.

Our team adopted scrum and we were incredibly consistent about retrospectives. Every two weeks the team met and discussed what worked, what didn't, and how to improve.

It was a difficult process. Sometimes feelings were hurt. Sometimes tough changes had to be made. Everybody had to let their ego go for a little bit for the betterment of the team.

In the beginning we were terrible at scrum. We didn't know how best to plan, estimate, or collaborate effectively in a two week period. We made a lot of mistakes. Every two weeks we would talk it over and try something a bit different. Sometimes our changes worked, sometimes they didn't. Over time we got more things right.

Every two weeks we got a bit better.

Over a period of a year and a half the team grew from three people to over ten people. The team added Ruby to the existing PHP based system. The team switched from Subversion to Git and the gitflow model. The team went from developing in Windows only to using Virtual Box and then Vagrant to match the production server. The team changed offices three times. People left and more people were hired. API first and mobile first development were embraced.

It was a busy time and it all happened around the time the team switched from a waterfall process to scrum.

The initial implementation of every change was not perfect or ideal at all. The one thing the team did was reset itself every two weeks to adapt to whatever change was needed.

The process at the end still had the three tent poles of scrum - planning meeting, daily stand ups, and retrospectives. Pretty much everything else was open to interpretation. Even how we did the planning, stand ups, and retrospectives changed.

I would say, and I think most people on that team would agree that it was the most remarkable team I've ever been a part of.

I credit the experience not to being on a team of highly skilled technicians, but to being on a team devoted to continuous improvement - to kaizen.

What about agile without improvement?

I've been on teams that were "agile". Some had a name for their process, some didn't. The teams had a lot of the hallmarks of what people think of agile. There were story points, time estimates, sprint intervals, daily stand ups. All that stuff.

Yet, none of those teams were as successful with agile. It never quite delivered the boost that they hoped it would. It didn't solve their problems.

None of those teams did retrospectives regularly or consistently.

The focus on agile for those teams was almost exclusively on time estimates, story points, and deadlines. There was always the hope that people would work harder and do more. Show more commitment, that kind of thing. I know the project managers always hoped people would take more ownership of the process and their work and it never happened.

Why? The process never evolved and the team never had much say in what the process was and how to make it better. Without autonomy, mastery, or purpose in their work, people aren't motivated.

When the process never changed, it started to feel like waterfall with shorter deadlines. When things don't always go well, managers ratchet up the pressure, and people become quite unhappy in their work. Eventually the team checks out and the process doesn't mean anything to anyone. It's just something the boss makes them do, like your parents telling you to brush your teeth or do your homework.

Work becomes a job. Everyone hates their job.

Yet, when retrospectives happened and people got involved in making things better the vibe was completely different.

To agile and back again

At another time I was on a team that went from not being agile to embracing the entire scrum philosophy (including retrospectives). Later it changed again to have that process thrown away and ended up in something that pretended to be agile.

Toward the end it was just the manager telling you what to do and when it needed to be done. No more planning, no more stand ups, no more retrospectives, no more team commitment. It was every man for himself. You against the deadline.

What I observed on that team is when the team was devoted to both doing and improving the process as a team, things got better. Morale was good, people were getting their work done, and in general it worked pretty well. At some point, there was a moment where the scrum process was thrown away for the sake of a deadline. Then another time it happened. Then again and again until every part of the process disappeared and what once was is no more.

It is disappointing to have gone through that. While the retrospectives were happening, life wasn't perfect, but the team was working hard to do excellent work. Once that went away, the team fell apart.

I've seen other teams where they ignored the retrospective and they were pretty much stagnant as a team. There was a feeling that they wanted to get better, they wanted to be a really high performing team, but they never got into doing retrospectives. Thus, their process stagnated and their work stagnated.

It wasn't because there was a lack of talent on those teams, but rather that they never saw the value in retrospectives. Heck, I've seen agile "experts" be part of teams that never did retrospectives. Those experts seemed surprised at their mediocre results.

Even reading the various blog posts that trash agile process or even those that seek to fix it all read like someone who has never been a part of a team that continuously improved. Instead, it's like a set of complaints that needs to be worked through with their team, but never gets resolved.

The real problem with agile is us.

You will never fix your process if you don't fix the people.

To make agile processes work, there needs to be a commitment to the principles. It starts with belief. If your team doesn't believe in constantly adapting to change and improving the process, it won't work.

People do what they believe and if they don't believe in being truly agile, they won't be. Let's go back to the first line of The Agile Manifesto:

We are uncovering better ways of developing software by doing it and helping others do it.

It's about finding better ways of developing software by building software and helping others build software. That is what agile is really about. If your team doesn't buy into that, then don't go through the facade of scrum or kanban or any other process. Don't waste your time pretending to be something that you don't believe in.

You can have a different process, you can call it what you want and you can do what you want. It doesn't matter. Just don't pretend that agile is the problem with your process if you are unwilling to change and grow.

What have you learned?

You know, learning is a funny thing. It's something that comes not just from the acquiring of knowledge, but the experience of living life. While The Agile Manifesto gives us a set of principles, it is important to see them for what they are. They are a snapshot of the collective experiences of those who wrote The Agile Manifesto at the time that they wrote it.

Through this work we have come to value:

That line is important. Through experience they learned that some things are more important than other things to creating software.

I guarantee that group has learned things since then. Have you?

The Agile Manifesto was written in 2001. It's been over a decade since then. A lot has changed since then. People have grown older and wiser. Some ideas still have merit, maybe some don't.

We should not look at The Agile Manifesto as an end point, but rather a jumping off point for our own learning and our own process. Every person and every team needs to find their own way.

The lesson to learn is not that responding to change is more important than following a plan. Or, that working software is more important than comprehensive documentation. The lesson is that you must keep learning and improving the process itself.

Only through their work did they come to value those things. What things have you come to value? What lessons have you learned? How have you grown and adapted to change?

We must continue to learn and we must continue to apply that learning to our daily work.

The success of agile

There will be teams that take a process like scrum and do terrible work and there will be teams that take scrum and do amazing things. Some will sing the praises of agile and some will hate it. The ultimate success or failure of agile is not bound to the implementation of these ideas on any one team.

The success of agile depends on our ability both individually and as teams to continuously get better at creating software as a team.

If we learn and apply, learn and apply, learn and apply, over and over again, I believe we will find success. If we stick our heads in the sand and blame the process for our failures without trying to make it better, it's our own fault.

As it turns out, agile without improvement isn't really agile at all.

About The Author

Brian Knapp is one half of the dynamic duo that runs Retro Mocha. You can follow his hilarious puns on Twitter.