Book Review: Implementing Lean Software Development

Last week, I invited the readers of shmula to pose questions to Mary Poppendieck, the author of Lean Software Development: An Agile Toolkit for Software Development Managers (Paperback), which won the Software Development Productivity Award in 2004 and, the sequel Implementing Lean Software Development: From Concept to Cash (Paperback) which will be available in early September 2006. For this interview, 12 Questions were submitted and Mary was gracious enough to answer them — the reader’s Questions and Mary’s responses are below.

 

1. Joe Spooner said, August 21, 2006 @ 1:58 pm What are some of the agile development success stories Mary has seen in government or higher education?

I haven’t done a lot of work with governmental organizations and none with higher education. But I did have one great experience with a defense contractor. The group had a very interesting project going, but every month they had to report to the general in charge how they were doing on something like 63 process measurements. They found this frustrating and so did the general so they wanted to know how to improve their measurements. I asked what does the general really want you to accomplish in the end? Find that out, and then figure out how to constantly test your current capability to do it. Show your progress to the general at the beginning of every report, and you’ll get his attention. Some months later I learned that the team had established a single, high-level performance measure, and then they focused like a laser on making sure that the software improves that measure every month. The general follows the team’s progress with great interest, the team is completely engaged, and the program is considered an outstanding success.

2. Mishkin Berteig said, August 21, 2006 @ 2:25 pm Mary, based on your experience with lean environments and your experience with agile environments, what do you think is the most important improvement or change to be made to the Scrum methodology to make it more lean?

Scrum should not be considered a static methodology, it should follow its own advice (inspect and adapt) and evolve over time. One way to make sure this happens is to keep up with what Scrum’s inventor, Jeff Sutherland, is doing with it today. It is important that Scrum teams focus on the whole product, not just developing software. At Jeff’s company, this is accomplished by defining the end of a sprint as successful live deployment at multiple customers’ production sites. The whole development team is engaged not in making the product owner happy, but in getting all targeted customer sites to go live on time. This means sending out release candidates early and happily accepting and adapting to the surprises they uncover. It makes the customers’ users and support people as much a part of the team as the developers.

According to Jeff, ScrumMasters must be true leaders who help teams self-organize to meet commitments. The team must focus on and adapt to customer needs dynamically, as part of every sprint. Architecture must support incremental development. Disciplined development and deployment practices must be in place. Product managers must have an accurate, up-to-date assessment of what is possible, and may commit only to what can be done within the team’s proven capacity. Finally, senior management and company culture must be fully engaged in supporting this way of working.

3. Jeremy B said, August 21, 2006 @ 2:31 pm What is your favorite experience about agile and lean development to interest people?

Agile development is a good description of how I developed process control software in the early 1980’s. I worked as a junior member of a very experienced and competent engineering team that built new plants and put in processes to make roll-based products such as adhesive tape, magnetic tape, graphics products, anything you could make by spreading some sort of soup on a film. This organization’s job was to build new plants and design and install new equipment every job was completely different than anything that had ever been done before. Computer controls were just coming in at the time, and I was the new kid who knew how to program minicomputers. I learned how to do disciplined, adaptive development as I watched these experts design a new plant and its new equipment and get it in shape to make a completely new product. Although there was no talk about process, the organization could repeatedly and reliably design and install a top-notch new manufacturing line in about a year.

4. TheBizofKnowledge said, August 21, 2006 @ 3:09 pm In your opinion, what is the best way for a company to switch from plan-driven projects to an Agile/Lean approach? Is agile/lean the best environment for all projects, or are there some that might work better if handled in a more traditional way?

I don’t like the term plan-driven’ when it is used as an antonym for Agile, because Agile is as plan-driven’ as any other approach; I would use the term forecast-driven’ instead. Agile is feedback-driven, while non-agile approaches tend to be driven by forecasts of the future. In domains where these forecasts are likely to be accurate, you can assume that the forecast is fact and devise a plan based on that assumption. Just remember when variances occur that they are as likely to be caused by faulty assumptions as faulty execution.

When a forecast is a mere guess, it is far better to use an approach which adapts to the future as it unfolds. In manufacturing feed-back-driven (pull) approaches produce better results than forecast-driven (push) approaches in almost all cases. I would speculate that the same is true of development, although I don’t believe that any single approach can fit all situations.

To return to your first question, the first step in moving from forecast-driven projects to feedback-driven agile/lean methods is to change the measurements. The book Rebirth of American Industry by William Waddell and Norman Bodek makes a good case that the measurements imposed by traditional cost-accounting methods are the biggest impediment to the successful implementation of lean manufacturing. Similarly, I believe that the measurements imposed by traditional project management methods are the biggest impediment to the successful implementation of lean development. In particular, instead of measuring variation from plan, we need to start measuring the delivery of realized business value.

If a company wants to move from a forecast-driven development to a feedback-driven development, it needs to shorten the cycle time from customer request to software delivery, or from concept to cash. The shorter your delivery cycle time, the more responsive you can be to feedback. Queuing theory says that short delivery cycle times depend on having very short queues of work-in-process, so a good approach for switching to agile is to take a hard look at how much partially done work you have in your system.

Start by looking for churn (rework): if you have test-and-fix churn, you are testing too late. If you have requirements churn, you are specifying requirements too soon. Next look at the defect list: test-driven development finds and fixes defects before they need to go on a list. Finally, look queues of work between departments: a cross-functional team can develop an increment of value from idea to deployment without using an inter-departmental list or queue.

5. Carlos Miranda said, August 21, 2006 @ 3:19 pm Which are the differences between your two books on Lean Software Development?

Our first book, Lean Software Development, is aimed at people who do not understand why Agile development is a good approach. It provides the underlying justification for rapid, incremental development. The second book, Implementing Lean Software Development, makes the assumption that the reader has bought-into agile development, and wants to figure out what to do next.

The second book is filled with things we have learned over the last few years: including answers to questions we have often heard and (anonymous) stories of enlightening situations that we have encountered. It delves more deeply into areas we have found to be increasingly important, including:

1. A Focus on the Whole Product 2. Test-driven Development 3. Respect for People

6. Horst Franzke said, August 22, 2006 @ 4:10 am I see teams entering the Lean software arena through a specific set of practices (e.g. XP), which often help them attack their most obvious problems. BUT at the same time I see many teams miss the underlying Principles, which would help them to really grow and improve in the long run. What would be your approach to help such a team with lean practices to adopt the underlying principles? Thanks!

The first thing I would do is ask these questions:

1. Is the team focused on delivering increments of real value to end customers and does everyone understand what that really means? 2. Is the team a Whole Team composed of everyone necessary to deliver value to the ultimate customer? 3. Does the team reliably and repeatedly delivers on its promises?

When you can answer yes’ to these three questions, I’ll bet you won’t feel the team is missing underlying principles. If your answer is no’, then the first order of business is to change it to yes’. But how do you do that? I observe that when teams miss the mark in the areas I just mentioned, there is often a leadership vacuum. Take a look at the team and see if it has both technical and marketing leadership. I believe that teams need a leader who understands and cares deeply about the customers and their problems. They also need a leader who understands and cares deeply about the technical integrity of the product. This leadership may reside in one or two people, or in small teams it may be distributed among several people. But teams which lack market and technical leadership tend to produce mediocre results.

7. Steve Hebert said, August 22, 2006 @ 2:37 pm In addition to the lean tools that programmers use, how do you influence lean processes outside the group (minimizing unchecked code downstream in QA and helping upstream specs arrive at the last responsible moment)? Also, what tools do you see that exist to help manage this type of process (i.e. seeing how many turns features take between development and QA, allowing all team members to triage their own lists, handle gating of large functions (release an item to QA when all needed components have been completed). Thank you

Your question seems to indicate that separate organizations exist to create specs and check code. I would recommend that these responsibilities should not reside in separate departments, but specialists in these areas should be members of a Whole Team everyone necessary to deliver value to the ultimate customer. An increment of customer value should go from specification to delivery in a very short time without spending time in interdepartmental queues. I would not measure turns of features between development and QA there should not be any such thing. In lean manufacturing, the job of QA is to mistake-proof processes so that it is impossible to produce defective material.

In lean software development, the job of QA is to create a test-driven environment that makes defects virtually impossible. Thus QA precedes development, it does not follow it. The idea is to go from feature request to production deployment in a very short cycle time. Jeff Sutherland’s company, PatientKeeper, has three cycle times: They deliver any maintenance fix they choose to implement in a week. They deliver any new feature they choose to implement in a month. They deliver any new application they choose to implement in three months. By deliver’ I mean they go live into production at customer sites customers who are using their software to store medical records. Each year for the last three years, the company has gone live with about 45 releases a year, never missing a deadline. PatientKeeper uses one tool to accomplish this fete it is a tool that lists in detail what remains to be done for every pending release, and how long that work is expected to take. The development team is never loaded beyond its capacity to deliver, and the management team at the highest level of the company adjusts release expectations every week to be sure that teams can meet their deadlines.

8. Vivian said, August 24, 2006 @ 3:11 am I am planning to launch Lean Concepts and Tools for process improvements. I would like to first introduce Lean thinking and Concepts and some time down the line introduce basic tools. My company is a part of the Outsourcing Industry. Could you show me a roadmap that’ll make the launch successful?

There is no roadmap that will guarantee lean success, but we do have a roadmap at the end of our new book that might help get you started in the right direction:

1. Begin where you are: How do you create value and make a profit?

2. Find your biggest constraint: What is the biggest problem limiting your ability to create value and make a profit?

3. Envision your biggest threat: What is the biggest threat to your ability to continue creating value and making a profit over the long term?

4. Evaluate your culture: Establish and reinforce a culture of deep respect for front line workers. Remove barriers that get in the way of pride in workmanship.

5. Train: Train team leads, supervisors and managers how to lead, how to teach, and how to help workers use a disciplined approach to improving work processes.

6. Solve the Biggest Problem: Turn the biggest constraint over to work teams. Expect many quick experiments that will eventually uncover a path to a solution.

7. Remove Accommodations: Uncover the rules that made it possible to live with the constraint. Decide what the new rules should be.

8. Measure: See if end-to-end cycle time, true profitability and real customer satisfaction have improved.

9. Implement: Adopt changes supported by results.

10. Repeat the Cycle: With the biggest problem addressed, something else will become your biggest problem. Find it and repeat the cycle.

9. Jon VanSweden said, August 24, 2006 @ 5:34 am Mary, I am looking to benchmark a company that has applied Hiejunka principles within the office. Have you applied this lean tool in your organization?

An office organization is very different than a development organization, and I focus on lean in development, so I would not be able to recommend a company for you to benchmark. Heijunka in an operating environment means producing at takt time. In a development organization, heijunka means establishing a cadence that keeps development moving forward at an even pace. In software development, the cadence is established through regular, short iterations and regular, closely spaced releases. Just as in bicycling, a steady, relatively fast cadence optimized for your situation is the best way to sustain high performance over the long term.

10. psabilla said, August 25, 2006 @ 4:58 am I’d love to see examples (and/or) suggestions of how to implement the following to software organizations: 1) Kanban, 2) 5S, 3) Visual Workplace, 4) The Big Room (Product Management, Development, QA, etc¦) ” how to best plan and coordinate between all stakeholders, 5) SMED, 6) 5 Why’s, 7) TPM. This is an Epic-sized question, I know, but I think sharing your knowledge will help all of us tremendously. Thanks so much!

An epic-sized question indeed! I’ll do just a brief comment on each one, and then refer you to my most recent book, Implementing Lean Software Development: From Concept to Cash, for more on some of them.

1. Kanban. The Kanban of Software development is the index card. Stories are written on cards, estimated and selected at the iteration planning meeting, posted in the team room, updated when the story is being worked on and when it is done.

2. 5S’s. Apply the 5s’s to the server which stores the code and associated documentation, to desktop environments used by more than one person, and to the code base. We have examples in our new book.

3. Visual Workspace. There are three aspects to a visual workspace the Kanban card (index card) which tells people what needs ot be done, the Andon, or signal that something is wrong (eg. lighting up a red light when the build breaks), and big visible charts which tell everyone how things are going. We also have a section on this in our book.

4. The Big Room, or Obeya. Take a look at the video 21st Century Jet (KTCS, Seattle) and you will see good examples of the Big Room in action during the development of the Boeing 777 in the 1990’s. This isn’t a new idea, but it certainly is a good one.

5. SMED, or Single Minute Set-up. The time it takes to test and deploy a release is the set-up time of software development. Many companies take so long to test software that releases are very far apart, and every effort is made to stuff as much as possible into each big-batch release. Some companies release to production several times a day (anti-virus software comes to mind) they have SMED figured out.

6. 5 Why’s. Root Cause Analysis for any problem is a fundamental skill that all teams need to learn. First we have to implement a Stop-the-Line mentality with test-driven development and continuous integration, so that we find and fix most defects instead of putting them on a rework list. Only then can we start using the 5 Why’s to discover the root cause of the remaining problems that occur.

7. TPM or Total Preventative Maintenance. In software development, we call this refactoring. We keep on improving the design of the code base to keep it healthy and prevent it from calcifying and becoming a jungle that can no longer be maintained. Legacy code is code that has not had regular TPM.

11. Mike Griffiths said, August 25, 2006 @ 2:25 pm When comparing lean manufacturing techniques to software engineering. Some authors map Toyota’s Set Based Concurrent Engineering to practices such as supplying multiple available time-slots options for a meeting, yet the manufacturing process is really based on parallel development and then survival of the fittest solution. This would be akin to having several developers or teams of developers creating the same components and then selecting the best version. Have you seen SW companies follow a parallel development approach and aggressive pruning of less suitable solutions? Also, what guidelines (circumstances, batch sizes, etc) would you give for concurrent engineering in SW project?

The most important time to evaluate multiple solutions is when an irreversible decision that is critical to the success of the system must be made. For example: Which language should we use? Which middleware will we standardize on? Which database will we go with? How do we structure the architecture to support the response time we need? How will we organize the main flow of the user interface? And so on. It is a good strategy to make these tough and critical decisions as late as possible, and I have seen companies develop multiple solutions in all of these areas when the decision was critical to success.

Note that developing options does not mean setting up separate teams to compete; usually it entails creating objects that are a bit more general, and maintaining multiple build-time options. For example, it is fairly easy for a code base to support multiple databases, user interaction strategies, and even middleware, with the selection made at build time. There certainly are critical choices that can’t wait until build time: language, security strategy and the core architectural strategy come to mind. When these types of choices are critical to the success of the system, it can be a good idea to thoroughly explore several approaches by actually developing and testing critical capabilities using each alternative. However, for most parts of the code and in most (but not all) domains, the best way to maintain options is to minimize the number of irreversible decisions that are necessary.

Early releases of software should not preclude future changes. Instead early releases should be as simple as possible and protected with tests, so that the code is easy to change later. The use of change-tolerant coding techniques is the quite often the best way to maintain options in software development this allows exploration of the design space sequentially, as the problem emerges. I find that embedded software and games are domains that are particularly amenable to set-based design, that is, the exploration of multiple solutions during development. This is because these types of software generally do not change once they are released, so the decisions made during development are, basically, irreversible.

12. Deborah Hartmann said, August 27, 2006 @ 8:34 am I’ve heard vaguely, several times, of something that’s beyond Agile and referred to as Lean, though I don’t really know if it is. Called a flow process, it seems to describe a short-cycle-time software production line, in which there is no backlog, or maybe just a short one. I was referred to your book but didn’t find it. I must admit, it sounds like lazy Agile to me – just do whatever comes in, don’t bother setting expectations. Any idea what I’m talking about here, where it comes from? Is this a photocopied too many times version of some valid approach?

Developing Software in a short cycle time from concept to cash is covered in some detail in our new book, Implementing Lean Software Development (release date August 29th, 2006.) Short cycle time development is hardly lazy! It is quite a challenge to be able to fill a request or develop a new feature set just as soon as the need is recognized. Short cycle time expectations are very demanding: for example, when a new virus threat is discovered, a security response team is expected to have software to defeat the threat ready to deploy within hours.

Just about every software development organization I know of has a list of work to do that is far longer than it can hope to accomplish in what customers would consider a reasonable amount of time. And rarely do these organizations have the luxury of turning off the spigot of requests coming in. So the waiting list of stuff to do gets longer and longer, and the development organization looks to be increasingly unresponsive.

Agile development solves this problem by having someone prioritize the list and then having the development team select from the top of the list the amount of work it can reasonably expect to accomplish within an iteration. But look at this practice from the point of view of the people who have their requests lower down on the list. They have no idea when their request might get filled, and in practice, most items lower down on the list will probably never get done. In a lean environment, the idea is to keep the list of work to be done as short as possible, by dealing with requests honestly at the onset and by not accepting work beyond the capacity of the team to deliver.

In the health care industry, some experiments were done with waiting lists at doctor’s offices. One clinic near us used a combination of overtime and limiting new patients to gradually shorten the typical waiting time for a doctor’s appointment from 60 days to 2 days. What they found was that there was no difference in the types of cases the doctors handled on a daily basis except for the fact that patients had only been waiting a day or two, rather than a month or two, to see the doctor. The 60 day waiting lists served no useful purpose at all.

Similarly in software development, long queues of work often serve no useful purpose and worse, they give the wrong message to customers about our intent to deal with their problems. Such lists should be pared down from years worth of work to perhaps a couple iterations worth of work. Agile development gives us visibility into the capacity of a team to deliver, lean development suggests that we do not queue up work beyond that capacity.