Agile methods propose to develop software that has a high value for the customer, releasing early and at a steady pace.
A typical agile team releases a first version of the software after only a couple of weeks, incomplete but already usable. This way the customer has immediate feedback on the quality of the delivered software, and can in turn provide developers with valuable feedback, to keep them focused on what the customer really needs. If the XP team is good, it's able to maintain a steady pace of frequent releases, typically every one to two weeks.
Agile methods share some fundamental principles and values, described in the Agile Manifesto. The most common are Extreme Programming and Scrum. The XPeppers approach is based on Extreme Programming (XP), with the addition of some practices taken in by Scrum.
Early and steadily
How long does it take from an idea to some financial gain in a project? If we trace the activities involved (without agile methods) we have:
- feasibility study
Even assuming that each of these steps requires only two weeks, and there are neither breaks nor rework, we deduce that it is not possible to profit from a software development activity in less than 14 weeks. Also, in the real world, these activities usually take much longer. The problem is that these steps are performed in sequence, and there is no way to get value from the project before the final phase: when it is installed.
Another problem is that it is difficult to have a concrete measure of when the software will be ready until the development is almost completed. It is in the process of integration that we find some surprises. In the same way, almost until the end of development, we don't have a concrete software quality measure.
The answer of agile methods to these problems is incremental and evolutionary development. Software is produced with a constant rate of short iterations, of the duration of one or two weeks. Every one or two weeks, the team delivers the software containing the features implemented so far. The system is incomplete, in the sense that it does not do everything it has to do; but the present features have the final quality, in the sense that they have been implemented at the level of detail and of sufficient quality to put them into production.
Then the customer may decide, at each iteration, if the delivered features are enough to go to production. You may decide to go to production in advance, and install the missing features at a later time. This way, the project begins to produce profits before being complete.
But that's not all; at each iteration, the customer can assess the quality of the work done so far, both from the point of view of robustness and the fulfillment of the requirements. This way, we avoid the danger of delivering on time and on budget only to be told that "it was not what I wanted"!
The iteration pace is also an opportunity for the customer to change their idea. The requirements change depending on the changes in market conditions, and also depending on the progress of understanding by the customer of what they actually want, of what actually has value. The agile team does not need to know all the requirements in advance, instead it can accept new and unforeseen requirements at any time. Whether the new requirements relate to functionality, or features that are already implemented, the cost of change is not exorbitant.
So the analysis, design, coding, testing and integration will still be necessary; but an agile team does not implement them as separate phases in sequence. Each iteration includes each of these activities, which are performed almost in parallel. The demolition of the temporal barrier of the phases increases productivity.
The cost of change
Let us now look at the issue from another point of view. Consider any existing software: how long does it take to put a change request into production? A sentence that we hear frequently is "why is such a simple fix taking so long?"
Agile methods are based on disproving the false assumption that a change costs proportionately more depending on how late in development it is requested. Traditional software engineering claims that a change request in the testing phase costs an order of magnitude more than a change in the coding phase, and two orders of magnitude more than in a change in the design phase, and so on.
In contrast, agile methods assume that you can keep the cost of the changes more or less constant throughout the entire life of a software, from analysis to coding, issue, maintenance.
For this to be true, agile methods require a different process in terms of engineering of coding practices and design, and also from the point of view of planning and work organization.
The journey of an agile project begins with the definition of a customer who has the responsibility of representing the project stakeholders in defining the functionality to be realized. The "customer" so often identified is not the only true customer of the system; but it is important to centralize the burden of defining what functionality should be made (and in what order of priority) in one person.
Agile development is based on a separation of roles and responsibilities. The customer defines functionality and product quality; the team organizes itself to make the product, and is responsible for the technical choices.
Planning for an agile project is based on the separation of functionality in user stories, which are fragments of usable functionalities . The analysis of the problem leads to the definition of a number of stories, each of which has a meaning for the user, in the sense that the customer is able to check it.
It produces value, in the sense that it is clear to the customer that the system is worth more with the additional story than without it
It's estimable, in the sense that the team is able to estimate how much work will be required to achieve it.
It's achievable in a limited time, typically one to three days.
Developers assign an estimated complexity to every story. The estimate is measured in abstract "story points." These points represent the difficulty of developing a story, in relation to other stories. For example, if the A story is estimated at 2 points and B and C user stories are estimated at 1 point each, we can expect that developing A would cost more or less as developing B and C.
The customer chooses the order in which the stories should be implemented. This way the user stories that will be addressed are the ones which have the highest value for the customer.
Iterations and velocity
An iteration takes one or two weeks.
During this time the developers complete the most important stories, specified by the customer, in order of priority. At the end of the iteration the customer can check the functioning of the stories that they requested on the system.
How many stories can the developers complete during an iteration? It depends on several factors. What we do know is that after a certain time, developers can achieve a more or less constant number of complexity points. It is measured that in a typical iteration we are able to develop stories for a total of, say, 12 complexity points. This number is called the team's speed. Speed is not a measure the productivity of the team: it is used to estimate how many stories we can expect to achieve in a given iteration.
Agile engineering practices
The key instrument to keep the cost of change low is the maintenance of a simple architecture. In a successful agile project , the complexity of plugging a new functionality in does not increase: instead, it even decreases with time, as the project evolves towards an architecture that makes it easy to insert the type of changes that are actually required.
How do you get a simple architecture? The first requirement is the correct use of object-oriented programming. In common practice all programmers use object-oriented languages such as Java or C #; many know and use design patterns; and however you often see projects that are realized by means of a procedural decomposition, just coated with a thin layer of objects. Not that there's anything wrong with procedural programming in itself; It can work well in traditional software engineering, but it does not lead to the type of flexible architecture that is necessary to apply agile methods.
Ideally, in the code of an agile project, each class has a single responsibility, and contains cyclomatic complexity of methods comprised between 1 and 2, and a length of 3 or 4 rows, with peaks of 7 or 8.
The second basic tool to get simple architecture is refactoring.
Refactoring is the practice of improving the architecture of existing code in a safe manner. The developer applies refactoring moves to improve the design.
It is all about improving the internal structure of the code without adding functionality; It is a practice meant to combat the natural entropy of all code bases.
It's also a gesture of realism and humility, because you admit to have code written in a way that is not ideal for the architecture, that can be transformed into better code. It's the opposite of the ever-present temptation of those who say "ah, if we could rewrite the system from scratch, then, yes, we would do it well." Realism tells us that even if the code we have is messy, rewriting it from scratch wouldn't get a much better code. So refactoring is a series of small changes, not a great rewriting.
Finally, refactoring must be done safely. The old engineering saying "do not touch what works" has a meaning; we do not want to risk introducing errors in the working code for the ambition to improve it. This safety is achieved through a network of automated unit tests, which are obtained as by-product of test-driven development.
TDD is a design practice, which allows you to make sure that every line of code of the system has been designed following the failure of an automatic unit test. The developer writes a first test, which specifies the behavior of the system in a particular scenario. Given that the test is written before the production code, initially the test will fail. The developer then writes the production code that allows the test to pass. Not only the last test, but all of the system unit tests must pass; This prevents the developer from making the mistake of making progress on one front while breaking existing functionality elsewhere.
When all the unit tests pass, the discipline of TDD requires to apply all of the refactoring needed to bring the system back in its simplest form, while still passing all tests. When the developer is satisfied, he writes the next test, and the micro cycle begins again.
This is the microcycle of TDD. It works in small steps; you write a little test, you add the little functionality needed to get through, you do all the refactoring that seems necessary. Working in small steps will allow you to only make small mistakes.
TDD aims therefore to obtain the best possible architecture. As a useful by-products, TDD produces an automated unit test suite that protects us from regressions, and allows us to refactor with the confidence to not break anything.
Code produced by a team who applies TDD is largely free of errors. I know this is a strong statement to make, but it corresponds to our experience in XPeppers. The debugging activity has almost completely disappeared.
It's clear that TDD is a practice that requires some discipline. To help developers to maintain this discipline, pair programming is a very useful practice.
Pair programming consists of two developers working side by side, at the same workstation, writing code for a specific feature. This practice is based on the assumption that coding is essentially a thinking activity and, as a consequence, two people thinking instead of one results in better software quality.
Coding is a really delicate job, because it affects the development team's most valuable asset: the codebase. Just as it is unwise to let a doctor perform a surgery alone, it's important to have two sets of eyes to check on the modifications that are being applied on the code.
Pair programming is also an important safety net against the danger (which is very real in software projects) presented by obscure areas in the code - the ones that are only properly understood by the developer that has written them. Those obscure areas are often the greatest source of bugs.
This practice is an extreme version of "code inspection", and as a side effect it allows for junior programmers to rapidly make progress towards seniority. Another side effect is collective code ownership.
Collective Code Ownership
An agile team is more than the sum of its parts, because the team members can organize their work together effectively. A strong component of the effectiveness of an agile team is having a common goal. This requires a shared way of thinking about the code and how it should be written, which goes beyond style conventions and also falls into the realm of architectural decisions.
Thus, a mature agile team does not think of the code as a set of separate, independently owned areas. Instead, the code is thought of as a shared asset, a common garden to nourish and to enrich. The rule is that any member of the team is allowed to modify any part of the system.
In this brief document we talked about some of the most important aspects of agile methods. There was no space left to address how the team can organize independently, or how you can set up continuous improvement. I hope it is clear by now that agile methods are development processes based on a set of core values: humanity, courage, transparency, technical excellence, quality, feedback and collaboration.
The bet that agile methods are making is that by adopting methodologies that are centered around human beings and their natural characteristics, the best possible results can be achieved. Agile methods are playing to win.
[agile-manifesto] The Agile Manifesto
[beck-tdd] Kent Beck, Test-Driven Development By Example, Addison-Wesley, 2002.
[beck-xp] Kent Beck, Extreme Programming Explained, 2nd edition, Addison-Wesley, 2005.
[brooks] Frederick Brooks, The Mythical Man-Month after 20 Years, in The Mythical Man Month, Anniversary Edition, Addison-Wesley, 1995.
[cohn] Mike Cohn, Agile Estimating and Planning, Prentice-Hall, 2006.
[fowler] Martin Fowler, Refactoring: Improving the Design of Existing Code, Addison-Wesley, 1999.
Ron Jeffries, A metric leading to agility
[martin] Robert Martin, The Big Redesign In The Sky