Chapter 3. Using Scrum for release management – Agile ALM: Lightweight tools and Agile strategies

Chapter 3. Using Scrum for release management

 

This chapter covers

  • An implementation guide for Scrum
  • An introduction to the functional view of ALM: core facets of functional releasing
  • Best practices of how to integrate functional and technical release management

 

If Agile ALM is about creating and tracking software, in which phases do you create the software? Using functional release management, you set up a guideline for how and when you’ll provide releases and how you’ll assign content to them. Together, functional release management and technical release management support and enable the development.

In this chapter, we’ll talk about the release as the central unit in the development process. After covering the basics and core aspects of functional release management, we’ll talk about vehicles that can support you in communicating your general release approach. I’ve based my release management approach on Scrum.[1] This method is popular, in part because it’s simple and easy to implement.

1 See the free Scrum primer at www.scrumalliance.org/resources/339.

Many teams say they’re working in an Agile way, but they’re only constantly planning and replanning, releasing something every two to four months. No one has ever clearly defined Agile release management, which has allowed a lot of dysfunctional projects to call themselves Agile.[2] Many projects use Scrum to support their release management. Scrum doesn’t provide a full definition of release management; rather it’s a rough management template—a framework, as discussed in the free Scrum primer and later in this chapter—and relying solely on the template increases the probability of complete failure. Many important details aren’t described in Scrum at all.

2 Unfortunately, a lot of dysfunctional projects call themselves Scrum as well.

With respect to lifecycle management, this chapter fills out the parts that are missing from what I call “textbook Scrum.”[3] I’ll also cover the basics of Scrum and put Scrum into context, as well as show you additional strategies to extend Scrum for Agile ALM. We’ll implement Scrum and bridge from this approach (and its functional release management) to technical release management.

3 For me, the term “textbook Scrum” (or “Scrum by the book”) refers to how Scrum was initially described by Ken Schwaber and Mike Beedle in the book Agile Software Development with Scrum (Prentice Hall, 2001).

3.1. Getting started with Scrum release management

Figure 3.1 shows an example ALM lifecycle that starts with requirements management and then immediately begins the release management function followed by design, development, version, and build management. The release management collects functional and technical requirements to be implemented in a given iteration.

Figure 3.1. Software engineering goes through a workflow cycle: requirements management kicks off the development process. Requirements belong to releases. Release management triggers the design/development of software, which in turn creates the artifacts implementing the requirements. Those implementations are then put into version and build management and are provided in releases.

This approach drives the design and development of the software, which in turn creates and modifies the configuration elements.

The artifacts created include source code, binaries, configuration files, documentation, and test cases, which are all put into version control. It’s important to trace requirements to the baselines that are approved for release. Requirements and changes are constantly being prioritized and assigned to iterations (and eventually releases); this is often handled by a specific change management function or by a release management function that’s also responsible for monitoring and tracking releases. Acceptance testing, supported by tools and ultimately by the customer or their representative, determines when the release is ready to be promoted. From there, new requirements are determined, prioritized, and approved, which starts the lifecycle again from the beginning.

 

Releases, iterations, versions, sprints, and baselines

In Scrum (as described in books by Ken Schwaber), iterations are called sprints. The outcome of a collection of sprints, delivering a significant version of the software (such as a feature set that has value for the customer), is called a release. A common approach is to decouple the sprints from the releases (which have value for the customer).

A version is an arbitrary snapshot of the software. Releases are also versions, but not every version is considered a release. Important versions of the code are often called releases to indicate that the exact versions of the source code have been identified (often by a tag or version label). Releases are reproducible baselines of the code that can be built, packaged, and released to the customer.

 

How are common project phases, like defining requirements, creating the design, and the development (see figure 3.1), organized in process frameworks? As an example, we’ll take a brief look at the rational unified process (RUP), which consists of four phases:

  • Inception phase— Developing the core idea. By reviewing and confirming the core business drivers, the product feasibility is established and the project scope is defined.
  • Elaboration phase— Defining the majority of use cases and system architecture. This is where risks are identified and a schedule is set up.
  • Construction phase— Implementing a system that fulfills the requirements necessary to enter the transition phase.
  • Transition phase— Ensuring all requirements are met to satisfy the stakeholders. Besides completing the software, documentation is also created. Often this phase starts with a beta release and ends with a software retrospective.

RUP is a popular iterative framework, and it has been successfully used in both small teams and large-scale development efforts.

This approach is valid for all kinds of development processes. Agile projects go through these common project phases as well, in an incremental and iterative way by defining requirements, designing the solution, implementing the requirements, and testing the solution. Although the Agile process uses an iterative approach involving different phases, it can be uneven and you can even slice the development process up into different processes that serve their primary stakeholders. One example of this is that Scrum is often used to manage the development process itself, whereas the overall process, including defining requirements and delivering increments, is managed by using a different process approach.

As a crosscutting discipline, release management acts as a progress-monitoring unit using different tools and data. The progress monitoring is mainly based on the information in the version-control system (VCS), where the artifacts are securely stored. Continuous build management delivers executable software increments that are configured and deployed in test environments.

 

Scrum as a comprehensive end-to-end approach

Scrum can be used as a comprehensive process framework covering all the major phases of the application lifecycle, from collecting requirements through to design, development, testing, and implementation. Scrum doesn’t provide a lot of detail, so it’s common to supplement it with best practices or guidance from other process models.

One popular framework is itSMF’s IT infrastructure library (ITIL), which provides comprehensive guidance on all aspects of IT service management. In its volume on transition, ITIL provides a rich framework for implementing change and release management. You could also supplement your Scrum framework with guidance from the Capability Maturity Model Integration (CMMI), CobiT (Control Objectives for Information and related Technology), or even software development standards from the IEEE (Institute of Electrical and Electronics Engineers) or ISO.

Scrum release management includes an approach to assigning features to release iterations and burn-down charts that make it much easier to monitor progress. All of these process frameworks and industry standards help provide a common set of terminology that facilitates effective communication.

 

Heavyweight release-management approaches traditionally isolate phases or views. ITIL (IT infrastructure library) is one example. ITIL is a helpful collection of publications and a de facto standard for describing processes, roles, and tools. In its most recent version, it gives more weight to the creation of software, the management of knowledge, and the alignment of business needs, and not merely to managing software in production. ITIL has its origins in the management of infrastructure.

Scrum can also be used as a complete process framework. It offers a flexible skeleton that’s open enough to accommodate changes that will occur during the development of software.[4] Although it’s often limited to the construction of software itself, Scrum, as part of a comprehensive end-to-end approach, can cover the major parts of the software lifecycle. All phases of the lifecycle, from collecting requirements to rolling out the software, are important and shouldn’t be managed in isolation. Although Scrum doesn’t specify any particular methodology for managing lifecycle phases, it can be used in conjunction with a wide range of methodologies.

4 “The only thing you know about a plan is that things won’t go according to it.” Kent Beck and Martin Fowler, Planning Extreme Programming (Addison-Wesley, 2001), p. 95.

A benefit of frameworks such as ITIL or lightweight approaches like Scrum is that you have a template you can adjust for your environment as needed, and part of the template is a glossary. All stakeholders communicate using the same terms to express the same things. A common language is a big help.

 

Kanban

Kanban is a concept that was introduced at Toyota in 1947 and in recent years has been gaining momentum in software engineering. In software engineering, Kanban is influenced by Agile and Lean software development (see Lean Software Development by Poppendieck & Poppendieck). Kanban focuses on eliminating waste and bottlenecks as well as reducing waiting times. As a result, the overall throughput is maximized. Kanban uses a pull approach to take new tasks to where they can be completed (that is, to where the resources are available) and it focuses on the complete supply chain. In contrast to Scrum, Kanban doesn’t use iterations but rather maximizes the flow by delivering software continuously.

 

In Scrum, the sum of the functionality to be implemented is located in the product backlog. Features are pulled from the backlog at well-defined points in the process. These points can be either when you’re done with your latest task or when the team moves items from the product backlog to the release backlog to be implemented during the next release. Consequently, the complete product backlog is worked off over several releases. The target scope of one release is placed in the release backlog (see figure 3.2).

Figure 3.2. The Scrum process includes assigning items from a product backlog to a release backlog, and the items are then implemented as part of the release. The output of the release is a working increment (built and delivered software). A typical release duration is 30 days. The team synchronizes daily in a Daily Scrum meeting.

All stakeholders commit to the content of the release in a planning meeting that takes place before development begins. This is also known as the “release kickoff.”

Many adopters of Scrum use the terms release, iteration and sprint interchangeably, and others prefer to talk about sprints that implement the release backlog. Some projects distinguish between releases and sprints, both having content and planning meetings, where a sprint is a sliced subset of the release. I will use the term release from now on, because, in my opinion, it’s more consistent and more approachable in traditional, conservative settings. Due to possible paradigm mismatches, it’s helpful to build bridges between the Scrum teachings and the existing organization’s structure and culture, and that’s what Agile ALM is intended to do.

The release should deliver a consistent, reasonable set of functionality that’s developed incrementally and iteratively. Four weeks is a common release cycle that often works well and is a good starting point. Shorter cycles are possible and sometimes appropriate.

The final, finished release will be packaged and deployed on target environments, although doing this more frequently with continuous integration is helpful. The deployment cycle ends when the release is put into a system test environment available for the customer.

The team synchronizes itself on the status in a brief, daily stand-up meeting.

The major driver of the development process, and of the release particularly, is the set of customer requirements. The prioritized requirements emerge frequently due to changes in the business conditions or the technology used. Requirements are assigned to releases, always having the best ratio in mind between high value and low effort. The implemented requirement is shipped in an increment. Requirements are often sliced into fine-grained tasks, which are more practical to deal with during development. A requirement consists of one or more tasks. The approach of task-based development (see chapter 4) helps to focus on the current tasks and to make progress as well as delivery traceable, in an efficient way.

Figure 3.3 illustrates the process of developing software according to Scrum, highlighting the requirement as the driver of the development process and release.

Figure 3.3. The Scrum process supports the user in transferring prioritized customer requirements into consistent releases and working increments.

Let’s summarize the key artifacts and roles that are used by Scrum. In Scrum, the following artifacts are created:

  • Product backlog— All work to be done. This is a list of items in the prioritized product backlog, containing new requirements to be implemented, open defects, and other work to be done.
  • Sprint (release) backlog— A list of features to be implemented in one release, containing fine-grained tasks arranged in order of priority. The states of tasks are updated on a task board (such as “to do,” “in progress,” and “done”).
  • Impediment list— A list of issues preventing the team from performing work as efficiently as possible.
  • Burn-down chart— A publicly displayed chart showing remaining work in the sprint backlog. The chart also shows the velocity of the work completed so the team and stakeholders can see the progress and whether they’re on track to complete.

 

The Scrum template and common engineering activities

Although they’re not explicitly mentioned in Scrum, essential activities of software engineering do still occur. For example, you will do some sort of requirement engineering (because the product backlog isn’t filled by magic); you’ll probably do some architecture work (but always the least that’s required); you may do some architecture and design work up front (but maybe in release 0); and you may do some additional functional post-testing after the release, although this process has a high degree of automation.

 

Scrum has the following roles:

  • Product owner— The major stakeholder (the user or the customer, or both) of the product or a representative (or proxy) if the stakeholder isn’t collocated. The product owner can make timely decisions so the team can get on with the project.
  • Scrum master— The person who maintains the process and looks out for impediments. The Scrum master mentors the team.
  • Team— A cross-functional group that implements the requirements. The team is colocated and focused on the shared goal. It’s self-organizing and self-accountable and is responsible for delivering a successful outcome at each release. Members of the team have different core competences; many are developers, others are testers. Besides strong technical skills, strong complementary skills are important such as skills in user interface design or design of processes and software architectures. Applying these skills during development makes a valuable solution out of a software application. Traditional soft skills such as strong communication skills are a must. Expertise and experience in the underlying business domain is helpful. The team is open-minded in order to reflect and tune continuously and to complement one another.

 

Note

Issues that will arise when using Scrum probably won’t have anything to do with the method itself. Instead, Scrum makes existing issues visible. Already existing issues become obvious.

 

A key feature of Scrum is that customers work closely together with the development team. The customer is ideally collocated and participates in all relevant meetings. With its continuous deliveries, frequent meetings, and open management of issues, Scrum makes progress transparent. Issues are made visible, and no one can hide behind the supposed mistakes of others. Consequently, Scrum implicitly manages risk. Additionally, Scrum enables continuous improvement and improves team communication. This method emphasizes that every working day is as important as the next, regardless of whether the day is the first or the last during a release. This also means that Scrum enables the project to run at a sustainable pace to balance workload and to make sure that the release backlog consists of a sustainable amount of work for the length of the release.

3.2. Implementing Scrum release management

In this section, we’ll take a closer look at implementing release management within the Scrum framework. Agile development, and Scrum in particular, rely heavily on excellent release management practices to support iterative development.

3.2.1. The release

The release isn’t merely some functionality; it’s numerous technically and functionally consistent configuration items that can be developed and distributed. Whereas versions (or iterations) may be created continuously, you only ship one final release.

Normally, the latest version that meets the requirements is named the final release. You should have measurable, testable acceptance criteria to verify that the software meets those requirements. You may also have customers looking at versions and deciding which will be chosen as the final one. The drawback to this second approach is that you can’t clearly reproduce the decisions (it’s better to have testable acceptance criteria). The most valuable approach is having the customer give feedback in addition to using testable acceptance criteria.

In a timeboxed approach, the resulting version at the end of the time frame (the timebox) is named the final release. Please keep in mind that you should release in a timeboxed way whenever possible, providing the final increment when the defined and fixed release duration is passed through. Timeboxing should be the preferred method. The development cycle is short in Agile, so it’s much better to release the latest version within the allotted time frame and implement other features in the following release, than to shift the deadline. When you’re approaching the release deadline, it’s also much better to take out a defective or uncompleted feature of the release instead of delivering defective software or moving deadlines.[5]

5 In a “death march” project, the scope, as well as the deadline of the project, are moving targets.

 

Detailed requirements

Overly detailed requirements lead to overly specialized solutions that are less likely to be reusable. Such requirements can be difficult to create, are even more difficult to maintain, and take a long time, which delays getting starting and ultimately finishing. One excellent approach is to use high-level requirements that can be reused for other projects, and then supplement them with detailed information in the form of detailed test cases and scripts. Epics and stories have become popular, along with use cases that can be utilized to write acceptance tests.[a]

a See Mike Cohn, User Stories Applied (Addison-Wesley, 2004).

 

Releases are incremental, self-contained pieces of work. Implementation should be done iteratively and incrementally, which means that the software is provided in stages, with a later stage (a later release) providing additional functionality. This additional functionality can comprise new features or enhancements to existing ones. Changes and bug fixes are also possible. Many projects do track and approve changes on the software with the help of official change requests. To take it to the extreme, all types of changes (such as feature enhancements, bug fixes, and so on) that impact the state of the software are handled as changes, without further distinguishing between them. In Scrum, new features, enhancements, changes, and bug fixes are all items in the backlog that need to be prioritized.

 

Strategic foresight: estimating effort and content

Many approaches are available to estimate the complexity of functional content (that is, functional requirements) and to determine the costs and effort required for implementing release content.

Often, time-based estimations are used to estimate the effort that’s necessary to implement the specified functionality. An Agile approach is to estimate the size of user stories with points and to relate the different stories to each other. Other, more traditional approaches include function point analysis to determine the complexity of the application. Function point analyses are calculated based on counts of particular types of elements in an application. This approach requires detailed design up front. Agile approaches use just-in-time design, so Agile projects don’t have accurate counts of various elements prior to building.

Requirement-based approaches are different from metrics like Halstead that do compute the complexity of programs. Other metrics, such as “Lines of Code” can be faked too easily and aren’t meaningful.

 

Agile teaches us that software evolves over the course of many iterations. The Agile process continuously improves design, development, and testing until you have a finished product that meets the users’ needs.

Release Types and Categories

Many projects find it helpful to distinguish release types and categories in order to improve planning and synchronization between all stakeholders, especially development teams, test teams with their respective test environments, and the production/operations teams.

Releases come in many different types and can vary in scope depending upon the sorts of changes implemented in the release. The release type defines the release on an organizational level. Typically, the release complexity, its implications, and the testing effort are metrics used to categorize releases. Some release types include structural releases, business releases, component releases, and emergency releases.

Structural releases are the biggest in terms of complexity and implications, because you change the underlying structure of the software. Common examples are structural changes in databases (data definition languages, or DDLs), which are normally put into this type of release. This comes in handy for communicating database changes to dedicated test and production departments that are responsible for maintaining and updating their respective databases. Test managers or database experts, such as database owners (DBOs) and database admins (DBAs), need to organize their activities. The impact of updating the structure of a database (with a DDL) is different from updating the data in the database (with a data manipulation language, DML), without changing the structure. Because of the difference in the impact and higher risks, structural releases are commonly targeted with a couple of months’ notice to their delivery base. On the other hand, emergency releases to fix a severe production issue can happen at any time, based on a defined process.

The release category expresses the number of changes and is aligned at technical borders. The category name identifies which software artifacts are released. Typical approaches are delta releases, full releases, and package releases. A delta release provides a “diff” between the old and the new version; a full release provides the complete set of artifacts; and a package release contains single packages. Releases not delivering the complete set of artifacts are called incremental releases.

Although release types and categories are independent of each other, there are common traits found between them. For example, a structural release often is a full release, and an emergency release is generally a delta (or patch) release.

3.2.2. The release duration

The release duration—how long it takes the team to prepare the release—should be equal for all releases. This is important for planning reliability and for keeping the team organized in a flow. It’s also considered a best practice to stick to the release flow regardless of any events that happen throughout the year (such as holidays, vacations, and so on). There’s almost always a reason for one stakeholder to want to prevent frequent release; don’t start with special cases to serve the individual wishes of one stakeholder at the expense of all other stakeholders, because every single stakeholder has their own individual preferences and wishes. It’s important that all stakeholders commit to a unified approach. Reduce the number of features delivered in that release, if necessary, and eliminate the root causes of bottlenecks.

If you do change the release duration for one future release, communicate this at once and before the release starts. Put this into the public release calendar (as discussed in section 3.3.1 of this chapter). Don’t extend the duration of the current release while it’s running. If you think your work won’t be done in time, reduce the amount of release content appropriately. You’ll be better off if your releases are driven in a strictly timeboxed fashion, where all dates are fixed.

 

Typical Release Durations

Every project is different and has its own requirements and basic conditions. My suggestion that release durations should be a matter of weeks is only a reference point. In projects applying Scrum, there are debates over whether two or four weeks is the best duration for a release (in this book’s meaning of release). The traditional recommendation is four weeks, but more and more Scrum experts tend to recommend two weeks as the standard.

 

No single rule of thumb exists for the best release duration. In general, the shorter the release period, the better. Shorter release periods lead to early feedback, namely from the customer, who uses the software on staging environments (such as during acceptance testing). Continuous integration, automated testing, and productive development environments help to speed development and facilitate the Scrum short release intervals.

Don’t confuse short release cycles with sequentially running releases at a high, unsuitable speed. A marathon runner won’t choose the same pace for a marathon that a sprinter will for a short-distance race. Similarly, running releases at an unsuitable speed leads to what are often called death sprints, which result in an overheated project, burned-out people, and low quality software. The project may initially look like it’s a success, but the quality of the code gets worse, new features become harder to implement, and new versions of the software introduce new bugs.

Some people say short releases don’t pay off because too much effort has to be invested in the release process itself. Often this isn’t the case: If you integrate, package, and deploy your software automatically, this process will become routine and will always take the same amount of time to run. This holds true even when you have a lot of manual work to do during releasing, but that’s what you must reduce to zero. To take it to an extreme, examples exist where companies deliver new software several times a day.

Release Duration and Automation

Automating the integration, build, packaging, and deployment steps will facilitate rapid iterative development. Automating the most error-prone, most repetitive, and most time-consuming activities is essential. Additionally, automation is critical in all areas where you’re interested in an objective, reproducible result. Another good place to start automation is with processes that aren’t transparent for the team, because this will force them to understand those processes: You can only automate what you understand and what you’re able to describe. Finally, automation helps a lot in areas where manual work is annoying. Good developers have always automated repetitive aspects of their work because of this.

Automation can lead to improved flexibility. A good approach to quality control that leads to improved flexibility is to get the same automated results continuously. You can then step in immediately if the automated process reveals any defects. Besides that, in your daily work, you should focus on those aspects that require human activities. Humans shouldn’t do the work of robots.

3.2.3. The release content

The content of the release is determined and committed to in the release planning meeting before the release starts. Agile approaches differ in dealing with changes that occur during the release. Many projects change the scope of the current release under the agreement that a similar amount of scope needs to be taken out of the backlog. As a general rule of thumb, it’s often best to stick to the approach of textbook Scrum: The scope of the release and its content mustn’t change during the release process.[6] This leads to stable requirements and basic conditions for implementing, and you can better measure what you’re doing.

6 For further discussion on how to handle bugs that come up during the release, see Rachel Davies and Liz Sedley, “Caring about Quality,” in Agile Coaching (Pragmatic Bookshelf, 2009), part III.

The one exception to this rule is that features can be removed from the release, particularly at the end of the release process, when it becomes obvious that a given feature can’t be implemented or fails to meet quality standards. Quality standards can be defined through metrics and tests, and these standards should be fixed and defined before the release starts. It’s not a good idea at the end of the release to remove some failed tests or to remove a metric that’s not met by the application, just in order to pass the quality requirements and complete the release.

During the release, you’ll further specify the features, and you may identify additional facets that are necessary. Specifying and implementing features will often result in new insights, leading to additional (or less) work.

Features taken out of one release go back into the product backlog. They may be put into the next release backlog, but this isn’t always the case, because priorities may change. Consequently, a skipped feature may be implemented much later, or perhaps never.

 

Release Content

A release consists of software modules, installation routines (and other configuration items), and dependencies, including drivers, documentation, installation guides, help, user documentation, release notes, test cases, use cases, user stories, and requirement specifications. Most importantly, the release puts all these in the hands of the end user.

 

The Release Container, and Release 0

It’s important to understand a software release as containing all the artifacts of the required engineering effort (use cases, user stories, and others); source code (Java, Cobol, Scala, and so on) and its related artifacts, such as interfaces, copybooks, and build scripts; test cases (functional and technical); and other artifacts. A release is the reproducible container of all deliverables created in the release, or created before the release started but that become part of the release afterwards. Functional release management defines and tracks the number of changes applied to the software (major aspects of functional release management are planning releases, escorting the progress and delivering the final increment), whereas technical release management tracks and references all configuration units in physical terms (such as JAR, WAR, and EAR files).

A release 0 can be used to kick off the development and creation of the initial infrastructure. This can encompass setting up a continuous integration environment and rolling out tools to the developers. Further improvements and activities concerning infrastructure can also be done in later releases. It’s important that you plan and track those activities.

In addition, a release 0 can help if you have a chaotic development status or don’t know the software’s status, perhaps because you recently introduced a new process model, are applying Agile strategies for the first time, or have only worked with software prototypes before. Collect everything you have (software, documents, and so on), and integrate and deploy it on target machines. Then you can identify the status of your software.

Release Content and Tests

The release content also includes tests, which should be implemented for all functionality. By writing unit tests, developers can find the best design and plan for future refactoring. Functional tests validate functionally correct implementations.

Functional tests that work as a specification for your application are also called acceptance tests. Acceptance tests are directly aligned with business requirements. In the best-case scenario, the customer writes the acceptance tests. This way, the tests are written in the customer’s domain language, supporting outside-in development, as discussed in more detail in chapter 8.

Behavior-driven development (BDD) is a vehicle for expressing the behavior of the application that can then be coded as automated tests. BDD focuses on example scenarios of how to use the developed application. Basing the written tests on example use cases is often referred to as specification by example.

For speed, you should write tests at a lower level than the user interface, except for those tests that are explicitly testing the interface. Tests don’t ensure absolute correctness: They can’t verify that an application is error-free. They can only find errors, and hopefully they find a lot of them.

The goal should be to automate as many tests as possible—automated tests deliver the best value for the effort applied. You also need to be confident in the results of the tests, so you can use manual exploratory testing for any parts that aren’t tested automatically. There are tools for doing that, which we’ll discuss in more detail in chapter 8.

Tests and metrics (such as test coverage or design and code audits) are run continuously and should be required to pass, particularly in the endgame, when the release is finalized. The development team will have greater confidence if tests must always pass, and if failed tests and suboptimal metrics are addressed immediately. Tests and metrics (which yield their results) determine the further promotion of the software. Software will be staged only when quality requirements are met.

3.2.4. Progress and size of working units

To measure progress, Agile teams often use burn-down charts, illustrating the amount of work that has been completed and what still needs to be done. Both the amount of work completed and the amount of pending work should be visible.

The degree of work completed is often less visible, and the process of how to deal with uncompleted work is frequently done wrong. Consider the following bad practice: You work on a feature repeatedly over the span of two releases. At the end of each release, you state that the degree of completion is 80 percent. This is more about guessing than about transparency, and you won’t gain insight into where you are with that feature.

In such a case, be sure to develop a fine-grained slicing of features and to communicate the progress in an objective, measurable way. It’s much easier to go with the binary “done” or “not done” on small pieces of work.

A fine-grained horizontal slicing of functionality could use more units for specifying the same functionality. For instance, suppose you’ve started with two use cases to specify functionality. After the tailoring, you’ve got four separate use cases, expressing the same functionality, but they’re more fine-grained. Their status is now more trackable and work progress on them is more measurable. This approach works for different units and aggregation levels; for example, if you aggregate use cases to features you can split both use cases and features.

The practice of vertical slicing is similar: you can break down the units into subunits in order to manage development and progress better on that level. A common example is to split features (or use cases, if you prefer to use them as the most fine-grained specification of functionality) into tasks where features are aligned with functional/business requirements, and tasks are the mapping from those requirements to fine-grained, technical tasks that have to be worked off to complete the requirement. If your most fine-grained planning unit is larger than two days, think about splitting it into more fine-grained parts. Mike Cohn suggests that you should create tasks with an approximate size so that each developer is able to finish an average of one per day.[7]

7 Mike Cohn, Agile Estimating and Planning (Prentice Hall, 2006), pg. 158.

You don’t have to do the slicing of features all at once, for all features. It’s enough to do it for the high-priority items that you’re going to implement next or for tasks that are clearly understood. On the other hand, larger units can be understood as placeholders for one or more additional units that will be added as soon as they’re understood.

3.2.5. Release commitments

The whole project team works with release commitments. If you have a big project consisting of multiple organizational units, they may all have their individual backlog. Be aware that “when the team is specialized, its goals don’t necessarily coincide with the overall goal of the project, or with the goals of another team.”[8] The team can pull single items out of the backlog to contribute to a release.

8 Gerald M. Weinberg, The Psychology of Computer Programming, silver anniversary edition (Dorset House, 1998), pg. 107.

On the other hand, occasionally, there’s work that needs to be done that’s outside the scope of the backlog. In Scrum, developers shouldn’t be spending time scheduled for development on maintenance, support, or side projects, but such time can be scheduled in if you want to. These activities must be measured and made transparent. For example, you may reserve 20 percent of the time for other tasks besides working on the release backlog. Open issue lists of teams or individuals can be managed by issue parking lots or can be held in a document, such as an Excel spreadsheet.

Traditional environments often use a work breakdown structure (WBS) to define how the team and its members organize their individual commitments. A WBS describes who will do what, in which order, and what the individual pieces of work are, and it can be written in the form of a spreadsheet. Applying “rolling wave planning,” the WBS is more detailed for the work that’s done next. This means that the next commitments are well planned, whereas future ones are planned with a coarse-grained outline.

A team with a conservative posture may manage the development flow with a WBS, whereas a team with an aggressive posture “bets heavily on changes happening and on the ability of the project manager or lead programmer to manage the pipeline.”[9]

9 Scott Berkun, The Art of Project Management (O’Reilly, 2005), p. 283.

Planning of individual and team work, and comprehensive overall release planning can be done simultaneously and in conjunction. As an example, a team of developers organizes their regular work with individual spreadsheet lists. At the same time, those who work in projects use the respective release backlogs of those projects.

The release commitment of the project (the release backlog) and individual team backlogs should be stored close at hand. The product and release backlog should be stored in a more formal way. The purist Agile approach involves writing cards and pinning them on the wall, but that’s not mandatory and often doesn’t scale well. You can also work with a lightweight ticketing system, like Bugzilla or JIRA. We’ll discuss this further in chapter 4, which discusses task-based development.

 

Customizing

This chapter provides implementation practices for Scrum, in order to fill out the abstract Scrum template. These implementation practices have been proofed and found to be resilient in many projects. In your project, you should customize the approach where needed. No silver bullet exists that applies to all environments. Every company and every project is different in some way, so you need different configurations.

 

3.2.6. Synchronization points

ALM is all about synchronization. In every single moment, ALM synchronizes the state of the project. What adds important value in the context of functional release management is hosting regular meetings during your releases. Three main implementations for meetings are the daily stand-up, also known as the daily Scrum or daily stand-up meeting, the retrospective, and the release planning meeting.

Daily Scrum

In the microcosmic view, it’s important for the team to meet every day to synchronize about the following questions:

  • What have you done since yesterday?
  • What are your blockers?
  • What do you want to do by tomorrow?

Do this with a daily Scrum and focus on those three questions. Don’t fall into verbose technical discussions. If a technical discussion is needed, arrange a technical meeting or a design session to discuss the design questions. The focus of the daily Scrum should be how this affects everyone in the team or how everyone in the team can help. The daily Scrum is the most essential meeting because it provides a chance to detect and address issues early, and it provides the chance to foster team communication and team building.

The strategy of not falling into verbose technical discussions is wise for any kind of regular status meeting. The stand-up shouldn’t take longer than 10 to 15 minutes and should take place daily, always at the same time. Good times can be early in the morning or before going to lunch. The involved team should participate; other stakeholders can join, but they’re not allowed to participate actively.

 

Responsibilities (during stand-up meetings)

Here’s a story that spells out the distinction between being committed and involved, in which pigs are committed to the success of the work and the chickens are only involved:

A pig and a chicken are walking down a road. The chicken looks at the pig and says, “Hey, why don’t we open a restaurant?” The pig looks back at the chicken and says, “Good idea. What do you want to call it?” The chicken thinks about it and says, “Why don’t we call it ‘Ham and Eggs’?” “I don’t think so,” says the pig. “I’d be committed but you’d only be involved.”

“Committed versus involved” is also valid in a broader sense. Developers are committed. They decide how much stuff goes into a sprint, whereas others in this case are bystanders.

 

Retrospective

Communication is important, but action is more important. Too often people get bogged down in meetings where they fail to reach consensus and agree upon a course of action. One essential forum for discussion is a meeting in which the discussion focuses on what was done well and what can be improved. This is known as a retrospective.

The retrospective is done at the end of the release and before the planning meeting for the next release. A retrospective can also be done more often if required. The last release will be analyzed to look for areas where the process can be improved (and mistakes avoided). To be sure, there are things that can be optimized. If you detect several issues, focus on the most relevant and biggest ones. Talk about the issues, and find the root cause for the trouble. Only then can you find an adequate counterstrategy. This may sound simple, but it’s often a challenge even for seasoned professionals.

 

Blame Storming and Finger Pointing

If you have people on your team blocking and hiding information (no, I don’t mean the “information hiding” design principle of object-oriented software development) or waiting to blame someone in a status meeting, you aren’t working the Agile way. For a retrospective to work well, you need the key values of honesty, trust, and courage and an environment where it’s safe to speak out.

 

It’s important not to blame anyone during the retrospective, but it’s important to talk about what happened. By applying the appreciative inquiry approach,[10] you can underline positive relationships and the basic goodness in a person, a situation, and an organization, instead of focusing on gaps and problems.

10 Recognizing the best in people while still exploring and discovering better ways of doing things.

In retrospectives, your thoughts and statements should be based on the agreement that, whatever the project members did during the release, they did it to help make the release a success. But this has limits. If you’re working in a culture where people are acting to thwart a project or against the company’s goals, you may need to quit the project as soon as possible, or you could at least establish that the team isn’t being Agile, so that any improvements arising from a retrospective will probably add little value. Retrospective meetings aren’t successful unless the entire team is contributing and feels free and safe to do so. Although this concerns the overall project atmosphere, in the beginning of the retrospective you can run a “safety check” to make people feel more comfortable. Entire books deal with how to set up and run retrospectives.[11]

11 See, for example, Esther Derby and Diana Larsen, Agile Retrospectives (Pragmatic Bookshelf, 2006).

Release Planning Meeting

During the release planning meeting, all important stakeholders will meet to discuss, synchronize, and commit to the content of the release. This doesn’t replace other discussions that will occur during the development process. The planning meeting should be the final point at which the team commits to delivering the content identified in the release backlog. In many projects, the planning meeting is the event where stakeholders present their visions, break features into stories, and plan the release. Dean Leffingwell’s Agile Software Requirements[12] offers great guidance in setting up agendas and checklists for such a planning meeting.

12 Dean Leffingwell, Agile Software Requirements (Addison-Wesley, 2011), chapter 16.

3.2.7. Feature teams, component teams, caretakers

Collaboration and teams are essential for reaching a project goal. “The manager’s job with respect to a team is to start it when a team is needed, leave it alone when it’s working effectively, and stop it when it’s not.”[13]

13 Gerald M. Weinberg, Quality Software Management, vol. 3 (Dorset House, 1994), pg. 264.

During release planning, feature teams are created where necessary. A feature team is a cross-functional team that traces and pushes the creation of a feature that has to be developed as part of the release to fulfill stakeholder needs. This team exists temporarily, working toward a common goal. A crucial benefit of feature teams is that the feature can be implemented using a strict focus that’s free of organizational borders (which could have a negative impact on the architecture of the solution). The feature team takes care of its own organization, including times and frequency of meetings. It’s created by the responsible stakeholder, usually the project management, and it’s responsible for achieving the goal. The team may be led by a feature team leader or it may be completely self-organized. Possible challenges are escalated to the decision makers where needed.

In contrast to a feature team, a component team is a traditional team responsible for creating and maintaining a component in a system. It’s more aligned at organizational borders or at architectural layers, such as the presentation layer or database layer.

Developers can work on both teams at the same time, although a full assignment to one team is often preferred to minimize time-consuming context switches. Developers who become part of a feature team leave this team either after a defined time or when the job is done. Typically, the feature team is retired when its goals are achieved. Then the created components are maintained and further developed solely by the component teams.

Figure 3.4 shows how feature teams and component teams coexist and how, in this example, one developer is part of both teams at the same time. A caretaker can be assigned the responsibility of achieving goals, tracking progress, or maintaining the process. This is often a special manager or expert who is good at achieving goals or solving problems. The Scrum master is an example of a caretaker. Often the caretaker isn’t part of the team, but rather acts from outside the team. For instance, the Scrum master isn’t part of the Scrum team. Another type of caretaker is someone who buys cold drinks after a long working day. Please let me know which pub you’ll be in![14]

14 In his book Agile Software Development: The Cooperative Game (Addison-Wesley, 2007), Alistair Cockburn mentions a use case where “getting seriously drunk together” helped to improve teamwork (pg. 231).

Figure 3.4. Feature team and component team in coexistence. The feature teams work cross-functionally on a feature, while the component teams work on a component. A developer may work as part of the component team and additionally be drafted into a feature team to work on a specific feature. The developer may work on both teams, or may solely work as part of the feature team for a defined period. A caretaker looks out for impediments and mentors a team.

3.2.8. Delivery slots, frozen zone, and code freeze

Although you can create versions of your software continuously (for instance, every night with nightly builds), ultimately you want to complete your work and provide a final stable version. The final version is labeled in the VCS and distributed and communicated as such. The final version is the release.

You develop, integrate, and build your software from the first day of the release process, continuously adapting to change as it occurs. At the end, you want to stabilize the software and pay special attention to the quality, also verifying that the requirements are met. Some organizations call that part of the sprint the “endgame,” the “frozen zone,” or the “hardening phase.” Here are several common strategies to focus on during the frozen zone:

  • Code is committed only after it’s peer reviewed (explicit review in a traditional project setup, organically through team collaboration and pair programming in Agile settings).
  • No new APIs are implemented, meaning only bugs are fixed and no additional features are implemented (there’s a feature-freeze, or features are feature-complete).
  • If you don’t have a complete integration build, including all components and databases, you want that to be done during the frozen zone at the latest. Scrum calls for a potentially shippable solution increment to be delivered in each sprint; the solution increment isn’t potentially shippable unless it includes a complete integration build.
  • If you have timed builds—once a day, for example—you may want to increase the number of builds to get more feedback.
  • The development team might continue to work on the trunk while the release branch is frozen.

The duration and the start of the frozen zone vary. Typically, it takes days. If you get good at Agile ALM, you may reduce this time interval to a minimum, but in a 30-day release, having a frozen zone of two days isn’t uncommon.

The frozen zone isn’t the same as a code freeze (see figure 3.5). The code freeze is a short interval at the end of the frozen zone that spans a few hours, such as the last hours of the release or the last afternoon of the release. The code freeze is the slot where no one can commit anything, not even new features or bug fixes. It’s the time where the releasing team creates the final release. But the procedure of creating the release should be the same as creating the continuous versions. The purpose of the code freeze is obvious: to eliminate any changes to the artifacts. Changes made during the last moments of the release dramatically increase the probability that the next (and final) build will be broken. Although you did build the software continuously, no former build took the last (changed) sources, so the last build can fail. Because the release is strictly timeboxed, it’s important not to have any surprises on the last day or get any new bugs later.

Figure 3.5. The frozen zone and the code freeze for stabilizing and finally releasing the software before the next release is developed (F = feature; B = bug). The way testing is done in an Agile environment often eliminates the need to have a project phase dedicated to detecting and fixing last-minute bugs. There may be a few last-minute bugs, but the dedicated phase should be as short as possible. In the best case, the number of last-minute bugs doesn’t justify a scheduled phase in the lifecycle.

If you detect some final showstopper bugs, they should be fixed. But only one person or a special group should work on those bugs, under the highest quality restrictions. Restricting access on the VCS can be done by conventions. You can email your team and tell them not to change anything in the VCS (they may continue work in their individual workspaces), and then send another email later, releasing the code freeze. In some environments, though, this lightweight approach doesn’t work. If you have a chaotic team, hypermotivated people, or too many team members to inform with emails, you may want to support the code freeze technically (see section 3.4). Code freezes are important for providing stability and the chance to complete a releasable unit of work. Success also depends on a good release plan.

3.2.9. Staging software

Staging software is the process of completely and consistently transferring a release with all its configuration items from one environment to another. The process of staging releases consists of deploying software to different staging levels, especially different test environments. Staging also involves configuring the software for various environments without needing to recompile or rebuild the software. Staging is necessary to transport the software to production systems in high quality.

It’s not always necessary to put a release that won’t be shipped to the customer into the complete releasing process, including promoting it on higher staging environments. But like other processes, the releasing process should be done automatically and continuously. If you skip deploying the software on higher staging environments, you’ll miss an important synchronization point. You’re done with delivering the release only after you have put the software onto the target machine with all its facets (including, for example, creating a technical user account that’s used by the system) and verified that it runs successfully. Many Agile projects try to optimize the cycle time between development software and the point when the end user is able to use the software in production. But having a software release available on a test environment can be of much value, especially if you don’t need or don’t want to put every single release into production.

Only a deployed and available software release illustrates the current status. No substitute exists for that—not any working software in a workspace, and surely no specification or other document in Microsoft Word. Often, and particularly for those projects in the biggest chaos, a complete deployment on a real machine is the only thing that demonstrates the quality of the software and what set of features is implemented and available.

Every significant release of the code needs to be deployed to a testing environment. Sometimes you’ll need to stage the release so that the current testing phase can be completed without interruption. You need to balance the stability of the testing effort with the value added by continuously integrating the code. If you postpone a release in order to include more features, you’re more likely to postpone uncovering serious integration issues, but deploying while QA is in the middle of an automated test run is also a bad idea.

The release itself shouldn’t be dependent upon any specific target environment, although obviously there may be required runtime environments, including external data feeds or other resources. You should be able to reconfigure any release for a specific environment without having to rebuild the entire codebase. Figure 3.6 summarizes how software is developed and integrated on lower staging (or promotion) environments and then is promoted to higher staging environments by configuration only.

Figure 3.6. Staging software: requirements will be implemented in the developers’ workspaces. The central development environment integrates all respective configuration items and is the base for releasing. Software is staged over different environments by configuration, without rebuilding. All changes go through the entire staging process.

All software changes must climb the whole staging ladder, from the lowest rung on the ladder to the highest rung, sequentially. (Commonly, this ladder is illustrated on its side, as in figure 3.6, with the higher rungs being the boxes further to the right.) It’s good practice not to skip any rungs during staging, such as by deploying a software version that was created in a local workspace to the system test environment. Exceptions in staging may be made for emergency patch releases, but how you handle emergencies must be planned wisely. The process for handling emergency patches must define rules for how to retrofit the changes back into the test environments and the current version in the development phase, to prevent environments from getting out of sync. In a complex projects, many companies minimize such synchronization issues by moving production versions of specific configuration items back to lower levels of the staging ladder.

Many projects define fallback strategies for reverting to an older release in case the deployment of the new one fails. Rolling back to an older (usually the most recent) version can involve redeploying all artifacts that were deployed on that staging level before the new release was unsuccessfully deployed.

Rolling back is a valid approach, but it’s better to always move forward and stage all the different types of artifacts, including database dependencies. Reverting a release on a staging level back to a previous release isn’t always feasible, especially in complex settings with many dependencies between configuration units. Database elements are often on the critical path during releasing new versions. Upgrading a database to a new release is often a one-way path, where the old version of the database structure and data can’t be fully recovered to the old version. This is a good reason to use a well-defined staging process and to test the delivery of your software on lower rungs of the staging ladder.

If you can’t revert to a previous release, you need to plan and test the entire release process to minimize risk and to address any possible issues that may arise. It’s always best for the test and production environments (and delivery processes) to be identical. This is the only way you can do a dry run of your production delivery. All staging is a set of changes that further develop the system, as opposed to rolling something back.

3.2.10. Quality gates

Quality gates allow the software to pass through only if it meets their requirements. Figure 3.7 shows the staging ladder with quality gates injected. Developers commit code to the VCS in order to update the central test environment only if the code satisfies the defined quality requirements; for instance, the local developer build may need to run successfully and have all tests pass locally. Build, test, and metrics should pass out of the central development environment, and then automated and manual acceptance tests are needed to pass the system test. The last quality gate to pass is the one from the production mirror to production. Here, for example, specific production tests are done or relevant documents must be filled in and signed.

Figure 3.7. The staging of software is allowed only if it successfully passes through the defined quality gates. Quality gates have an obligatory character, validating the defined quality requirements (tests and metrics).

It’s mandatory to define the quality requirements in advance and to resist customizing them after the fact, when the software has failed. Quality gates are different at lower and higher stages; the latter normally consist of a more severe or broader set of quality requirements, and they often include the requirements of the lower gates.

Feature specification tests aren’t always available when the release starts; it’s normal to detail the specifications of requirements, features, and tests during the release process. Bug fixing occurs not only on coding artifacts, but also on tests. Tests must be corrected and detailed where necessary. Tests must also be put into a VCS and baselined along with the contents of the release.

3.3. Release planning vehicles

It’s always best to create release plans and then communicate them to all stakeholders. A release plan is a central and important utility, and a couple of vehicles are available to express and communicate these plans.

In chapter 4, we’ll look at a tool-driven approach based on JIRA and Trac for setting up releases with milestones and plan and track contents. But that’s the second step. The first step is to set up and create a release overview that communicates the timeboxed slots, indicating when important things happen. This overview can have different implementations. One is a release calendar; another is a release screenplay.

3.3.1. Release calendar

The release calendar is set up and updated by a release manager, if one exists. If not, it may be the project manager. It shows all times for all releases and the key activities required within each. The goal is to provide a comprehensive view of the process with dates, and the calendar broadcasts the information. This calendar is lightweight and not locked down; it’s open to change if required. The release calendar provides the exclusive view of the releasing of one project.

Figure 3.8 shows an extract of a release calendar. In the first column you see the days, and in the second you see a shaded link expressing some activities spanning days. The third column contains comments about what has to be done during those days. This is only an example; the calendar will vary from case to case.

Figure 3.8. The release calendar defines the dates and activities. It’s the single view on the timeboxed releasing that may include activities and deliveries.

In this example, during the first four days in June, the focus is on acceptance testing release 2.2.0.0 of the software on environment B. On June 5, a mandatory delivery is defined: structural database changes influencing release 2.3.0.0 must be finalized and communicated.

During the next four days, the team and the software go through the frozen zone. Here, you also see a common pattern. Structural database changes are often handled differently compared to coding artifacts. In the best approach, all artifact types are managed identically, but in practice this may vary for database elements: database changes must often be defined earlier, as expressed in this example. On June 10, the code freeze for the new release happens and the release is finalized.

You may want to distinguish between a project release calendar and a corporate, central release calendar. The corporate release calendar aggregates all of the distributed releases.

3.3.2. Release screenplay

The release screenplay is a timetable balanced at a zero point, which is normally the day of release. It’s further aligned with important activities and responsibilities. Figure 3.9 shows such a schedule that’s balanced at day X—the moment when the release is created. Looking back from this time, you can see different milestone activities, like the start of the frozen zone two days before.

Figure 3.9. Release screenplay balanced at day X (release day), aligning times, actions, responsibilities, and the stakeholder.

Time

Action

Responsible

Stakeholder

Starting day X-5 Preparing, setting up DB team Release manager; team
Day X-3 Frozen zone start Team Release manager
Day X-2 Writing reminder mail Release manager Dev-team
Day X-2/X-1 Functional tests Customer proxy Dev-team
Day X-2/X-1 Last bug fixes Dev-team  
Day X Creating release branch, tagging Releasing team Dev-team
Day X+1 Deployment environment A Release manager  
Day X+2 Drinking coffee Team  
Day X+3 ... ... ...

Other actions can start when the database structures are frozen, when the creation of target environments begin, and so on. Looking at the time interval after day X, you can see actions such as deploying in target environments or special bug-fixing slots. Typically in this timetable, there’s a column that lists the person in charge of the action. It can also be helpful to mention the major stakeholders in an additional column.

Let’s look at one example: Calling out and starting the frozen zone is the task of the central releasing or software configuration management team, and one stakeholder is the complete development team. A screenplay for this scenario can be set up centrally for common use. It can also be implemented for every concrete release, so you’ll have to transform the variable times to concrete dates. Then the screenplay will increasingly match the release calendar we discussed before.

Another derivative of the release screenplay looks at dates and activities from one role’s perspective. Consider the development team or the release manager. The latter can set up a schedule that describes all actions from their point of view. Figure 3.10 shows an example.

Figure 3.10. Release screenplay, aligned with one role.

Time

Action

Starting day X-5 Sending GO to DB-team for DML changes
Day X-3 Communicate frozen zone
Day X-2 Check status
Day X-2/X-1 Monitor functional tests
Day X-2/X-1 Check status
Day X Creating release branch, tagging
Day X+1 Deployment environment A
Day X+2 Drinking coffee
Day X+3 ...

3.4. Supporting strategies with Subversion

We’ve discussed the basics of functional ALM and have talked about Scrum and the fundamental aspects of releasing. We’ve also talked about calendars as tools for rolling out and supporting releasing. In this section, we’ll cover technical strategies and tools you can use to bridge the functional and technical releasing. We’ll look at technical strategies to support functional releasing.

One strategy is to use hooks (sometimes also called triggers). Another wise choice is to use locking. Both strategies are tool-agnostic.

Some VCS tools use a distributed approach (like Git), whereas others use a more restrictive approach (like ClearCase); the examples in this chapter are based on Subversion (see http://subversion.apache.org). A distributed VCS has a different usage for branches, but the illustrated approaches are useful for both, particularly for handling a main trunk (or a main stream that’s necessary for continuous integration). In chapter 7, we’ll discuss how to bridge different version-control systems (such as with Subversion or Git) to enable feature branching.

Why bother with hooks and locking? Technical releasing collects configuration items for inclusion in releases, and they’re primarily stored in a VCS, checked out by developers, updated, and again checked in. Access to the VCS should be aligned with Agile strategies leveraging the overall release process.

3.4.1. The one-medium approach

Many tools are available on the VCS market. At their core, they all have the same intent and functionality: Artifacts should be stored centrally; stakeholders must be able to retrieve the artifacts, update them, and provide the new versions to the others. This holds true for all artifact types, not only source code. But opinions differ about whether the VCS is the right tool to use for managing all artifact types, including binaries (for example, Word documents), and acting as a file server. In my opinion, it’s better to reduce the number of different channels, to avoid, for example, having a separate file server for storing binary documents. Using Subversion as a file server has the advantage that you benefit from version control and know who has touched which artifact, and when they did so. The information about which is the most recent version of the artifact isn’t always available when using a plain file server.

If lots of large binaries in the VCS slow down the checkout, consider improving the folder structure in the repository. But not all stakeholders are interested in the same artifacts and don’t have to check out all the artifacts. For example, a project manager may be primarily interested in the project archive, including project protocols, whereas an architect is more interested in design documents. Refining the project definition to include roles, the roles’ access paths to the Subversion repository, and the roles’ permissions is often a good way to organize the work with the VCS.

If you have team members who don’t want to store large documents in the Subversion repository, you should gather your requirements and consider the alternatives. For example, pin down what large means. You may also consider storing references to large artifacts, and not the artifacts themselves.

Subversion is a lightweight, open source tool that enables a team to apply Agile strategies. As CVS’s successor, Subversion is popular and its use is widespread throughout the industry.

Subversion is a good tool for storing all kinds of artifacts, including binaries (through its efficient storage strategies). By using it when working on different artifact types, you always profit from Subversion features like version history (which is important for tracking changes) or hooks and locks.

Another advantage of using Subversion is that all stakeholders can access the tool and the artifacts stored in it. Artifacts in their versions can be accessed by developers (via Subversion integration in the IDE) or by nontechnical stakeholders, who can use the free TortoiseSVN tool. Once installed, TortoiseSVN adds Subversion commands to the directory explorer of your local operating system (see figure 3.11).

Figure 3.11. Accessing Subversion on Windows with TortoiseSVN. Menu action items are part of Windows Explorer after installing TortoiseSVN.

Both Subversion and TortoiseSVN are easy to use, though it may take a bit of training to get the team up to speed on using them.

Another interface for accessing the repository is a browser. You can peruse the central Subversion repository and its content with a web browser (if the repository is hosted via Apache, which is the recommended strategy). This way, you can provide read-only access when necessary.

Subversion has several interfaces that make it much easier to use. Automating common tasks through scripts, known as hooks, also helps improve Subversion’s usability.

3.4.2. Hooks

A hook is a program or script that’s started when entry conditions apply. For example, a trigger could be thrown due to a Subversion repository event. These events are aligned with significant VCS lifecycle operations. A commit is an operation, for example.

A hook is a self-contained piece of work. It knows its context, including who did (or wants to do) what. The hook can have output or a return status that can influence the further behavior of the system. Hooks are powerful vehicles for synchronizing the technical and functional worlds.

Hooks are used in the niche between functional and technical releasing and they can allow you to enforce certain rules, such as the following:

  • During code freeze, developers aren’t allowed to check in changes to VCS.
  • During code freeze, only lead developers are allowed to check in changes to VCS, on demand.
  • Multiple users can simultaneously update source/ASCII artifacts.
  • Multiple users can’t simultaneously update binary artifacts like Word documents.
  • Stakeholders must enter a commit message (with a minimum length of x letters) for all updating actions.
  • Stakeholders must enter a commit message, including a ticket number, to enable task-based development.
  • Stakeholders aren’t allowed to access the repository when the administrator wants to drive a repository backup.
  • Stakeholders should receive emails with details of updating operations.
  • Stakeholders aren’t allowed to update frozen versions of the software. This system must prevent updating changes on repository paths that contain tags (after you update a Subversion tag, it becomes a branch).
  • When synchronizing a ticketing system with changes in the VCS (depending on the commit message), you change the state of tickets; for instance, you close or comment on them.

Hooks are helpful, but having too many can slow down the response time of the VCS and cause unexpected (and undesirable) outcomes.

In the Subversion cosmos, after creating a Subversion repository, you also create a directory named hooks inside the repository. All executables put in that directory are automatically run in their individual context, depending on the trigger event. Here are the most important hook triggers available in Subversion:

  • Start-commit— Runs before commit transaction begins
  • Precommit— Runs at the end of the transaction but before commit
  • Postcommit— Runs at the end of the commit
  • Postlock— Runs after a repository path is locked
  • Post-unlock— Runs after a repository path is unlocked
  • Pre-unlock— Runs before an exclusive lock is destroyed

 

Project role: Captain Hook

Using hooks can be helpful in application development. Often hooks are used for extending or modifying runtime behavior. Hooks are valid OO tools for designing good APIs. Examples occur in the context of delegation and inheritance—such as in the template method design pattern.

Please use hooks and the word itself sparingly. There are people (let me call them Captain Hooks) who patch and hack sources and add functionality systematically on a daily basis, while calling that hooking. Tinkering with sources without caring about the design, extensibility, and sustainability of software isn’t using hooks; it’s tinkering.

By the way, hooking a hook script is a bad idea—as bad as modifying the Subversion repository with operating system commands instead of the intended interface. If you need to extend Subversion, use the dedicated API!

 

Subversion hooks are used by convention. The hook script in the directory must be executable (often the hook scripts are Python scripts) and named with the intention and context you want that hook to be applied to. For example, if you want to create a start-commit hook with Python, you would name the file start-commit.py. On Windows, executable extensions are exe and bat. To make it easier, the directory already includes hook templates named correctly. Subversion uses the extension .tmpl for templates, which you can easily rename.

 

Warning

You can run Subversion and host Subversion repositories on many different platforms, including Windows and Linux. But be aware of platform-specific peculiarities. For example, Subversion configuration is case-sensitive and Windows is case-insensitive. The decision of what platform to use can depend on many different factors; for example, if you want to run Subversion with 64-bit Apache, this installation isn’t as mainstream on Windows as on Linux.

 

Further details about hook scripts can be found in the free Version Control with Subversion book, available at http://svnbook.red-bean.com.

3.4.3. Flow and locking

Subversion naturally supports Agile workflow because it uses the copy-modify-merge model as an alternative to the classic lock-modify-unlock model. Copy-modify-merge is optimistic about locking where files aren’t generally locked initially. This means all stakeholders can work on and change a file at the same time. Developers have local copies of the central repository (or parts of it). They work simultaneously, modifying their private copies, and after their work is done, the private copies are merged together into a new, final version. Subversion assists with merging, but you’ll still have to make sure that the merges are performed correctly.

This approach can dramatically accelerate the development flow and the velocity, because work can be done in parallel and waiting times are reduced (idle time is waste). The drawback is that someone has to deal with the merges, particularly the merge conflicts. Agile suggests frequent small commits to avoid large merge conflicts. The alternative is to apply pessimistic locking, where only one person at a time can work on a file, but merging scenarios are much easier to handle.

Copy-modify-merge is based on the experience that in software projects, it’s rare that more than one person is working on one file. If multiple people work on one file simultaneously, they (mostly) work on different parts of the file so there’s no reason for additional communication—the tool can merge such cases automatically.

Tools like Subversion automatically merge different developers’ versions into a central, leading version. In cases where this isn’t possible, the tool can provide merging information. The human in front of the keyboard must then decide what the new version of the final file should look like.

 

Locking and Communication

Locking doesn’t replace communication. Rather, it supports releases and communication. Stealing or breaking locks is technically easy with Subversion. Please use the locking features wisely and in cooperation with your peers. There are good and bad scenarios for stealing locks. First, a good one: A colleague is on holiday and forgot to release his lock. And a bad one: Hey, the guy on the other side of my table locked the file; but it’s my file, and I want to update it now.

 

Subversion’s support for merging binaries is limited. That’s no surprise. How should the tool, for example, merge two versions of a Word document or a JPEG image? For binary files, a stricter approach is helpful when working with locks. With the Subversion command svn lock word.doc, you lock the file word.doc. This command runs directly on the file in the repository. After it’s applied, other people can’t change the file; the file is locked for your changes. Conversely, svn unlock word.doc releases the lock again (also, a commit releases the lock by default). These operations will work only when the file isn’t locked by another user. You can overwrite another user’s lock by using the force attribute—for example, svn unlock --force word.doc. This is known as breaking or stealing locks.

 

Note

Working with locks and properties is also conveniently possible with Subversion clients like TortoiseSVN or SmartSVN.

 

It’s also helpful to tell Subversion to expect locks for individual artifact types by default. This means that setting locks isn’t optional but mandatory before you can change a file of a special type. In such cases, or if you want to simulate a complete lock-modify-unlock approach, you can configure Subversion to operate accordingly. Subversion can deal with properties (metadata) to achieve that. They can be set (and likewise also edited after being added). Setting the needs-lock property then looks like svn propset svn:needs-lock yes file.doc. This is something that happens in your local sandbox first (you don’t run the command changing the attributes directly in the repository). This means you have to commit this in the same way you commit other changes to files.

This is convenient for single cases. But you have to do that for all your file types or files individually and manually. Your local Subversion client installation contains a conf directory, and you can set it so that for all newly added files of a special type, the property is added automatically:

[miscellany]
# enable-auto-props = yes
[auto-props]
*.doc = svn:needs-lock = true

This snippet activates automatic property setting and defines the individual property rule. You can combine locks with a precommit hook that ensures that the property is set on newly added files. You can also use prelock and pre-unlock hooks to let administrators decide when to permit lock creation and lock releases. With these hooks, you can prevent certain users from breaking or stealing locks, or allow them to do so.

 

Tip

In the same neat configuration file, you can also customize which editor to use for merge tracking and many other things.

 

Subversion is a popular open source VCS and lends itself nicely to Agile development.

3.5. Summary

In this chapter, we looked at what functional ALM is, and we introduced Scrum as a popular high-level management template. We talked about releases as the central item in the functional releasing process and discussed releasing strategies inspired by Agile strategies. You can use the strategies in pure Agile projects and in rich, traditional approaches. We also discussed expressing and communicating release plans. Release calendars and release screenplays are valuable tools for defining and communicating the functional releasing process. Tools for version control like Subversion support running an Agile approach. Running the copy-modify-merge optimistic locking pattern increases flow and velocity. In special cases, Subversion still allows administrators to create stricter enforcement policies through the use of locks or hook scripts.

In the next chapter, we’ll discuss using toolchains to provide functional and technically consistent releases in a task-based way.