(This is item 3 in the code sharing cookbook)
Today, no CV comes along without “Scrum” and “Agile” on it and Scrum has gained acceptance so quickly that it should definitely set off any hype-warning systems. I personally think there’s a lot more to Scrum than just a hype – if you do it right, it’s extremely useful for productivity. There are some aspects of Scrum done right that are particularly valuable for sharing code:
- Empowering teams to deliver end-to-end functionality.
- (Relatively) short iterations.
- Delivering potentially shippable results with every sprint.
The biggest one is empowering teams to deliver fully functioning features. The way I think that affects code sharing is that rather than having your organisation focus on developing technology horizontals, you aim at developing feature verticals.
As much as feasible, Scrum tells you to have teams whose focus is to deliver the blue verticals in the above diagram. The red horizontals will be developed and maintained as a consequence of needs driven by the products. At the other end of the spectrum, your teams are aligned along the libraries or technology components, with each team responsible for one or more services or libraries. Obviously, you can vary your focus from totally red to totally blue, and you get different advantages and disadvantages depending on where you are. At the ‘red’ end of the spectrum, with teams very much aligned along technical component lines, you get the following advantages and disadvantages:
- Teams get a very deep knowledge of their components leading to solid, strong technology.
- Teams get a strong feeling of ownership of components, and can take a long view when developing them.
- You need to align product roadmaps for different products so that they can take advantage of features and changes made in the underlying libraries.
- You will get queues and blockages in the product development, where one team is waiting for the result of another team’s work, or where one team has finished its work and the result “sits on a shelf” until the downstream team is ready to pick it up. (This is what Lean tells you to avoid.)
- Most teams are not customer-facing, meaning that their priorities tend to shift from what is important to the business to what is important to their component. This in turn increases the risk of developing technology for its own sake rather than due to a business need.
At the ‘blue’ end, on the other hand, shared code is collectively owned by multiple teams and you get a situation where:
- Teams rarely need to wait for others in order to get their features finished and launched. This is great for innovation.
- The fact that there is never any waiting and the teams are typically in control of their own destiny is energising: nobody else is to blame for failures or gets credit for successes. Work done leads to something visible. This makes it more rewarding, fun and efficient.
- Features and changes that are developed are typically well aligned with business priorities.
- There is a real risk of under-investing in shared technology. Larger restructuring tasks may never happen because no single team owns the responsibility for technology components.
- There is no roadmap for individual components, which can lead to bloat and sprawl.
- The lack of continuity increases the risk that it is unclear how some feature was intended to work and the reasons why it was implemented in a certain way are more likely to be forgotten.
- Developers (by which I don’t mean just programmers, but all team members) need to be jacks-of-all-trades and risk being masters of none.
I think that the best solution in general is to get redder the deeper you go in the technology stack, because that’s where you have more complex and general technologies that a) need deep knowledge to develop and b) typically don’t affect end-user functionality very directly. I also think that most organisations I’ve seen have been too red. Having product-specific teams that are allowed to make modifications to much of the shared codebase allows you to develop your business-driving products quickly and based on their individual needs. So there should be more focus on products and teams that can develop features end-to-end, just as Scrum tells us!
The next thing that makes Scrum great for sharing code is the combination of time-limited sprints and the focus on delivering finished code at each iteration. The ‘deliver potentially shippable code’ bit seems to be one of the hardest things about Scrum, even though it is actually quite trivial as a concept: if you don’t feel like you could launch the code demoed at the end of the sprint the day after, don’t call the story done, and don’t grant yourself any story points for it. That way, you’ll not be able to let anything out of your sights until it is shippable and your velocity will be reduced until you’re great at getting things really ready – which is exactly right! If it is difficult for your team to take things all the way to potentially shippable because of environmental or process problems, then fix those issues until it is easy.
Actually finishing things is great for productivity in general, but it is even better in a code-sharing context. To illustrate how, I’ll use one of my favourite diagrams:
Assume you have three features to complete, each of which requires three tasks to be done. Each of the tasks takes one day, and you can only do one task at a time. If, as above, you start each feature as early as possible, all the code that is touched by feature A is in a ‘being modified’ state for 7 days (from the start of A1 to the end of A3), and the same applies to features B and C. In the second version below, the corresponding time is 3 days per feature, meaning that the risk that another team will need to make modifications concurrently is less than half of the first version.
Also, having any updates to shared code completed quickly means that the time between merge opportunities is decreased, which decreases the risks of isolation. You really want to shorten the branch – modify – merge back cycle as much as possible, and Scrum’s insistence on getting things production-ready within the time period of a single sprint (which is supposed to be less than 4 weeks – I’ve found that 2 or 3 weeks seems to work even better in most situations) is a great support in pushing for a short such cycle.
I think Scrum is great in general, and done right, it also helps you with sharing code. Jeff Sutherland said in a presentation I attended that the requirement to produce potentially shippable code with each iteration is the hardest requirement in the Nokia test. I can reluctantly understand why that’s the case. It’s not conceptually hard and unlike some classes of technical problem, it doesn’t require any exceptional talent to succeed with. What makes it hard is that it requires discipline and an environment where it is OK to flag up problems without management seeing that as being obstructive. It’s worth doing, so don’t allow yourself any shortcuts, and fix every process or environment obstacle that stands in the way of producing shippable code with each sprint. Combine that with teams that are empowered to own and modify pretty much all the code that makes up their product, and you’ve come a long way towards a great environment for code sharing.
#1 by oskar on April 1, 2010 - 14:17
Nice post again. As I am not an engineer I don’t see the engineering side of scrum as clearly as I might see other aspects. A kind of problematic trend I have seen from a decent number of projects that were very “blue” is that these tend to be either under funded or taking damage from problems with scope control.
I believe this is a cultural problem within the “IT industry” in general. I hope I am wrong here but the problem as I see it shows up as a lack of understanding the challenges involved with delivering user value.
When you go very blue the only useful measurement of productivity is in reality taken hostage by the end user. But since you are very blue you are also expected to deliver the product relatively fast, and cheap.
Lots of organisations appear to be stuck in a hole where productivity is measured by features rather than by user value and this has the effect that budgetary negotiations end up cutting out user testing rather than reducing the number of features. You don’t have to follow this argument more than just a little bit before you realize that the whole thing becomes waste. You might still find business value within waste if you are playing with an aggressive strategy or get rewarded for the local optimization of getting paid per feature.
The reason why I call this problem cultural is because I get the impression that people who bring “red problems” to the budgetary negotiations use arguments that are less negotiable. Blue problems are generally of a soft nature and they tend to be negotiated by aggregating risk on a strategic level for the project. Most strategical risks are very far away from the development team but in the long run the developers will suffer when projects run with almost 100% strategical risk.
I don’t think the solution lies with going red, but I do believe that you need to invest a lot of effort within ever blue project to mitigate this risk continuously. Until the culture has changed to recognize the importance of validating the design hypothesis implemented with each backlog item.
As some kind of summary of this argument I volunteer another negative point for going blue which says something like: “There is a risk of the whole project getting under funded.” ^^
#2 by Petter Måhlén on April 1, 2010 - 17:02
I’m not completely sure I understand exactly, but it sounds as if you’re at least partly talking about the risk that the features that the business owner decides to focus on are not the ones that deliver the most business value? If so, my easiest way to respond to the comment would be to say “I’m a developer, not a problem I can try to comment intelligently on”. ;)
If I would be foolhardy enough to try to comment intelligently, I’d like to understand how different positions on the red/blue scale affect the risk of selecting the wrong features or not funding the project sufficiently. To me, that should ideally be a separate axis. Could you elaborate on that?
#3 by oskar on April 6, 2010 - 09:39
I can try, but its tricky!
There are three angles on this. The first is as you already mentioned the product owner, but since that is constant I’ll drop that one. The second one is the difference between incrementing and iterating. The third one is the difference between business value and user value.
My, rather non-scientific, argument here is that the blue or red choice has a significant impact on the second and third angles. So now I’ll try to make a case for why.
To increment or to iterate? If you go red the answer comes from within the team and some technical interfaces. If you go blue the answer to this question comes from the market. In most implementations of scrum that I have encountered the product owner takes on a role as a proxy for information from the market which then ends up outside the information horizon from the team. Since the team is focused on solving the blue problem rather than the red you need the creative power of the team to validate produced value. Information filtered by a PO is not sufficient to feed the process which keeps the team producing value. I believe scrum mandates a cross functional team, but not clearly motivate or define this aspect of information management. Different types of market have a huge impact on this, if your market is the operation of a machine you can get the benefits of both blue and red, but if the market is represented by end users your team needs to spend its own time handling the information. So why this is more problematic for blue than for red? :P
Forces which operate on the PO outside the range of the team, be they political, social, bureaucratic or otherwise tend to force the PO into incrementing rather than iterating. This commonly leads to design debt in the product. You get a lot of features which are inconclusive and increase the complexity of the product without improving the user value. From what I have seen with going very blue is that the real need for iteration increases dramatically, beyond a point which most people find reasonable btw. This might be a bit special for entertainment products but I actually don’t believe that. It is far more common that more features just adds distance between the user and the value.
So how does business value and user value change when you go from red to blue? In the red corner you should find that they align purely with business value. There is no reason to even care about user value since this is defined within the specification, and your specification defines if you are productive or not. When you go all the way blue you tend to get a compromise. Those “outside forces” I mentioned above seems to give the PO a definition of business value which rarely align with user needs. A benefit of staying red here is that you get almost binary results of “value Produced: yes/no” for each piece of work done. If you go blue the possibility space for productivity stretches dramatically from negative to positive. And here is a trap when you compromise between business value and user value, the business value of a product iteration might be a “yes” but the user value might be negative. You don’t know until after you validate the design.
A good PO should know these things, but I find that good PO’s are statistically rare for IT projects in general. They are more like ordinary humans than superman.
There are some technical solutions to both of these problems, but I rarely see anyone who is honestly interested in implementing them.
(Make business value a slave and user value a master and use truly cross functional teams which include analytical expertise for user testing and design validation.)
Sorry about the long text again, I could probably have made this a lot shorter. ^^
#4 by Petter Måhlén on April 8, 2010 - 20:10
Hm, I still wonder if the problem you’re describing is more a case of something that is invisible when you’re technology-focused, or red. That is, I agree that one of the hardest things in developing software systems, especially consumer-oriented ones, is to validate that one’s ideas about features indeed bring value (I’m guessing “user value” is what you want, and “business value” is what the PO has assigned to a feature). But if you’re looking at a technology perspective, the user value is so far away that you don’t even notice that there is a difference between the two. With a product-oriented perspective, you at least have a team, including the PO, that has the opportunity to do user testing and design validation.
#5 by oskar on April 9, 2010 - 11:12
Yes, I think that is a decent way to sum it up. I do think it is tricky to figure out if these issues are symptoms or problems. The structure appears to be highly abstract and perhaps the root cause is really a familiar one about what risks the project is taking with whose money. Going red maybe implicates that you probably gamble big with some other peoples money.