Three-layer architecture violates DRY!

I didn’t go to NDC this year, but I just watched this video by Tore Vestues and Jonas Follesø. They show that in a typical three-layer architecture you duplicate the entity structure about ten times, and that by focusing on the simplest thing that can possibly work, they can pretty much remove all of that duplication.

A fresh and very though-provoking talk, indeed! It is definitely time to question how SOLID code can be DRY when we keep adding all this complexity disguised as flexibility.

Actually, the best take-away from the talk was that you should not implement design patterns if the net result is more complexity rather than less. This reminds me about one of my previous projects, where someone backed each and every class with an interface up front, not knowing if any of them would ever become useful.

I think some questions about Tore’s and Jonas’s agile architecture need to be investigated further:

  • How likely is it that you would eventually need to add a relational DB (for easier structure modifications and faster queries) and a domain layer (for more expressive business logic and a faster test suite) anyway?
  • How easy is it to refactor in these changes later rather than building them in from the get-go?
Posted in Uncategorized | Tagged , , | Leave a comment

Can iterative development work at all in enterprise projects?

If you work on an agile project and you happily churn out functionality iteration after iteration, you should read Alistair Cockburn’s “Are iterations hazardous to your project?” if you haven’t already. His point is, unless you deploy the code after each iteration to real people who will actually use the new features, you are just fooling yourself. Your project may be indistinguishable from a waterfall project, except for the iterative planning window. As a side note, see “Waterscrum vs. Scrummerfall.”

But what do you do if your users don’t want a new version every month? What if you are replacing a legacy system? Then the value of the new system will be zero until the day it is superior to the old one. Or what if your customers are so paranoid about change that they need 6 months just to verify your new version? Kevin Brennan goes into some more detail about such problems here.

Following Cockburn’s reasoning, perhaps you need to run multiple-month iterations in those cases. Although I would say that the monthly planning window has other benefits, so I don’t think I would go much longer than that. But I think the point is that it is hard to be agile under such circumstances.

It seems that the key to breaking this dilemma is to convince at least one of your customers to be a pilot user, agreeing to actually try the new system with some real life tasks, even if he relies on the legacy system for most of his work. Many users are willing to do this after attending a successful demo where some really nifty features are shown.

The worst thing we can do is denying enthusiastic users an iteration release, for whatever reason. Those users are the key that enables our agile development operation.

Posted in Uncategorized | Tagged , | Leave a comment

Scrum and Evidence Based Scheduling

How do you predict a release date? In a traditional waterfall project the project manager probably maintains an unwieldy Gantt project plan that tells him the exact date when the system will be finished, but we all know that doesn’t work in real life. Agile teams realise this, and usually conclude that a ship date cannot be predicted, because it depends on the customer’s continuously changing requirements and priorities. That isn’t very helpful, however, if your customers demand a prediction or if your organisation needs to plan a roadmap of future projects and releases.

Joel Spolsky proposes “Evidence Based Scheduling,” a technique that combines detailed estimates and historical estimation precision with Monte Carlo simulations in order to output a confidence distribution of ship dates. While I’m sure this works perfectly well, it isn’t exactly agile, as you have to identify and estimate all tasks in painstaking detail. That means you have to design the entire system down to a certain level up-front. He actually says this himself. Why is that so bad? Because you are bound to spend time designing and estimating features that won’t make it to the end product. And of course you don’t get the benefit of designing features after the requirements have matured from several demos and a lot of communication.

You can predict ship dates in agile projects too. If you use a product backlog and plot a product burndown chart, you can extrapolate a trend line after a few sprints and see when you’ll be finished. You need a reasonably complete backlog to do this, however, and all items must have been estimated. But you won’t spend much time estimating them, and you don’t have to worry about the estimates being very accurate. Actually, you don’t predict the ship date by summing up all the estimates. That would be as inaccurate as the Gantt method mentioned above. You predict using your team’s velocity, i.e. your team’s ability to reduce the backlog per sprint. That is why it takes a couple of sprints before you can start predicting.

Now, there are still a few problems with this approach. First, the burndown goes down as you implement features, but it also goes up and down as the product owner revises the backlog, creating havoc with your trend lines. That could be rectified by using a burn-up instead, where the ceiling changes with the backlog scope. Second, the trend line may fluctuate a lot from sprint to sprint, and there may not even be a trend line if the burndown is curved or erratic. It may be useful to bring in some more advanced metrics, like AgileEVM. Third, there is no confidence distribution as with the Evidence Based Scheduling. Hmm, why don’t we just take advantage of the same technique using historical data and Monte Carlo simulations?

What I’m proposing, then, is this. Build a product backlog. Make it as complete as you can. The backlog will evolve over the course of the project, of course, but be aware that the release prediction will assume that the product backlog represents what you actually intend to release. Now estimate all items on the backlog. How you estimate them doesn’t matter. You could use hours, function points or T-shirt sizes. Next, run a few sprints. If you have estimated using T-shirt sizes (S, M, L, XL) or something else discrete, try to implement some features of all sizes. Track the hours spent on each of the items. You can now calculate a set of evidence (hours spent per estimated hour or function point), or several sets (hours spent on XL items, hours spent on L items, etc.). Now do the following a hundred times:

  • For each unfinished item on the backlog, select a random evidence element and use it to predict how many hours needs to be spent on the item.
  • Divide the sum of hours remaining on the average number of hours the team is able to work during a sprint.
  • The result is the number of sprints remaining.

You will end up with a hundred predictions. Sort them and plot them. There is your confidence distribution.

Posted in Uncategorized | Tagged , | Leave a comment

Agile inspections

I went to Smidig 2007 earlier this week, and it was pretty good. The highlight came at the end of two intense days of lightning talks and open spaces, when Tom Gilb offered to give a session on agile inspections to three other guys and me.

Tom Gilb is known to many as an authority on software inspection, but as he made a point of himself, his 1993 book isn’t agile. The big problem with inspections is that they take a lot of time, especially if you want them anywhere close to being effective. The Pareto principle doesn’t apply here. You don’t find 80% of the defects during the first 20% of the review session. If you try to cover a lot of material in a short time, the effectiveness will be close to zero, because the checkers won’t be able to look closely at anything. Gilb claims that the optimal effectiveness occurs when you inspect about a page per hour, or 20-30 lines of code per hour. Even then, inspections won’t get more than 60% effective. Who can afford to review everything?

How can you do this more agile? By redefining the purpose of the inspection. The traditional view is to use inspections to find as many defects as possible (ideally all of them), so that they can be fixed. Forget that, it’s futile anyway. Look at inspections as a way to measure quality, not as a way to detect bugs. Use inspections as a way to enforce the lean principle of building quality in. By inspecting just a small amount of material, you can get the job done in an hour, and you will know if the material is of sufficient quality. If it isn’t, ask the author to rework it. The idea is that he will learn about quality requirements gradually by participating in the review, and he will find out on his own how to build more quality in. At the second round of inspection, quality will hopefully have improved, but it may still be insufficient with regard to your exit criteria. In that case, you repeat.

When the material finally gets approved, you will have achieved the following:

  • Production material meeting your quality standards.
  • One of your co-workers has learned a lot about quality and is much more likely to produce good material next time.
  • You didn’t have to inspect very much.

See Gilb’s own information about all of this here. I’d love to hear your opinions.

Posted in Uncategorized | Tagged , , | Leave a comment