Scrum Training

This week we held certified scrum master, 'Writing Effective User Stories', and 'Scrum Primer' courses for about 40 staff which I'd organised through Rowan Bunning at Scrum With Style. We had the benefit of having two trainers for most of the sessions with Simon Bennett in town for Agile Australia. Very interesting and enlightening sessions even for those, like me, who thought they pretty much knew it all. I'm sure it will generate a few posts i nthe coming months as the lessons sink in...

Testers not required

I don't like testers. Let me rephrase that, I don't like needing testers. We had a debate today regarding how to future-proof a legacy system to make it viable for the long term. We've had a number of production issues that have caused us some embarrassment and the project manager wanted to find ways to avoid these in the future. The system itself is a large, poorly architected, poorly coded system with many external dependencies. The code is brittle and without a completely automated build it takes significant time to setup even a manual regression test.

The project manager (and many of the developers) wanted to get testers into the team to ensure full test cycles are carried out before putting a release out to the business. Whilst not completely against the idea I saw this as an attempt to treat the symptoms of the problem (bugs reaching production) rather than the cause (bad code). If we're seriously suggesting using human testers to perform regression tests in 2010 then we're doing something wrong. Testers have a place for new functionality but we're really not producing enough new features - due to the bad architecture - to warrant a dedicated tester just for new work, and in any case we have a BA capable of doing this testing.

This is a simple case of technical debt. Paying testers to regression test is like paying interest-only on that debt - not reducing the principle but still servicing the loan. In order to reduce the interest payment we need to payoff some of the principle - i.e. refactor, replace, re-architect the code. To do this we need skilled senior developers not testers.

But there is another reason I don't like testers, especially on teams with more junior developers. The developers seem to believe that having testers absolves them of the responsibility for preventing bugs getting into production. They can say things like 'well who tested that function - how did this get through?' with genuine indignation. This transforms into a tendency not to test their own code, automate tests, or even think about tests before coding - all things we _require_ to be done on new projects.

So if you find you need testers to stop bugs leaking into production software what you probably need is better software with continuous integration and automated tests - i.e. what you need is better developers. Once you have this in place you'll find you only need BAs and business users to test new features.

Ergo - Don't become a tester!

Behaviour Driven Development

Craig Dunn sent me some reading material today introducing BDD or Behaviour Driven Development.

Basically it explains what I always thought TDD should be, and suggests that BDD is merely TDD 'done right' and with a name change to make us start thinking in terms of behaviour specification rather than code verification. Very worthwhile read...

Philosophy of development?

We're having a bit of a group think about our project management methodology here at pblMedia. A colleague, Peter Kearney, sent the following interesting email which I quote (almost) in full.

Most people don't realise this - but the methodologies we use for ... project management ... are quite remarkably influenced by philosophy:

From the late 1800's to about the 1980's and 90's a philosophical paradigm known as Positivism dominated almost every aspect of thinking and shaped many of the ideas we have about how things should be done today. Positivism is grounded in empiricism and the idea that reality is an objective thing. In a project management context this would lead to thinking as tasks and user requirements as things that can be defined absolutely. You can see that the waterfall method would be a natural response to this paradigm of thought.

Interpretivism started in about the 1920's but really only managed to get serious recognition in the last couple of decades. This philosophy is based on the idea that reality is not absolute - its in our minds. This paradigm recognises that our perception of reality changes depending on the circumstances we are in. In our world - this means that users perception of reality IS reality and that reality therefore changes as users circumstances change. You can see that agile is much more aligned to this way of thinking.

The trick though is to find the middle ground between positivism and interpretivism. Both are right: realities such as budgets are absolute, but realities such as user expectations are not. The ultimate trick in all of this is not to force rigid adherence to a primary scope of work, but rather to help users and stakeholders to shape their expectations to fit the budget and the reality that is evolving in their minds as the system is being developed.

Well I found it interesting anyway. This also reminds me of the Wind and Sun Aesop's fable

The North Wind and the Sun were disputing which was the stronger, when a traveler came along wrapped in a warm cloak.
They agreed that the one who first succeeded in making the traveler take his cloak off should be considered stronger than the other.
Then the North Wind blew as hard as he could, but the more he blew the more closely did the traveler fold his cloak around him;
and at last the North Wind gave up the attempt. Then the Sun shined out warmly, and immediately the traveler took off his cloak.
And so the North Wind was obliged to confess that the Sun was the stronger of the two.

The moral was stated at the end of the fable as: Persuasion is better than force. The history of project management has focused on imposing more and more process and up-front planning and forcing the project to comply with the initial plan. Newer agile techniques have focused on persuading the project to a conclusion, through constant adjustment to changing requirements, collaboration of project team members, and ensuring developed (rather than planned) functionality meets business expectations.

Dividing application work horizontally rather than vertically

My colleague and I took a trip to the Microsoft SDC in Sydney the other week as part of an 'open day', where MS staff and partners described their development and project management methodology - basically a flavour of Scrum. Although a lot of what they are doing is very similar to our own process (which is a good sign I guess) there were a few things I thought worth considering. One of these was horizontal allocation of tasks amongst developers

Basically they advocated splitting the development of functions / user stories into tasks to be divided amongst several developers according to the tiers in the architecture. Whereas we (and many others) normally assign a single function/story to a single developer to code the entire stack (DA, BO, UI) they would assign this work to three developers. The argument they put forward for this was skill specialisation - a data access developer has a different set of expertise than a user interface developer.

Whilst I agree with this to some extent, no developer I know would be happy to be stuck working in a single tier of the app for the entire project.

However thinking about this more, if we did enforce the standard that the tiers of a single story must be developed by different developers what would be the outcome:
1. Developers would have to talk more about the design before coding
2. No single developer would come up with the design
3. Separation of responsibilities of each tier in the app would be easier to enforce (you'd need at least two developers to collude to change the design). Coupling between tiers would hopefully be minimised.
4. Each developer would know about more parts of the application.
5. More stories would be completed within an iteration (rather than having multiple stories still being worked on by individual developers at the end of the iteration, we'd have one or two being worked on at the end)
6. If developers worked on different 'tiers' for different stories there would also be more consistency in terms of coding style - patterns, naming, layout, etc
7. Higher chance of code re-use since developers would know about more areas of the application.
All of these are positive effects, completely inline with the tenets of agile - collective code ownership, collaborative design, pair programming (sort of). I can't really see any downside to this except a co-ordination issue (we need _something_ for project managers to do!), though the proof will be in the doing..

Now I'm off to find a suitable project and some developer guinea pigs...

In defence of custom builds

We seem to be having an attack of platform man at work recently. We've replaced so many legacy third party software with custom built applications that management seem to be questioning the value of all this effort. Now I'm no zealot in this regard (I've even pushed to cancel some development projects as being a waste of time) but it does appear that we've lost sight of the reasons why custom built (as in tailor made) software was attractive to start with.

Here's my take:

1. Expect significant integration work. Unless you're buying a whole suite of products to cover your entire business integration is hard. Many vendors are singing the SOA tune but at the end of the day IT has to figure out how to get data in and out of the application. Of course the big vendors will sell you the tools to do the integration (and the hardware too!).

2. Unwanted maintenance releases. Third party tools require upgrades otherwise they become obsolete. However too often the upgrade provides no value to the business but consumes IT time and resources which could be better spent developing new code, building frameworks, improving existing apps, etc.

3. Wanted maintenance releases. When you _do_ want some additional functionality you need to wait for the vendor to supply it or revert back to (gasp) custom development to plug the gap.

4. Basic support is more time consuming. Take user management for example, every third party tool will have its own user permission management system. Whereas all of our internal, custom-built apps share a common user management service managed centrally by help desk staff, those same staff will need to learn and use a separate system for the third party apps.

How fast could a 1000 piece jigsaw puzzle be completed

Suppose you take 12 hours to complete a 1000 piece jigsaw puzzle. If you had a friend helping how long would it have taken? it would probably not be less than 6 hours, unless you had some complimentary skills (e.g. he was good at recognising subtle shades of blue found in the sky peices whereas you could match a peice in your hand to the image on the box). For the sake of this discussion we'll say 6 hours.

What if we add another friend? We might find this one is particularly good at the matching the fine detail of the foliage and we get our time down to 4 hours. In all three scenarios the number of completed pieces per person-hour (ppph) is the same (83). Eventually though, adding another friend will slow down the ppph rate. For example when we have five friends working on the puzzle this may take 3 hours at a ppph of 67.

Most IT 'managers' understand this analogy - they understand that adding developers to a project (in order to deliver it earlier) will also reduce overall productivity. Many also believe that adding those extra developers will always reduce the time taken to complete the project. Let's return to our puzzling friends...

Suppose we now have a party of 10 friends working on the puzzle. We would surely start to see instances where two people are looking for peices which each other is holding, more than one person will want to look at the picture on the box at the same time, it will be impossible for everyone to access the puzzle at the same time - people will start taking peices away fit them together and bring them back for 'integration' into the main puzzle. The time taken to complete the puzzle may go back up to say 3.5 hours at a ppph of 29. (In reality some of the friends will get tired of the elbow pushing and go off to play diplomacy in the other room leaving 3 or 4 to complete the puzzle in less time - but this is an analogy so it didn't happen that way)

So adding resources can in some cases reduce productivty AND increase the time taken to complete a project. You may argue that this is not a good analogy - software development is more complicated than completing a jigsaw puzzle. Let's introduce some specialisation...

We could add some processes and specialisation of roles. First we could assign a few people to be 'puzzlers' who do the actual fitting together of pieces - one could do sky, one could do foliage, one could do buildings, etc. Others could be collectors - finding peices that the puzzlers need. One could be responsible for integration - constantly collecting and fitting puzzlers' completed puzzlets into the main puzzle. Once could be responsible for troubleshooting puzzlers problems (the puzzle gun). One could be responsible for making sure the puzzlers have access to the picture on the box when required.

We may have:
5 puzzlers
2 collectors
1 integrator
1 puzzle troubleshooter
1 picture manager

With this arrangement we may be able to make some improvements on our completion time - say to 2.5 hours at a ppph of 40. By specialising roles and adding processes and procedures (and hours of practice) we may even be able to add more resources and complete the puzzle in 2 hours (at a reduced ppph).

There would still come a point however that adding an extra person would tend to lengthen the time taken rather than reduce it. What is even more true is that if, say, you had a target of 2 hours to complete the puzzle and after 1 hour and 45 minutes it was clear that you were not going to make it, getting a few more friends involved to help will not be of no use if they are unfamiliar with the puzzle itself (the software in development) or the roles, responsibilities, and processes used (the development methodology & standards). The likely effect is that the puzzle takes longer to complete.

What have we learned:

1. The most efficient (cost effective) way to develop software is for one (highly skilled) person to do all the work.
2. Adding more people to a project is useful for small teams
3. Adding more people to a large project requires specialisation of roles and well defined processes
4. At some point adding an additional resource (even at the start) will make the project longer
5. Toward the end of a project adding an additional resource will normally make the project longer (this is Brook's Law)

The following charts show (purely conjectural) figures for efficiency and duration of puzzle completion.

Projected ppph figures for different numbers of friends completing the 1000 piece puzzle (process improvements made when 10 friend's were involved)

Projected overall time taken for different numbers of friends completing the 1000 piece puzzle