Scrum is a Mirror

I recently organised Scrum training for the internal development and PMO team here at Nine Entertainment, consisting of two days of Srum Master training for 17 project managers / lead developers and 1 day of effective user story writing for around 20 business analyst types. One of the things that the trainers said early on is that 'Scrum is like a mirror' highlighting the best and worst things about your product and development process.

Now, at first I didn't really grok this at all, but after mulling it over I am coming to the realisation that this is true. Retrospectives if done regularly and correctly highlight issues with your process very quickly - whether this be technical design problems, lack of responsiveness by the product owner, too much micro management by the scrum master, they can all come quickly to the surface and solutions be discussed. But it's not just the process that has a mirror held up to it's face, the product gets the same treatment every sprint review. The stakeholders get to answer the question 'is this the product we wanted?' time and again.

Compare this with a waterfall approach and the difference is clear. Lack of engagement by a product owner - we'll just make sure we document everything. Stakeholders not sure what they want - we'll force a decision and stick to it. Not sure if the design is up to scratch - don't worry it will come out in testing. Waterfall avoids looking at the process or product too closely and instead reassures itself by looking at deadlines and budgets - if we're meeting them we must be fine. No matter if we build the wrong product for the wrong people within the allocated budget and timeframe

Scrum Training

This week we held certified scrum master, 'Writing Effective User Stories', and 'Scrum Primer' courses for about 40 staff which I'd organised through Rowan Bunning at Scrum With Style. We had the benefit of having two trainers for most of the sessions with Simon Bennett in town for Agile Australia. Very interesting and enlightening sessions even for those, like me, who thought they pretty much knew it all. I'm sure it will generate a few posts i nthe coming months as the lessons sink in...

Testers not required

I don't like testers. Let me rephrase that, I don't like needing testers. We had a debate today regarding how to future-proof a legacy system to make it viable for the long term. We've had a number of production issues that have caused us some embarrassment and the project manager wanted to find ways to avoid these in the future. The system itself is a large, poorly architected, poorly coded system with many external dependencies. The code is brittle and without a completely automated build it takes significant time to setup even a manual regression test.

The project manager (and many of the developers) wanted to get testers into the team to ensure full test cycles are carried out before putting a release out to the business. Whilst not completely against the idea I saw this as an attempt to treat the symptoms of the problem (bugs reaching production) rather than the cause (bad code). If we're seriously suggesting using human testers to perform regression tests in 2010 then we're doing something wrong. Testers have a place for new functionality but we're really not producing enough new features - due to the bad architecture - to warrant a dedicated tester just for new work, and in any case we have a BA capable of doing this testing.

This is a simple case of technical debt. Paying testers to regression test is like paying interest-only on that debt - not reducing the principle but still servicing the loan. In order to reduce the interest payment we need to payoff some of the principle - i.e. refactor, replace, re-architect the code. To do this we need skilled senior developers not testers.

But there is another reason I don't like testers, especially on teams with more junior developers. The developers seem to believe that having testers absolves them of the responsibility for preventing bugs getting into production. They can say things like 'well who tested that function - how did this get through?' with genuine indignation. This transforms into a tendency not to test their own code, automate tests, or even think about tests before coding - all things we _require_ to be done on new projects.

So if you find you need testers to stop bugs leaking into production software what you probably need is better software with continuous integration and automated tests - i.e. what you need is better developers. Once you have this in place you'll find you only need BAs and business users to test new features.

Ergo - Don't become a tester!

Behaviour Driven Development

Craig Dunn sent me some reading material today introducing BDD or Behaviour Driven Development.

Basically it explains what I always thought TDD should be, and suggests that BDD is merely TDD 'done right' and with a name change to make us start thinking in terms of behaviour specification rather than code verification. Very worthwhile read...

Philosophy of development?

We're having a bit of a group think about our project management methodology here at pblMedia. A colleague, Peter Kearney, sent the following interesting email which I quote (almost) in full.

Most people don't realise this - but the methodologies we use for ... project management ... are quite remarkably influenced by philosophy:

From the late 1800's to about the 1980's and 90's a philosophical paradigm known as Positivism dominated almost every aspect of thinking and shaped many of the ideas we have about how things should be done today. Positivism is grounded in empiricism and the idea that reality is an objective thing. In a project management context this would lead to thinking as tasks and user requirements as things that can be defined absolutely. You can see that the waterfall method would be a natural response to this paradigm of thought.

Interpretivism started in about the 1920's but really only managed to get serious recognition in the last couple of decades. This philosophy is based on the idea that reality is not absolute - its in our minds. This paradigm recognises that our perception of reality changes depending on the circumstances we are in. In our world - this means that users perception of reality IS reality and that reality therefore changes as users circumstances change. You can see that agile is much more aligned to this way of thinking.

The trick though is to find the middle ground between positivism and interpretivism. Both are right: realities such as budgets are absolute, but realities such as user expectations are not. The ultimate trick in all of this is not to force rigid adherence to a primary scope of work, but rather to help users and stakeholders to shape their expectations to fit the budget and the reality that is evolving in their minds as the system is being developed.

Well I found it interesting anyway. This also reminds me of the Wind and Sun Aesop's fable

The North Wind and the Sun were disputing which was the stronger, when a traveler came along wrapped in a warm cloak.
They agreed that the one who first succeeded in making the traveler take his cloak off should be considered stronger than the other.
Then the North Wind blew as hard as he could, but the more he blew the more closely did the traveler fold his cloak around him;
and at last the North Wind gave up the attempt. Then the Sun shined out warmly, and immediately the traveler took off his cloak.
And so the North Wind was obliged to confess that the Sun was the stronger of the two.

The moral was stated at the end of the fable as: Persuasion is better than force. The history of project management has focused on imposing more and more process and up-front planning and forcing the project to comply with the initial plan. Newer agile techniques have focused on persuading the project to a conclusion, through constant adjustment to changing requirements, collaboration of project team members, and ensuring developed (rather than planned) functionality meets business expectations.

Dividing application work horizontally rather than vertically

My colleague and I took a trip to the Microsoft SDC in Sydney the other week as part of an 'open day', where MS staff and partners described their development and project management methodology - basically a flavour of Scrum. Although a lot of what they are doing is very similar to our own process (which is a good sign I guess) there were a few things I thought worth considering. One of these was horizontal allocation of tasks amongst developers

Basically they advocated splitting the development of functions / user stories into tasks to be divided amongst several developers according to the tiers in the architecture. Whereas we (and many others) normally assign a single function/story to a single developer to code the entire stack (DA, BO, UI) they would assign this work to three developers. The argument they put forward for this was skill specialisation - a data access developer has a different set of expertise than a user interface developer.

Whilst I agree with this to some extent, no developer I know would be happy to be stuck working in a single tier of the app for the entire project.

However thinking about this more, if we did enforce the standard that the tiers of a single story must be developed by different developers what would be the outcome:
1. Developers would have to talk more about the design before coding
2. No single developer would come up with the design
3. Separation of responsibilities of each tier in the app would be easier to enforce (you'd need at least two developers to collude to change the design). Coupling between tiers would hopefully be minimised.
4. Each developer would know about more parts of the application.
5. More stories would be completed within an iteration (rather than having multiple stories still being worked on by individual developers at the end of the iteration, we'd have one or two being worked on at the end)
6. If developers worked on different 'tiers' for different stories there would also be more consistency in terms of coding style - patterns, naming, layout, etc
7. Higher chance of code re-use since developers would know about more areas of the application.
All of these are positive effects, completely inline with the tenets of agile - collective code ownership, collaborative design, pair programming (sort of). I can't really see any downside to this except a co-ordination issue (we need _something_ for project managers to do!), though the proof will be in the doing..

Now I'm off to find a suitable project and some developer guinea pigs...

In defence of custom builds

We seem to be having an attack of platform man at work recently. We've replaced so many legacy third party software with custom built applications that management seem to be questioning the value of all this effort. Now I'm no zealot in this regard (I've even pushed to cancel some development projects as being a waste of time) but it does appear that we've lost sight of the reasons why custom built (as in tailor made) software was attractive to start with.

Here's my take:

1. Expect significant integration work. Unless you're buying a whole suite of products to cover your entire business integration is hard. Many vendors are singing the SOA tune but at the end of the day IT has to figure out how to get data in and out of the application. Of course the big vendors will sell you the tools to do the integration (and the hardware too!).

2. Unwanted maintenance releases. Third party tools require upgrades otherwise they become obsolete. However too often the upgrade provides no value to the business but consumes IT time and resources which could be better spent developing new code, building frameworks, improving existing apps, etc.

3. Wanted maintenance releases. When you _do_ want some additional functionality you need to wait for the vendor to supply it or revert back to (gasp) custom development to plug the gap.

4. Basic support is more time consuming. Take user management for example, every third party tool will have its own user permission management system. Whereas all of our internal, custom-built apps share a common user management service managed centrally by help desk staff, those same staff will need to learn and use a separate system for the third party apps.