Lessons Learnt - Over the Years of Iterative Agile Development



I was sat the other day remembering when I completed my first Scrum Master Certification training course back in 2010. Hosted in central London, it was my first formal training in Scrum. Apart from the obvious feeling that I was getting older and, hopefully, wiser, I started to reflect on some of the things that I'd learnt since the course. I've been fortunate enough to work with teams across different industries spread across countries and organisational functions, so have a wide variety of experience in applying agile principles. This includes the use of the Kanban process, as well as Scrum. I noted down a few of my thoughts on some of the main learnings from the course and how I've seen these used in practice over the years.


1. Self organisation and servant leadership


I remember this a big learning point for me in the Scrum master training course, this concept seemed a big change from traditional project management. Role-play exercises were used to illustrate the difference with traditional project management by emphasising a more self-organised way of working, for example with team members defining and selecting individual tasks from a board based on a goal, instead of a project manager maintaining a list of tasks and assigning them to the team members. Whilst the benefits of this way of working, made a lot sense to me at the time with my background as an independent software developer, some of the organisations I've worked in since have more traditional project management processes and are in transition to more agile ways of working. Where team managers have adopted this approach, I've noticed the individuals are happier and work together more effectively, as well as maintaining more positive relationships with stakeholders through more timely delivery, improved quality and an ability to handle changes in requirements more effectively.


2. Frequent delivery of working software

This is surely one of the most important agile principles; the ability to deliver working software, early and frequently. Whilst the introduction of the scrum process can be a good start, this can't be seen as a silver bullet. Far from immediately resolving all the team's process issues, I've often found this results in greater visibility of the problems and the issues in the current processes. I found these can be divided into 2 main aspects; the ability of the team to communicate and collaborate to support a more frequent iteration with business feedback and secondly the technical processes required to support agile ways of working - for example, is the design and architecture of the system flexible so that it allows changes to be made easily based on feedback from the business, is there a release and deployment process that allows changes to be released more frequently (or continuously) ? *

Team performance can also be affected by wider organisational challenges or blockers, such as project prioritisation conflicts, dependencies on other teams or financial approvals. So it's important these are taken care of by the Product Owner/Scrum Master(or Project Manager) so that the team can focus on meeting their sprint commitments. The product owner or other key stakeholder should have sufficient understanding of the business requirements and goals to help the team make important decisions for example to resolve uncertainty over requirements, striking the balance between new features, technical debt and production support or giving business input into decisions around technology and infrastructure choices.

Some items in the backlog may not require as much iterative feedback from stakeholders, for example infrastructure related projects or data projects that don't have a front-end that can be shown to the customer to get feedback iteratively. These can be broken down into smaller tasks so that they can be estimated and tracked against agreed dates; though often still benefit from iterative review to ensure items stay on track. 


3. Iteration 

Iteration is an important part of agile as oppose to waterfall approaches that rely on a large up-front planning with a lot of assumptions, captured in an up-front project plan. Instead requirements should be broken up into small stories that can be developed in a shorter time frame, ideally within a sprint. User stories, as oppose to overly detailed requirements documents, better allow for changing requirements, since they use a shorter, more concise description of  the main user requirements. This prompts a conversation between the developer and product owner, rather than attempting to list all the requirements to a high level of detail when they are not known at the initial stage. At the end or during the iteration, feedback is captured at the and factored into existing or further user stories, if necessary, for the next iteration. Iteration, or sprint length can be up to 4 weeks though in general, I've found the 2 weeks iterations used more often. In one case we moved 4 from 4 weeks to 2 weeks which helped the team focus and allowed us to get feedback more frequently from the stakeholders.

4. Customer feedback 

Demos at the end of the sprint are a great way to get feedback from customers and users. Often it maybe that not everything in the sprint is demo-able so it's important to ensure that the developers are ready to give the demo for agreed set of items - this could mean preparing an agenda with the team and running through it before the demo. This might avoid any issues for example ensuring and tech set-up in done in advance so the demo runs more smoothly. For some sprints, we've had stakeholders in the demos which has the advantage of feedback but others we've not had demos or they've been scheduled at later stage, when things are ready. The important thing, is to maintain frequent contact with those who can give user feedback to get their feedback, as often as possible. So if a demo isn't possible, maybe a meeting within the sprint can be used instead to get feedback on items in design stage or to clarify uncertainty over a requirement. 

4. Responding to change over following a plan

I remember an exercise from the course, a team game where we where given some rules on how instructions could be passed between team members to get from one point into the room to another -- one team had a short discussion and then started make their moves around based on the instructions. The other spent longer trying to discuss and make a more complete plan of all their moves. The team whose members started to move earlier, easily won, illustrating - sometimes, it's better to move forward early to the next step. Further steps will then become clearer, rather than attempting to plan all your moves up front. Scrum emphasises planning the next sprint with the most clarity as this decreases as you head further into the future so it becomes harder to plan. Certainly in the context of a sprint iteration, there should be a clearly set of prioritised stories for the sprint backlog though it's important not to obsess over planning every detail of each up-front, keep stories simple and concise where possible. Once development has started, it's important to get feedback from the business as frequently as possible to ensure that what is being created has value to the business and development effort is not being wasted on changes that do not have any benefit to the users.

5. Delivery of value, through a potentially shippable product 

Agile emphasises the frequent delivery of value through sprints with a potentially shippable product at the end of each sprint. In order for the product to be potentially shippable, the features in the sprint that form part of the shippable product usually must have completed some form of QA and user testing. Typically it isn't easy to do this efficiently within a sprint. It can be a time consuming process to ensure that all the requirements have passed their tests and met their acceptance criteria, after development has been completed within the sprint. In some cases, I found the QA function may not exist or maybe new to the organisation and QA falls to the project management or business analysts teams, though typically I found this doesn't replace having a dedicated QA person who understands the system in detail and can work directly with the developers. The introduction of a TDD or test-driven development approach has been helpful in prioritising requirements and ensuring that test scenarios are written prior to development. This has included introducing user-stories, with clearer acceptance criteria and user benefits statements and the adoption of a BDD (behaviour driven development) approach to writing test scenarios. 

* Continuous delivery, as mentioned in the Principles behind the Agile Manifesto