Thursday, 30 January 2014

Cost of manual VS automated Releasing in Scrum (Agile)

Due to the nature of Scrum, which is an iterative and incremental process, it is required to complete the user stories by the end of the so-called iteration (also known as Sprint). In theory, at the end of a sprint, a shippable product is made available. It can be released to production. This means the software has passed successfully all the quality gates in place at that time. 

However, in order to complete and effectively close the User stories, the features (or improvements or bug fixes) need to pass all the tests. On top of this, the team must know how to release that software to Production (or to the client).

In other words, and the end of the Sprint, the software can only be:
  • Potentially deployed
  • Potentially packaged
  • Potentially released
  • Fully tested against the acceptance criteria (whether it’s manual or automated testing, the latest being the recommended option).
In order to achieve releasable and working software, we can consider two options:
  1. Finish the sprint and test/release after the Sprint (deviation from the Scrum process)
  2. Test and release during the Sprint with the goal to complete those tasks by the end of the Sprint (typical Scrum process)


The first option implicates that the features won’t be tested before the end of the sprint but rather after that. The team will not be able to close the user stories immediately. This is an alternative if the deployment and testing process is not automated and consumes some resources from the Scrum team.
During this transition phase, between two Sprints, the developers and testers will work together to discovering bugs, fixing them, deploying a new version and so forth (until acceptance criteria are met). This process is represented in the diagram below:

Note that the subsequent releases might take longer as the amount of non-regression tests increases (the system comprises of more and more features). Although the manual release process is well known, the testing might take more and more time: 
We can identify some issues with this process:
  • The sprint cannot be closed, as the user stories have not been tested. Therefore the software is not shippable at the end of the work iteration.
  • Issues and bug will be discovered late after the code was written, increasing the time needed to fix it
  • The team is busy doing manual deployment and fixing bug rather than working on the next Sprint
  • The release process takes longer as many more features are added and need to be tested again over and over again.
A better solution is to automate both the automation of testing and the deployment process. This is a required practice of Continuous Delivery.


Once the tests and deployment process have been fully automated, it is possible to continuously deploy and test as soon as the code gets changed. Developers can therefore focus on designing and implementing features. Stakeholders, product owners and testers will have access to the latest version in a Testing environment, soon after the code has been written.
That way, the deployment and testing are embedded in the sprint. See below:
The benefits of this solution are:
  • Bugs are discovered much earlier and easier to fix
  • People are busy working on value added work (designing, coding) rather than manually deploying
  • Repetitive tasks are automated, reducing the risk of errors
  • A shippable product is available at the end of sprint. It might or might not include all the new features, depending whether the team had done a good job at estimating and scoping the Sprint backlog.
The drawbacks are:
  • The automation of the testing and deployment process requires an investment in time and energy. As a result, team will have to acquire new skills for that purpose.
  • The amount of work achievable during each sprint will be less, taking into account the effort to automate the tasks
  • There is still no guarantee that all the features will be completed, tested and shippable by the end of the Sprint. We can only ensure we have the most efficient process to achieve the same.


Due to the nature of the Iterative and Incremental approach of most Agile process (including Scrum), it is necessary to automate the testing and deployment tasks, in order to deliver shippable product at the end of each iteration (Sprint).
Then only the business is able to predict what will be delivered at the end of the next iteration, while keeping the cost of releasing and testing low, even in the long run.

Wednesday, 15 January 2014

Kanban, Scrum and Continuous Delivery: Where do they fit in?

Since the day it began, Scrum is well known for short term project delivery. The strength of Scrum is in its speed and flexibility, much like a sprint running. Within this philosophy, Scrum development process is even called Sprint.
On the other hand, the Continuous Delivery (CD) is known for its long distance running that keeps on producing the code on the pipeline through several “pit-stop” before it achieves the state of “releasable.” CD is well-known for its high-productivity.
The continuous delivery, as the name implies, produces the codes continuously, regardless the code will be implemented in the software or not. Meanwhile Scrum only produces the code which listed on the backlog (Scrum initial documentation).
Let us see how they fit in together and deliver the product that focus on customer as Scrum emphasizes, with the Continuous Delivery high productivity.


The idea of this post came after reading the following article:


The link above talks about Infrastructure and how Infrastructure tasks can fit into Scrum. The truth is that they probably don't. 
Scrum focus on deliver value to user. Therefore, any work that does not deliver directly any value to the user, but is nonetheless key to a successful delivery of working software, might not fit in Scrum.
This is due to the Scrum backlog that requires a user story to be told and the development will run based on that backlog.


In order to understand the issue, we need look back at the ideas of Sprints and commitments. It is a tool to allow the business to prioritize the user stories. And they get to know what will be deployed to Production in the next iteration. In other word, they know what value will be delivered to the business and the users.
We can question though the purpose of committing to sprints for works that do not deliver direct value to the business or the users. 
For example, automating the deployment of the component X is necessary but it is hard to assign a business value to the component, since component X is the value itself. 
So the only advantage to using Sprints, in this case, is to keep the illusion that we are keeping track of the progress.


Another issue can arise: sprints are not flexible enough. As a matter of fact, they are not meant to be flexible. We only develop user stories that are in the Sprint, without add anything, nor remove anything. 
However, sometimes, something goes wrong. For instance the deployment is broken, the testing framework has become too slow or a new process needs to be automated.


The issues listed above cannot always be planned during a planning session. There might be unplanned situations arise, for example:
  • We are not sure yet that we are going to introduce a new component during the Sprint, and the deployment of this component needs to be automated
  • Developer adds 100 new automated tests and suddenly the test suite is running too slow. It takes an hour to get the test reports which becomes a major bottleneck for the team (which relies 100% on the tests before releasing)
  • An automated deployment that use to work stops working. It seems a new version has broken the build, but nobody saw it coming
In some cases, we don't know yet what is going to come in the next 2 or 3 weeks at an automation and delivery level. Most often, it is not worth delaying any further and waiting for the next Sprint planning. 
The Continuous Delivery mechanism works like a car factory. If any of the operation of the assembly chain were to fail, the entire production machine is stalled. Why should you keep producing cars when you cannot put all the screws together?
In the TQM (Total Quality Management) which adopted as Lean Management in America, when the assembly chain noticed an error, there is only one solution: stop the production, fix the error and then resume your work. This is called Poke Yoke, which means error-proofing in Japanese language. The idea of this method is to stop the production when error is noticed, quickly fix the error before the error becomes the defect and defect lead to failure. Using this method, we are able to reduce the severity of error and reducing the amount of cost spent to correct the codes.
In software development, the implementation of Poke Yoke is: stop coding, fix the deployment and resume your work. 
This Poke Yoke requires all code to be inspected before delivery to the next stage (pipeline) of Continuous Development. Normally the CD pipeline consists of: Coding, Unit Testing, Integration, Acceptance Testing and Deployment.
Note that it's not recommended to let developers write too much code when that same code cannot be tested. Nobody wants to have to do big-bang commit and testing once everybody resumes their work.


Another time when developers struggle is the Sprint planning and phrasing technical user stories.
Some bad examples:
  • As a tester I want to make the test run faster so that I get faster feedback
  • As a sysadmin I want to refactor the script that automates the configuration of our web servers
User stories are great because they start with "as a user" and force you to think about what you are going to build for the user.
However, when user stories do not serve directly the user, you will have hard time writing it. This is the time to stop writing that kind of user stories and find solution to “technical user stories.”



The first option is to "hide" CI/Infrastructure tasks into other user stories.
Say we need to need to automate the deployment of the component X and Y. We can add a few more tasks in User Story X, and maybe a few more in the story Y.
Unfortunately, that approach does not always work:
  • you are wasting your time doing this
  • your story Y is not independent and is waiting for story X
Since component X and Y is not a user, which we cannot make a story out of component. Therefore for some cases, this might work, but not for others.


For another project we had a continuous technical user story. The challenge is to explain the Product Owner why your team needs to spend 10 points on a technical story that does not provide any business value.
Unless the Product Owner used to be a software engineer, you will have a hard time justifying it.


In a recent Enterprise size project, we ended up having a "project support team" for any Continuous Delivery work. The CI/Infrastructure/Data Migration effort was so huge; so that we had to separate it out from the main development stream.
In fact we were running a data and infrastructure migration project as well as building an asset search website.
That Continuous Delivery team was autonomous and would reprioritize almost daily, depending on the new issues and the long terms tasks. Their goal was to make it possible to test and release the software at all time. They understood the priorities: serve first, maintain second, and improve later. They need to justify the value of works they have been doing by some management tools.


Kanban is a Japanese term for “Sign-Board.” It is a visual management tools that started in the 1970’s world-famous Toyota Production System (TPS). Its main purpose is to view, track and monitor the process in the production line, This production line in CD is called pipeline. In Agile world, Scrum adopted Kanban as one of their artifacts: Taskboard.
In Kanban board, it divides the status of Work In Progress according to the stage of the work. Starting from the Planning to Execution. In Scrum, the stages are User Requirement – To Do – In Progress – To Verify – Done.

The Kanban board will show exactly the job to do and the workflow instantaneously. A Kanban board is useful for more adaptability when urgent situation arises. In a way, the CI/Infrastructure is there to support the developers. One has to be ready to jump in and take responsibility. That means to make the situation as his priority to fix and improve (like support engineers do for other business units).
Kanban helps limiting the Work In Progress and allows one to prioritize the work nonetheless. Thus, when the situations arise, we can add the situation into the Kanban board so every team member can see the situation, and anyone can take action upon the situation immediately.


Scrum seems easy to adopt for pure software development tasks that are serving users. However, when we are facing a complex situation, we propose to implement Kanban as solution to deal with such situations, such as the following:
  • you are running a very technical project like Data migration or Infrastructure transformation
  • you have automation tasks that are required but the tasks do not serve the user directly
  • you have very technical tasks and are having a hard time phrasing user stories
Depending on the size of the project or the team, Kanban might be of a great help. In any case don't try to use user stories when you are not building a feature, but rather supporting the delivery of the feature.