Release with peace

While I wait for the coffee machine in the kitchen, my phone vibrates. I look at the phone and the notification tells me that today’s #4 release has just rolled out. No stress, I grab my cup and return to the office to finish up my work for release #5.

I would normally be hit by a feeling of panic and anxiety uptil and during a production release. I know many people in tech has it the same way.

In many companies releasing is a huge project taking more than half a day for the whole team. I have often heard the story:

Suddenly something has changed in the market and the requested feature should have been released yesterday. The business pinpoints some new features they want released. The team merges all the requested features into one big pile. A huge manual test is performed and the team hopes they have covered all the changes.

After this the operation team takes over and try to release the new version of the application on the custom configured production environment.

This makes a release very expensive and it will not happen more than once a month or every quarter because of the risk and time usage. Errors will happen, and because of the size of the releases it is hard to pinpoint a new bug and a rollback will pull back all the promised features. You know how the story ends.

But my fear for production releases has disappeared over the last year.

We have right now released to our production environment around 40 times so far this month, without any manual processes, no downtime, and any bad feelings.

By lowering the amount of release content and increasing the release interval, we are right now able to release around 40-50 times to production per month. It all has to do with automation. We are using some of the newest trends like microservices, Docker, Continuous integration (CI) and Continuous delivery (CD), and this has helped us automate most of our development processes.

The use of microservices have made it possible for us to develop and release faster than ever. It is easier to understand the impact of a code change in a small service. And by having good coding standards and test coverage we are able to scale and perform independent rolling releases of every part of our application without downtime.

In the spirit of DevOps we have had huge success by making the person making the code, responsible for the release. It it first possible for him to merge his code into the main branch when the code has been tested on our CI environment and a team member has reviewed the changes. Every time a commit is pushed, our tests verifies the quality by running all the unit and integration tests. When the code are merged our CD flow takes our and pushes the changes to our staging environment where our automated end-to-end tests verify that everything works as expected. Now the release responsible verifies the changes on staging, and by pushing a button, releases the code to production.

As you might know, we are PCI DSS level 1 certified at Reepay. For many companies this would have been an impossible task. But because our many years in the payment industry, we had planned the certification right from the start. The PCI standard requires that we perform additional checks during release and that we run our services on a super tight, secure and locked system. By splitting the applications domain in multiple services it was possible for us to isolate the code with access to sensitive card data and achieve our certification in record time.

These changes have had a huge impact in how we do business. The combination of a mixed team of smart and highly motivated people and the capability of fast releases, have made it possible for us to plan, execute and release features within hours.

For instance, we had a customer asking for a feature in our customer support chat. The change was fairly simple and we decided to implement it immediately, instead of putting it on the backlog. The change was made and pushed to production environment after 15 minutes, and we could proudly respond to a very happy and surprised customer that the feature was ready to use. There is no better motivation than seeing your work being used by customers without having to wait for 3 months.

The whole process has made us more change seeking and stress free. Our huge trust in our automated processes has made it possible for us to push ourselves by releasing faster and on times where we normally would hesitate.

If you want more information and inspiration don’t hesitate to contact me or Ole Borup. Further more Nutcache have wrote a more comprehensive and well written blog post about Continuous Integration.