Our development process was the same for a long time. We wrote features, ran our test suite, deployed to dev, clicked some buttons and then deployed to production. That worked for us until we wanted to integrate our QA team into the development cycle. This is where having one deployable branch (think of a branch as a bundle of code) no longer worked for us.
We have three server clusters for three different purposes. There is the Production cluster, the QA cluster and the Dev cluster. Production will (ideally) always be running and stable. QA is where our new code goes to get checked over by the QA team. The Dev server is what we use for everything else.
In order to develop a process we came up with a few requirements:
- Developers get to push our code often.
- Developers can integrate the QA team into our workflow.
- Everyone always knows exactly what code is on Production.
- Everyone always knows the differences between QA and Production.
- Likewise, everyone knows the differences between QA and master.
Based on those requirements, we came up with this git-workflow:
- Developers work on feature branches based off of master.
- When the developer has finished testing/writing their code, they merge that code into the master branch.
- Jenkins runs the test suite on master after every change.
- During the day we can merge master into the QA branch.
- The QA cluster always reflects the code in the QA branch.
- QA gives sign off to deploy the difference of QA and master to production.
- The production cluster always reflects the code in the production branch.
- Each production push generates a new git tag.
This deploy strategy fills all the requirements we had.
- Developers can push to master as soon as we finish a feature (and we can push to our feature branch whenever we want)
- QA now has a place to test our code
- Since we
git tagproduction, we always know exactly what is on the production server
- Using GitHub’s compare view, we always know the difference between production and qa
- GitHub’s compare view also gives us the difference between QA and master
The two “gotchas” that have come out of this workflow are stable-time on the QA cluster and a weekly freeze on the master branch.
Having time when the code on the QA server is not changing is good for the QA team. Stable-time gives them confidence that the code is not changing under their feet.
The code freeze came about when we had introduced a bug into QA and we could not push to production until the bug was fixed. But in order to fix the bug we had to merge master and QA again. Unfortunately, while the QA team was discovering the bug, we had already pushed new code to the master branch. That means the master branch contained both the fix for the bug and new code which added to the list of features QA needed to check. In order to fix this problem, we introduced a freeze on master just before the production release.
It’s not a perfect deployment strategy and people do make mistakes. In order to reduce mistakes, we added several safe guards to ensure the correct branch goes to the correct cluster. Think simple sanity checks: “are you deploying the QA branch to the QA cluster?”, etc.
I think this deployment strategy has worked well in practice. The most exciting part is that scripting this whole process would not be too hard. We could even present QA with a button and say “just push this button when you’re ready for the code on QA to go live”. I’m looking forward to being a part of building the tools that get us ever closer to a one-step deploy.