Organising Distributed Teams Working on a Single Codebase
Collaboration has long been a main ingredient to high-performing and successful businesses. One of the main table topics today is how to encourage team collaboration in fully remote or hybrid mode of work. To add to the scope of collaboration challenges, projects can be a part of a large, single, codebase to which several dislocated teams are contributing to; teams which use different approaches to work.
In this case study, we will describe the challenges we were facing on a project in which multiple teams from different organisations were involved. Each team had its custom sign-off and release procedures, but the code had to be synchronised into a single production release by the end of each cycle.
How it started?
In this particular setup, three companies with several teams joined the effort on an existing project. The existing software supported a set of features which were shared among services from different business domains. Shared features, as well as services, were all part of a single codebase. Each team was in charge of a particular service, including design, development and testing. There were already established guidelines in terms of procedures and processes all teams were to follow.
Main challenges
Inherited processes were designed in such way that two versions of the software were worked on at the same time. The focus of the QA team was on the version that is released to Production next. To simplify, we can call it Version 1.0. QA team had two weeks to perform testing for that version of the software. During those two weeks, Dev team was developing features for the version that comes after (Version 2.0).
Visually represented, the process looked like this:
However, inherited processes seem to have several drawbacks which influenced overall efficiency:
- Long release cycles – the time between a requirement is received until it could go to Production at the earliest was effectively one month. That seemed too long for a software so actively developed and widely used. Users would benefit from a shorter release cycle. On the other hand, development teams always benefit from the shorter feedback loops.
- Different goals – with QA team working on one version of the software and Dev team on another, the cohesion between those teams was broken. They worked on different goals at the same time, which was exactly the opposite of what Scrum stands for – the whole team working toward a single sprint goal.
- Context switching – switching between two versions of the software was inefficient for everyone on the team. Dev team was supposed to focus on the features for Version 2.0 while fixing the bugs for Version 1.0. It always felt like the QA team had to make an extra effort to draw Dev’s team attention to 1.0. For a PO, prioritising between the two versions was also a challenge.
Additional problem, not directly related to how processes were designed, was the fact that a bug found in one of the services just before the release would postpone the release for all teams, so it was difficult to establish a regular release cycle.
Our first proposal was to have all teams use a regular 2-week Sprint, which would include development, testing and integration stage with other services. However, introducing traditional Scrum proved to be much harder task than we expected as it requires a significant shift in organisational culture.
In order to make the changes we rooted for, a different approach was necessary. So we had to be creative in order to come up with the process design that will work for all.
Getting to a compromise
Some teams decided to keep the release cycle of 2 weeks for development + 2 weeks for testing. At the same time, teams here in Atlantbh started with regular 2-week Sprints during each of those phases. Everything related to the (e.g.) Version 1.0 (development, PR review, testing, bug fixing) was done in one Sprint. This extension of the processes can be visualised as:
Even though the idea itself doesn’t sound particularly brilliant in its essence, things got a bit more complex with establishing git rules, having in mind that all teams still committed to a single codebase and used shared features. This is how git flow was described for Scrum teams:
- Sprint Day 1:
- service branch is created from the latest develop branch. (Note: Service branch was regularly updated with latest changes from develop, so that changes made by other teams are picked up).
- service branch name convention: {service_name}-{software_version}.
- Sprint Days 2 to 7:
- Service-specific pull requests are reviewed and merged to the service branch.
- service branch is deployed to each team’s DEV environment, where its QA team verifies features.
- After all features are tested in service’s DEV environment, the service branch is merged to develop.
- Sprint Day 7: Final day for merging service to develop branch.
- Sprint Days 8 to 10:
- Regression testing is performed on develop. All bug fixes PRs are merged directly to it.
- At the end of Sprint, develop is merged to main branch and deployed to Production.
Other teams (with one month release cycle) used following git flow:
-
- Development cycle (2 weeks): Pull requests are merged to Project’s feature branch.
-
- QA cycle (2 weeks):
- feature branch is merged to develop and deployed to QA environment for QA teams to start testing.
- At the end of QA cycle, develop is merged to main branch and deployed to Production (same step as for Scrum teams).
- QA cycle (2 weeks):
(Note: 1st day of QA cycle is equivalent to Sprint Day 1 for Scrum teams.)
Conclusion
Even though we made this hybrid model work for some time, ultimately it was decided to split the codebase per services. For most teams this was a significant effort of several months, but it payed off in a long run by eliminating the issue of blocking each-other releases. Individual teams could have independent release cycles and Production deployments. Also, the effort resulted in a codebase that is much easier to maintain and onboard new people.
Still, going through different collaboration models taught us a few valuable lessons. Just as we incrementally improved our software, we learned how to incrementally improve our processes and achieve changes in a complex project setup. It took some courage to initiate changes across multiple organisations, but it was also a good practice of open communication and adaptability skills as means of achieving a desired goal.
Ultimately, we have reached the model that works best for all the teams involved. We still collaborate very successfully on the same project but with a high level of independence.
Elixir Migrations using Ecto
Protractor parallel execution
