Challenges of testing a live app with millions of users

Software testing is not, nor it was ever expected to be, a strict discipline in the world of software development. While software development is continuously evolving in all its aspects, including languages, methodologies, and application purposes, the user base is growing exponentially. This leads to a situation where many applications are used by a large number of users on a daily basis.

As a software tester, I have had experience with XML/SOAP from web services testing, and also experience in manual and automated testing of web applications, since I worked on several, all of them being startups, however.

I will try to describe what I experienced once I found myself faced with a million users and what I suggest for others to experience.

Teamer is currently the application that actually consists of two segments, one of which (Teamer) is used for sports team organization, and the other one, called TeamerPayments, is used for managing financial parts of sports teams. Still, both of these stand as a single application under a single URL. Teamer allows you to create and track teams, events, photos, documents, chat, collect money, collect information, and much more, whether you are a regular web user or you prefer Android or iOS mobile devices.

Therefore, as a very useful application, Teamer is currently used by over 2.5 million people worldwide on a daily basis.

This set of functionalities was not there once we came in, because I remember TeamerPayments was just started then, iOS and Android applications were there with only a basic set of functionalities and all of it seemed like it was a few years late. Together with TeamerPayments, a set of functionalities, including money collection, was supposed to be transferred from Go Cardless to Stripe. However, even then, Teamer had over 1.5 million users.

None of us who started working on Teamer had any previous experience of working with a live system with so many users and it was new to all of us.

I remember the first thing I had trouble with was functionality. Teamer had just so many functionalities and options on the web and some of these accessible from more than one place. I remember saying that “Teamer seems to have everything from everywhere”.

The other thing was the amount of pressure that grew in just a few days. I believe that it is in human nature to see the same things in different places in completely different ways. This was the case with me. Working on Teamer seemed completely different from any other application I worked on before only because Teamer had so many daily users.

Another thing is the team. As I mentioned before, none of us worked on something like this before and none of us were completely comfortable while working. And in order to build quality, you need a certain amount of comfort.

After a couple of months, new developments occurred. Since Teamer is a worldwide supported application, and the world is not the same everywhere, we had to learn some rules in languages, political and geographical facts we did not know before.

All of that referred to software and rules that make it. Of other things, I remember that Teamer had a production environment and we did not know how it looked like. There was no operational testing or staging environment, and everything was tested on the local environment. From a local machine, you went directly to production. There were no guidelines on how to manage code. Basically, you knew that the ‘master’ branch is the production one and that was it.

So from this you can understand what we were faced with once we started working on it.

As time passed by, these issues were tackled and I believe we have a truly nice process in place now. The amount of pressure dropped significantly and I will try to summarize some guidelines on how we managed to do that.

Put together a working process

Being aware of the situation we were in, the team organized a very good working process of development. Stories and bugs were worked on a separate branch each and only after they were tested on their branches locally, they were merged on a newly created ‘staging’ branch. Now, the ‘staging’ branch is something that acts like “a ‘master’ branch if everything works well”. One story was tested, it was merged to staging where it was tested together with all other things we did or fixed. Only if everything works on the ‘staging’ branch, it is ready to go into production.

Decrease a gap of going from local to production

One of the next things we did was that we learned there was actually a deprecated environment not used for almost a year. We dedicated some time to it, fixed it, and set it up in order to act as a Staging environment. So from there on, we had a pretty good Staging environment to use as a Production mimic environment.

Learn your functionalities

When you work on such an application as Teamer, it is not enough to know what something should do. You have to go deeper and learn how it does it. Let’s not confuse this with whitebox testing. You do not have to go into single method details, but you should be aware of how something works, so if you change something in one place, what other things could potentially be affected. Especially when working on web and 2 mobile platforms. Yes, before you go to production, you will regression test your application, but you should be aware of which parts you should pay the utmost attention to.

Go “out of the box”

So you have 2.5 million users from all over the world. It is completely normal that you expect that some functionality will be used only to serve its original purpose, but considering the circumstances that is not completely true. You have to think differently, go “out of the box”, be innovative. Your users do not think the same way and will use some segments of your application for things that are not focused on it. You would be amazed to see what teamtalk messages and its comments are used for. Just imagine different backgrounds of your users, from education, culture, computer skills, sports skills, approach… and you’ll know what I am talking about.

Try to love what you work on

Now I had luck and it quite helped that I really love a variety of sports and I was playing a couple of different ones for some time. Understanding the basic idea behind your software could sometimes be crucial.

The most important thing is the team

It has to think the same and strive to achieve the same goals. It is of the utmost importance that all members of the team, not only testers and developers, share the same passion and responsibility. Only then you will be able to trust each other and work it out when things are not going smoothly. The improvement of Teamer, since we started working on it, has been accomplished owing to every single person included in the development of this project, from the guys in the United States and Ireland to us here in Atlantbh.

Teamer is now a much bigger and better application with many new features. It grew from less than 2 million users to over 2.5 million users in a year. It moved from a basic functionality mobile application to full functionality one. It experienced 1 major and 1 minor redesign in this period. It successfully incorporated TeamerPayments, and now it processes hundreds of team payments every day. It successfully moved from one Push Notification service provider to another. It successfully supports Spanish, French, Italian and German now, … and much, much more.

And things are improving faster and faster as we are trying to learn something new every day, whether application or technology related.

I sincerely hope that this will help many of those who find themselves in similar situations or at least give hints on what to do to overcome any obstacles and build a better software while working with great people in a superb environment with as less pressure as possible.