We have been discussing different aspects of mobile gaming development, testing, and other required architectures. This has helped us to create more robust mobile games. We are grateful for all the great feedback and encourage you to contact us. Let’s now look at the importance of testing in the development process for mobile games. What are the top metrics to consider?
Crossing the Mobile Game Development Chasm
Mobile game testing is still a lot of manual work. This makes it difficult to cover all the bases. Effective mobile game testing requires a structured and systematic approach. You should also use test automation frameworks and seamless integration to your agile process.
Many test automation frameworks were originally designed for native apps, or web apps/elements. One of the most common misconceptions among mobile game designers is that test automation frameworks don’t conform to game testing – certain native UI elements, such as their IDs and characteristics, are not allowed. It is not possible to immediately identify. This is not the case. Many of these frameworks offer excellent ways to test mobile apps. You can use image recognition.
Testing is a simple process where bugs are identified and documented to allow developers to remove them. Software testing metrics such as number of bugs discovered and number of test cases created are important but don’t show the value of testing. This makes QA hardly a productive and worthwhile effort for organizations. Thanks to the instant feedback from gamers via App Markets, and concerns about user retention has transformed how serious mobile game development teams think about testing. Bad ratings and negative feedback can lead to low downloads, eventually leading to the game becoming a rogue investment.
Can crash reporting be helpful? Yes. However, this only gives you information about whether your mobile game crashes, doesn’t work or causes other problems – . It’s already too late. Negative feedback has been made public and it is not always possible to fix the problem. Bad ratings/reputation may even prevent from getting the next title approved or from getting that title before hundreds of millions of gamers.
Three Factors to Improve Mobile Game Testing Value
The value of testing in the development process can also be seen in effectiveness, speed up the development (improved production), and the possibility to release your title earlier (time-to market). These ‘improvements” can be directly measured in terms of their value.
Your QA team’s effectiveness in finding and fixing defects. This is the number one cost driver in mobile app development today. This is the number one cost driver in mobile game and app development today. Direct costs of defects can have an immediate effect on your bottom line. These include the cost of fixing defects in testing, the cost of fixing and verifying crash report reports from the market, lost customer acquisition costs (CPI), and the loss of lifetime value (LTV), which will impact top-line.
Too often, efficiency is overlooked as a cost driver. But if you are honest with yourself, you’ll see that bug-fixing late in the process is taking a lot of time and causing delays to the release. This leads to lost revenue.
High costs of time-to-market directly impact your bottom-line. It is important to have the right time, not only to compete but also in order for you to get revenue as quickly as possible. Every day is important when you launch your game. You start generating revenue earlier. You should be able to measure your monthly/daily acquired users, MAU/DAU, and average revenue per user (daily active), (ARPU, ARPDAU). If your development process allows for agile development and testing, delays in publishing won’t be caused by testing.
Many game developers don’t calculate the effects of delays. However, it is very easy to calculate each day’s impact on your bottom-line. Manual testing – agile mobile game development process was discussed and we learned how to maximize the benefits of this approach – and improve your QA.
How can you add value to the game development process?
Efficiency. It is the easiest way to address the issues related to effectiveness. Integrate game development and testing process to create a cohesive agile development system. This enables you to automate a huge number of different types of tests, using real mobile devices (all sorts of those) and check code after every build (daily/nightly/weekly builds). These are the benefits and attributes that make this effectiveness:
1. Mobile development offers instant feedback on defects. Developers will find instant feedback on defects a huge benefit, as you can still use the code that you just touched. Who can remember what you ate for lunch two weeks ago vs. what you eat today?
2. Field failures are less likely to occur if defect detection is done before the release. Let’s look at an example: A mobile game with 50,000 monthly users makes 2,200 crash reports per month. That’s 6,600 crashes per monthly if you have three games. If the analysis-fix-verification takes around 1h, the cost is huge. It’s not because of engineering costs. But the cost to reputation, lost users, and potential revenue loss.
The game would have been more stable and reliable if it had been tested on real mobile phones before release. This would have made development easier by an additional 2-4. Monitoring after launch is still important, but it’s best to not go into reactive mode when it comes to finding defects.
3. LTV and CPI lost. You probably know the cost of each install and how much your users pay. It is very likely that your game will fail after a release. You’ll lose both the CPI and LTV for that user/all users who use the device model. To understand the consequences of any crashes, it is highly recommended that you plug in your CPI and LTV for every crash from the field.
Proactive development using continuous gaming testing on real mobile devices will result in a higher top-line as well as happier developers. Every developer wants to focus on new features and not fix existing ones in firefighting mode, based on market crashes. Remember! Manual testing is not agile and is not the answer to these issues. Instant feedback is not possible. It is almost impossible to get instant feedback if you need to test your game on the top 50-100 devices.
Time to Market. A shorter time-to market will increase your bottom-line. You can generate revenues faster if you launch your game earlier. Your new game or an old game with new monetized elements is released 2 weeks before the previous release cycle. This allows you to save 14 days using automated mobile testing and agile combined with manual testing. It saves you 84 days annually if you release 6 times per year (each 2 months).
It is very easy to calculate lost revenue.
- Imagine 100,000 users per day (DAU).
- Assume $0.05 average revenue per user daily (ARPDAU).
- 14 days (x6) 6 release/year (x100 000) (x0.05 = $420 000/year/game) to increase your top line.
- A portfolio of five games would yield a total value of around $2M. This is faster than the time-to-market.
All this can be achieved by reducing 2 weeks from each release cycle. Some games generate direct revenue, so you can replace ARPDAU by the value that each mobile user requires.
With integrated testing and development, the time-to-market can be improved. It is true for all software development. However, mobile app development is unique in that it allows you to develop your apps efficiently. End-user platforms are numerous (e.g. There are many end-user platforms (e.g.
Automating a lot of testing can be done on real devices, after each nightly build and code check. This will reduce your development-testing cycles, increase defect-fixing time and improve your top line.
Your competition will be unable to catch up with you because they have a faster time-to-market. You won’t be able to promote your offerings as much as your competitors, which can lead to your releases falling behind. With more confidence in your quality releases, you can release more often.
Productivity. Quality assurance (QA), is an essential part of creating and maintaining mobile games that are popular with hundreds of millions of players. The productivity of QA is a key metric in the game creation process. This refers to the ability to quickly find and fix problems in games before they reach users. You can measure this type of productivity through a variety of metrics.
- Rate of bugs found – How efficient the QA team locates bugs during testing. This is based on the time spent testing.
- Rate of issues found post-launch – How many issues were discovered after the go live release
- Customer satisfaction rate – complaints to customer service, issues reported via app stores, overall rating of the game
- Developers can easily understand and fix issues based on documentation and quality of deliverables.
- The number of test cases that are added each time.
Device diversity, various operating systems versions, hardware configurations, and OEM/carrier customizations are all challenges that mobile game development &QA faces. These factors increase complexity and lead to longer development cycles (times-to-market), higher field crash rates (lower customer satisfaction, uninstalls hurting your bottom line) and higher customer support cost (more customer complaints) as well as inefficient development teams (dealing with customer issues vs. creating new features). Teams need access to real devices in order to increase QA productivity. These devices should be the same ones end-users use.
Developers and QA team members need to have access to these devices when they encounter an issue during development. They also need it for post-launch issues, or customer complaints. The cost of fixing the issue is lower the earlier they are discovered. Automated testing can be used to maximize bug-finding efficiency. It can be done on real devices, after each new code check and/or nightly builds. This will reduce your development-testing time, increase defect-fixing speeds, and improve your bottom line.
Use cloud-based service to mobile development and quality assurance. Your team’s performance will improve dramatically. Developers are happier (instant feedback, no need for purchasing new devices), product owners are happier (higher customer loyalty and faster time-to market), customer service is happier, with fewer customer complaints and, most importantly, your customers are happier because they don’t get frustrated by “not working” on their device.
Conclusion
There are many practical ways to increase the value of existing mobile game development processes. Focusing on these three things and following the agile process with continuous test will quickly bring you benefits and allow you to create more robust games. This is how to get better quality.
- Automate most of the testing by using any of the open-source frameworks for testing available on the most popular mobile platforms
- You can avoid vendor lock-in completely by using standard languages like Java, perl and php. There are millions of developers who use these languages.
- Automate as many of the testing as possible to increase organizational agility. Developers can even be responsible for automating tests
- Continuous integration and continuous testing are two of the best practices in agile. They will give you results 24 hours a day.
- To improve collaboration and transparency within the organization, integrate with your test management system. Collaboration must be possible with tools
- Instant accessibility. Development teams don’t have the time or resources to deal with lengthy sales cycles and procurement processes for each device and tool. You can choose a service that provides rapid provisioning, without the need to wait for hardware, procurement, or other internal processes.
- The most important thing is the last: Tools are important, but it’s more about how you use them. Your success will depend on how you use tools.
Happy Mobile Game Testing!