For the second year in a row, uTest will be making an appearance at SXSW, the world-famous music/film/interactive conference in Austin, Texas. Unlike last year – where we spent most of our time eating, drinking and schmoozing with storm troopers – we have some new, big plans in store.
The obvious difference is that we’ll be cruising around Austin in the RVIP Lounge, hitting up hotspots, giving rides, singing karaoke (poorly) and playing host to SXSW attendees throughout the week. More to come on that, but you can follow @InTheWildTest for deets on our adventures, and real-time locations if you’re at SXSW..
The other difference is that, instead of just talking about the merits of in-the-wild testing, we decided to show a real-world demonstration. So, over the last 36 hours, we assembled a select group of US-based testers to put the official SXSW mobile apps through their paces. In-the-wild testing means live testers, real devices, imperfect connectivity… basically, true real-world conditions. So we went to work testing SXSW’s official apps for iOS, Android, Windows Phone and BlackBerry. For iOS and Android, we also included tablet testing, to bring the comparison total to six.
Below are some top-level results (note that each category ranged from 1-5):
********
 |
 |
 |
 |
 |
 |
|
| % of Total Bugs | 17.7% | 18.3% | 18% | 6.6% | 23% | 16.4% |
| Overall Score | 4.1 | 4.0 | 4.6 | 4.7 | 3.8 | 4.2 |
| Usability & Design | 4.2 | 4.1 | 4.6 | 4.8 | 4.1 | 4.2 |
| Features & Functionality | 4.1 | 4.1 | 4.7 | 4.7 | 3.2 | 3.9 |
| Application & Performance | 3.3 | 3.2 | 3.9 | 4.0 | 3.5 | 3.7 |
********
Of course, these figures only tell part of the story. As the apps were tested in terms of functionality, performance, design, connectivity and other factors, several issues popped up on more than one occasion. Here were a few areas where some notable bugs were uncovered:
- Incorrect time displays
- Sync issues with registration and deleted items
- Crashes on various tablet OS versions
- Issues with installation
- Social media integration
- Issues with rating and uploading photos
It should be noted that despite these issues, the overall reaction from our community was positive for each of these applications. In fact, the overall ratings you see above are substantially higher than the industry norms, so kudos to the respective dev teams.
Anyway, if you’re at SXSW and want to learn more about In-The-Wild Testing, be sure to stop by the RVIP Lounge. If you’re not able to attend, then head on over to inthewildtesting.com.
