Guest Post: Slow mobile app = Millions in lost sales [Part 1]

New HeadSpin Holiday Season Site Speed Test: 60% of top etailers too slow

App speed is widely understood as a key UX metric. And more companies are grasping that “load time affects bottom line.” But how, more precisely, does app performance impact online sales? That was the big question behind HeadSpin’s new Holiday Ecommerce Site Survey.

It’s a big question. Good news: Mcommerce made up 34.5% of $106 billion in 2017 holiday season online sales, according to eMarketer. That share is expected reach 50% by 2021.

Forrester predicts U.S. retail sales from mobile devices will break $280 billion by 2021, so the stakes will only get higher.

But oh-oh, Akamai research shows a 2X increase in mobile load time cuts conversions by 50%. The firm found 53% of mobile shoppers will bail at three seconds. Sweet spot: 1.3 -1.4 seconds.

So we wanted to know:

• What does a one-second load delay cost in lost sales?
• How bad is the problem at real companies?
• What are the root causes and best remedies?
• What are the fastest phone and Android OS combos?

To find answers, we recently tested mobile apps from 36 major retailers. We then calculated the potential impact on each brand’s online sales. We got some interesting results, which may surprise or even shock you.

Some highlights:

• Brick and Mortar (B&M) brand apps had slower average load speeds (3.07 seconds) than online only (Oo) brands (2.09 seconds)

• The slowest-loading app (8.8 seconds) reduced the retailer’s online sales by an estimated 43.7%.

• The fastest-loading apps (<0.4 seconds) boosted sales an estimated 23.3% for top-performing companies tested

Keep in mind these are our calculations, not actual sales figures. But they’re fresh, real-world data illuminating a widespread, previously hidden challenge.

The bottom line is pretty clear: Companies with fast-loading apps can boost online sales and avoid abandonment by frustrated and impatient shoppers. Slow-loaders, the opposite.

In part one of this two-part post, we’ll take a closer look at how we did the testing and at detailed results. In part 2, we’ll look at common causes of slowness in Android apps, and what developers and companies can do to avoid getting Grinched all year long.

HOW WE TESTED

Targets

We selected our study group from the top ten brick and mortar and ten online only apps listed in the Google Play store.

Test Phones

We choose a good representative mix of Android phones from HeadSpin’s pool of testing devices:

All operated in Wi-Fi-only mode. Like the choice to test in New York city, this was done to avoid the risk of introducing latency into the equation.

What We Tested

We decided it would be most useful to test cold start response time. To refresh on terms: A cold start occurs when a user runs your app after not using it for a while. In this stage, Android probably has removed the app from cache to save memory.

[In contrast, first start is when a user runs your app after a fresh initial install. During first start, Android or your app does some extra work, such as initializing SQLite. Warm start occurs when a user switches away from an app, then switches back. Your app is still in Android’s cache, so warm start is typically fast.] Because cold start is the most typical case and affects UX the most, that was our test choice.

Lastly, we needed to choose a benchmark target load time. Research shows humans can detect lags as short as 22ms; a fourth of the population can perceive lag between 2 and 16 ms. For our tests, we settled on a two second load time, based on studies of app users and abandonment, including this citation by Akamai. This is reasonable, and .6-.7 s more forgiving than the optimum speed for conversions cited above.

Where, When, How

Testing was done November 2018 at our data center in New York, NY, using NimbleDroid. To minimize latency between the remote connection, we selected devices from HeadSpin’s local New York device pool. That keeps load impact below 1-ms.

Test were manually conducted on two or three apps at a time and monitored by a human operator for errors. Each batch of results took about six hours to complete.

We used a heuristic to tell when an app finished startup by detecting when:

1. the main Activity has been displayed, and
2. things like animated progress bars in the main Activity stopped.

To identify the main Activity, we ran the app at least twice, waited for up to 30 seconds, then checked the last Activity displayed. If the result was consistent across multiple startups, it was designated the main Activity. Once identified, the app was started and measured multiple times, and the average reported. Apps that did not allow resigning and on rooted devices were run manually.

AVERAGE RESULTS: SUMMARY BY PLATFORM

Go time. Here are detailed results by type of phone and retailer:

A few things pop out. For phones, the biggest gap between slowest and fastest loads was 2.72 seconds (4.46 seconds B&M vs 1.38 average for Oo) on the Motorola E2 and Google Pixel 3, respectively. Second, average load time for brick and mortar retail apps was 0.62 second slower than for online only apps. Neither category average beat our target two-second load time, though individual brands did. More in a moment.

Here are results by OS and category. You’ll in see that Android 6 at its quickest was nearly four times faster than Android 4.

RESULTS: BY BRAND

As expected, average load times for brick and mortar stores were .8 second slower than online-only.

Here’s a graph of results. You’ll note the wide range between fastest and slowest. Best-of-breed B&M’s got very close to top OO’s.

You can see the speediest B&M site (Target) was nearly 7x faster than the slowest (Bed Bath and Beyond). The fastest OO site (Stitch Fix) was 5x faster than the slowest (Mercari). Ecommerce titans Amazon and Alibaba exceeded three seconds.

CALCULATING SALES IMPACT

Once we had load-time data, we were ready to calculate how those performance numbers impacted sales. Prior to the test, we’d found and analyzed several eye-openers: Google’s discovery that an extra .5 seconds in search page generation time dropped traffic by 20%, meaning a broker could lose $4 million in revenues per millisecond. And this: Amazon found every 100ms of latency cost them 1% in sales.

We chose the latter as our formula to calculate % loss in sales:

Assume every 100ms delay leads to 1% loss in sales.

1,000ms = 1 second

From that, we derive:

100ms delay -> 99% sales (loss of 1%)

200ms delay -> 99% * 99% sales (loss of 1% after losing 1%)

5s delay -> (99%) ^ 50 = 60.5% sales (loss of 39.5%)

As you can see, things get uglier and uglier pretty quickly, and disastrous not soon after that. Armed with best-available formula, it was time for the moment of truth.

RESULTS: LOAD SPEED IMPACT ON ONLINE SALES

And there it is. Follow the link below to see the detailed results below, and you’ll see a clear correlation between app load speed and online sales. The data speaks for itself. (Keep in mind when reading that a negative sign in the two far right columns actually indicates sales growth). Note that load time difference between the lowest sales gainer (1%) and the highest (23.32%) is “only” <1.5 seconds.

One quick reminders and a request before you dive into the complete list:

In part 2, we’ll look at five common causes of app slowness and three proven remedies.

See Full Results Here

Got a comment, suggestion or tweak to our study? We’d love to hear from you. Twitter: @headspin_io