Is dependability the determining factor? - A quantitative comparison of the user experience of Netflix, Amazon Prime Video and Disney+

If you've already read one of our other articles comparing Netflix, Amazon Prime Video, and Disney+, feel free to skip the following introduction and read straight on to the "Dependability" section.

The three established video-on-demand (VoD) providers Netflix, Amazon Prime Video and Disney+ all basically promise the same thing: watch whatever you like best at any time. And yet the implementation in the details and thus the success of the providers are different. Netflix currently leads the way with around 208 million subscribers worldwide, followed by Amazon Prime Video with strong growth in 2020 and currently around 150 million subscribers. Still quite new to the market, the provider Disney+ can count 100 million subscribers worldwide (as of 24.06.2021).

Do the numbers of subscribers also reflect the user experience with the product? Why do we like Netflix better than Amazon Prime Video or the other way around? Are there specific things we can name or is it simply the big picture that convinces us more with one provider than the other? Most of the time, we can't put it exactly into words, or it's just a couple of good experiences with the one product that inspires a positive feeling in us. That's why we tried to put into numbers what differentiates Netflix, Amazon Prime Video and Disney+ from each other and present our findings here.

To measure quantitatively, we used the User Experience Questionnaire + (UEQ+). The UEQ measures user experience based on 6 scales. Each scale has opposing word pairs that describe the experience (eg. complicated - simple). The UEQ+ is the modular version of the UEQ questionnaire and allows to choose from multiple scales. In this way, the survey can be better adapted to the respective product, since not all scales are suitable for all products.

The respondents were asked exclusively about their experience with one of the three providers in relation to TV usage. Thus, the results only reflect the experience of TV usage; mobile and PC usage are not included. We asked more than 60 respondents to rate the following five scales, 20 respondents per provider:

Infographic with the 5 measured values: Aesthetics, Novelty, Trust, Dependability and Perspicuity. — The selected scales of the UEQ+ for the quantitative comparison of the user experience of Netflix, Amazon Prime Video and Disney+

‍

The range of the scales is between -3 (terribly bad) and +3 (extremely good). Due to the calculation of mean values across different subjects with different opinions, it is extremely unlikely to observe values above +2 or below -2. In the following article, I describe the evaluation of the dependability scale. The provider Amazon Prime Video scored best in this regard. Why is that? I will get to the bottom of this in the following.

‍

Dependability

“Does the user feel in control of the interaction? Is it secure and predictable?”

––––––

‍

Infographic with the results — In the area of dependability, Amazon Prime Video performs best with a mean score of 1.68. Netflix (1.39) and Disney+ (1.23) are slightly behind.

‍

When using Netflix frequently, one thing quickly becomes clear: Netflix is always trying to improve the interface and the interactions offered and therefore tests a lot. For example, it happens that the language selection in the player works completely differently from one day to the next. However, dependability and the impression of control (security) means no surprises. Frequent feature changes can thus have a negative impact on the user experience.

When looking at the content offered, it is noticeable that Netflix shows an abundance of different movies and series that are highly personalized. In our UX review, we noticed that the order of the cover lanes changes frequently. For example, the "Currently Popular" category is not always in the same place, depending on how Netflix prioritizes its content at the time. This change is clearly due to Netflix's recommendation logic. For a good UX of VoD portals, it has long been advocated to offer as much variety and variance as possible. Viewers should be kept in the application with ever new and different content.

However, this dominant prioritization can also lead to users having a low sense of control. If content is frequently placed in other locations and is thus less quickly (re)discoverable, this leads to uncertainty. The items "frequent feature changes" and "prioritization of content" may thus contribute to Netflix's relatively poor performance on the scales "Unpredictable - Predictable" (1.45) and "Not secure - Secure" (1.32).

If we look at the features that relate to the account itself, we may also be able to explain the scores of the two other scales "Obstructive - Supportive" (1.27) and "Does not meet expectations - Meets expectations" (1.50). The first thing that stands out is that Netflix does not offer a convenience or quick-login function. Users are forced to enter their login data on the TV via the Virtual Keyboard. This is time-consuming and annoying with the remote control. The Amazon Prime Video and Disney+ services, on the other hand, offer login on the TV via their smartphone app. To do this, a QR code is scanned on the TV screen. Users enter their login data on their smartphone or log in automatically on the TV if they are already logged in on their phone.

If Netflix customers want to make changes to their account settings, they can only do so via the desktop. This is not possible on the TV. Netflix could thus be more supportive and expectant in the area of account settings on the TV.

The Amazon Prime Video service offers a few more settings than its competitors in terms of dependability. In addition to the now standard player features, Amazon Prime Video offers the option to adjust the display of subtitles in great detail in terms of font size, type and color. Furthermore, the Player UI displays additional information about the respective actors. On the scale "Obstructive - Supporting", such supporting features possibly bring a positive swing.

Amazon Prime Video is in the upper range for the scales "Unpredictable - Predictable" (1.65) and "Does not meet expectations - Meets expectations" (1.70). This positive rating possibly arises from the fact that Amazon offers content in a less personalized way. The content and recommendations are always in the same place when logged in and out. In this case, the retrievability of content creates a feeling of security and conformity to expectations.

The fact that Amazon Prime Video offers a quick login function via QR code also contributes to the compliance with expectations. Today, no user wants to spend time with cumbersome text input via the TV remote control.

Although the Disney+ service also scores positively on the overall scale (1.23), it is in last place in a direct comparison of the three providers (Amazon Prime Video 1.68 | Netflix 1.39). In terms of content presentation and findability, Amazon Prime Video and Disney+ are very similar. Disney+ also offers the same player features and a convenient login function via QR code. What is the reason for the comparatively lower rating?

I have a theory that the previous experience and familiarity with a service is a factor in the rating. Since Disney+ has only been on the market in Germany for a short time, this is significantly lower than with Amazon Prime Video. However, qualitative evaluations would be necessary to verify this hypothesis and to further analyze the other conclusions.

‍

This blog article is part of the series "A quantitative comparison of the user experience of Netflix, Amazon Prime Video and Disney +". If you're interested in how the three providers scored on the other four scales, we'll link to the articles here.

Who's the most novel?

The importance of aesthetics for UX

How perspicuous does a VOD provider have to be?

Who inspires the most trust?

‍