Usability Testing

For this session I had:

  • Usability testing script (the same example I use at work as well, which is made by Steve Krug's instructions in "Rocket Surgery Made Easy"
  • Scenario with 3 tasks
  • Consent form

Here are the notes from the usability testing session I did for the existing app:

Round Trip (Aer Lingus)

  • Uses SkyScanner mostly to search for the flights because she has wider picture 
  • Didn’t do multi-city trips so far 
  • Uses desktop version when searching for the flight because has more info and can open more tabs if wants to search for more flights at once 
  • Likes the fact that she can see prices for more days at once even on the mobile app 
  • When booking the flight, she takes the basic/cheapest option 
  • Doesn’t really like the fact that she has to put all her personal details  
  • Hates dark patterns (tick the box if you don’t want to receive)
    “I hate when they do this! Like, you’re supposed to tick the box if you WANT TO receive, not the other way around! That’s a tricky one” 
  • Doesn’t see the reason to give her phone number because they never contact her via phone 
  • “Why do I have to select the tittle? Why do you need the title? Doctor??” 
  • Would pick the seat only on long destinations, not for the 2 hour flight 
  • “I would remove the phone number and the title” 
  • Likes the “take me anywhere” option on SkyScanner 
  • Would like to have an info for the dates when there are no flights
     

Multi-City Trip (SkyScanner)
 

  • Would use SkySCanner for multi-city trip 
  • Would prefer to take direct flight rather than stop-over, even if it would be more expensive
  • Likes the monthly view with prices
  • Likes the fact that she can choose between different airports in one city 
Usability Testing Session
Usability testing session - click on the image to check the entire video

 

There are also some other important key insight from usability testing sessions conducted by André, that also gave us a great wind in the back to finally start with prototype that will actually fix at least some of the issues mentioned below:

    • Trial/error to find the best price.
    • Struggle with complex/unnecessary input fields.
    • Multi-trip search requires memorization, trial/error.
    • Frustrated when the company tries to force extras.
    • Confusion with price system.
    • Can't book multi-city on app.
    • You can select dates with no flights.
    • Trial/error to search for best prices.
    • Can't edit details later.
    • Seat selection isn't clear.
    • Confusing fields (title / phone).

4 - Usability Testing

It's well known that what users say and what users do can be a completely different thing. And they can't really tell you what they want, but when they see something and use it - they know what they don't want.

Participants

From the data collected during our survey, we were able to more accurately outline who our target demographic is. We wanted to follow this demographic in order to recruit suitable user testing participants.

Frequent Travellers wanted.

Age: 25-45
Gender: Male or Female
Nationality: Any
Technology Skill: Any

 

Materials

In order to get started with our testing, we had to have the materials prepared. We got most of the materials from Steve Krug's "Rocket Surgery Made Easy", especially the instructions for the usability test script.

André created the consent form, looking up to materials Rob provided during the classes, and I found the instructions for the Pre-Test and Post-Test questionnaire in the Usability Testing course I finished on IDF. After 2 sessions, we changed the rate scale from 10 to 7 on our Post-Test following Andrew's advice that might be better to have an odd number so users can find the middle.

We also came up with the scenario containing specific goals and tasks that will help us to conduct successful aimed sessions every time.

 

Usability Testing Materials
Usability Testing Materials

Process

First, we welcomed and explained to every participant what are the session objectives and their role in the usability testing session. We used open-ended and profiling questions to discover more about participants' travel habits and their experiences with Aer Lingus mobile and desktop app.

Each participant was given two tasks:

  • find and book the cheapest return flight to Paris,
  • find and book the cheapest multi-city flight to Paris and Amsterdam (in some scenarios we added 25 kg of luggage).

Guerilla usability tests

Apart from conducting the normal tests in the office or at home, I also decided to do some tests in the Starbucks close to my place. Two visits resulted in getting feedback from 9 participants and in less than 2 hours.

One more proof that usability testing can be done anywhere, and the feedback is equally important as it is from the lab/office/apartment.

Analysing the data

After getting feedback from my 17 (8+9) and André's 2 participants, we decided that the best way to organise them would be through affinity diagrams. We grouped together all important  findings we got in the common themes, and then we voted what to fix.

Affinity Diagram
Affinity Diagram

This helped us to gain a shared understanding of the users' problems and to summarise the results really fast. Also, thanks to affinity diagramming we had a better insight of the common issues repeated between the participants.

Here you can find all the usability testing videos.

From 8 participants, here's the data I analysed from Pre-Test & Post-Test Questionnaires:

 

The most important thing when booking a flight
The most important thing when booking a flight is price, which we highlighted in our prototype

 

 

The most important feature used in the flight app
The most important feature used in the flight app is search for the flights, which is our no.1 focus in redesigned flight app
Overall rating after using the app - from 1 to 7
Overall rating after using the app - from 1 to 7

Final Product

https://www.youtube.com/watch?v=Cxx4eT3zTRM

You can find the prototype and the Figma file separately by clicking on the links.

 

References

Conducting Usability Testing. (n.d.). Retrieved January 25, 2019, from https://www.interaction-design.org/courses/conducting-usability-testing

Krug, S. (2010). Rocket surgery made easy: The do-it-yourself guide to finding and fixing usability problems. Berkeley, CA: New Riders.

Nielsen, J. (2000, March 19). Why You Only Need to Test with 5 Users. Retrieved January 20, 2019, from https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Nielsen, J. (2012, June 4). How Many Test Users in a Usability Study? Retrieved January 20, 2019, from https://www.nngroup.com/articles/how-many-test-users/

Pernice, K. (2018, February 18). Affinity Diagramming: Collaboratively Sort UX Findings & Design Ideas. Retrieved January 20, 2019, from https://www.nngroup.com/articles/affinity-diagram/

Preece, J., Sharp, H., & Rogers, Y. (2017). Interaction design: Beyond human-computer interaction. Chichester (West Sussex, United Kingdom): Wiley.


usability testing

4 reasons why developers should definitely attend usability testing sessions

image source: https://blog.testlodge.com/usability-testing-examples/

Developers should be present on usability testing sessions from the very early stages, even when testing has been made on paper prototypes and mock-ups.

Very important fact here is that usability testing should start as soon as possible in the process. This means that it’s extremely crucial to identify the problem in initial phase, before you start with development. Most of the testing should be happen with mock-ups and prototypes, whether they are low fidelity prototypes or even high-fidelity interactive prototypes in later stages. 

Low fidelity prototypes are great because they give us the focus on the concept and functionality itself, rather than on the details – which are better handled in one of the next phases of the design process. The other advantage is that they also take less time, which means it’s very easy to correct the mistake and solve the problem.

Spotting the problem in design early in the process is cost effective as well. Developers work is very expensive, and you want developers to actually develop rather than spend their time fixing the imperfections. And besides that, it would be extremely expensive to do the redesign, take the existing code out, code again with new design and finally make sure that the new code works properly.

It’s well known that developers don’t really prefer when they see the changes that have been made in a design. They feel like they are doing unnecessary work over and over again, which should be easily avoided in the first place. And I agree with that as well. To fail quickly and early in the process is the key for a good project!

The fact is that everybody should attend usability testing sessions, not just designers and developers. Project managers, product owners, business analysts, entire team should be present to see, hear and feel when the user succeeds or fails in completing the given scenario. 

1. Real time users

When doing user testing, we always prompt participants to think aloud and to let us know how they feel when they are e.g. completing the form, uploading the document or sending a message etc. We want to know what are they expecting when they click the exact button, how are they feeling when the certain action happens and what would they change if they can. When developers are there, they can understand the problem better (if it exists) and see what that means for a user. This situation is for all those times when the things developed are different from the things that are designed. 

For example, they might think that a certain feature, a button size or its position is not so important, but if a user cannot find a way to send a message because the button position is not on the place they expect it to be, they will struggle, loose the patience and in the end have a bad experience. And all because of the small thing that can be easily avoided in the first place.

2. New solutions

Other reason why developers should attend usability sessions would be because of the ideas they can generate observing the users. For example, we had a situation when we presented a prototype form to our users with an assignment to fill it an as they are applying. 

The feedback was absolutely great on the entire user interface and functionality, the form had a conversational approach and it was very simple to use. We had only one question per screen and they could focus easily. The only complaint they had was around the speed of the transition between screens. So, 3 out of 5 users said it was too fast and they didn’t even notice they were on the next screen.

On that same session, one of our developers suggested that we can slow down the transition between the pages when we start with development. He also pointed out that we can add a button animation, ripple effect or similar, so they have a feedback when they click. It doesn’t require a lot of work, but it gives a lot of value.

It was a great experience because we all felt like a good team, all of us working together and for the same goal – build things that our customers will use without any difficulties.

3. Emphasise with the user

Sometimes it happens that developers don’t completely understand the reason behind the designer’s request. That can also be avoided by including them in usability testing sessions. 

At some point they might think there is no sense in building the certain feature because it won’t bring value, or it’s too similar to existing one. But when they hear the real reason from the real users and their feelings and thoughts, developers can put themselves in their shoes and figure out along with the designer what’s the best solution to make the best experience for a user.

For example, we had a similar situation on the previous project when one of the developers didn’t see the difference between using the appropriate keyboard for email input and the default one. I asked him to join us on the next usability testing session where we tested an entire end-to-end flow. 

He was a bit sceptic at first, which is always normal in those situations when people are attending the sessions for the 1st time. Then we started with the first participant, and we got the feedback straight away! Users don’t like extra clicks, actually – users hate extra clicks. Every time they tried to type in their email address, they had to switch between keyboards, look for “@” sign, and then again in order to put “.com”.

They were angry and felt like they were wasting their time, and my colleague developer realized he was wrong. He understood the importance of using the proper keyboard and what does it mean for our user. He also became more conscious about the future requests made by designers, and gave more importance to them because he knew that we don’t ask for anything without a specific reason behind.

The most important thing that I would like to highlight here is the fact that - if developers attend usability testing from the very start - this would completely affect the way they work. They would to be more eager to fix the problems when they feel the compassion for the user. It gives a lot more meaning to their work. 

4. Seeing the bigger picture

It’s quite different to do something without knowing the reason why you do it, it’s like tapping in the dark and using no brain or whatsoever. Instead, on the other side you see the entire story behind every single designed screen, why are they in that particular order and why are they made in that way. Behind every “Why?” there is a proper “Because”, supported with user’s actions, positive and negative feedback and the good and bad experiences they previously had.

In the end, we all know that it’s all about the users. And it’s all about making them happy and careless when they are using our program, app or a service. Bad user experience can be avoided from the start, and usability testing is the right tool if we want to skip that part. 

 

You don’t want to end up having a final product that constantly crashes, or has bad reviews and people actually hate it. Much better option is spotting the problems early in the start, and with having designers and developers included in usability testing in the early stage – a lot of things can be resolved. Developers can give us a great feedback as well, from the technical side of the e.g. an app.
For example, imagine the scenario where business decides to have a specific feature in the task app, a separate counter of the tasks that are marked as “Done” and “To-Do” ones. Designer does his part of the job, designs it and puts it in front of the user. In this case, let’s say that our users don’t have a reaction on the feature, in fact they don’t even notice that it’s there.

Developer, in that case, can give us a feedback from the technical side and say: “Well, from my perspective, this requires a lot of work. I’m going to spend 5 days in order to build this feature. On the other side, we can see now that user doesn’t even notice this feature nor appreciates it. Let’s try to propose something that requires less work and adds more value to the customer.” 

And that’s the best thing. Have a feedback from a user and from the technical side in the real time and all at once. Helps us to make better decisions and act faster, because the time is precious and there is no sense in wasting it if we can use it better! 

So, yeah! Invite your developers to join you on usability testing sessions. It helped us a lot in our projects, and it helped them as well because now they have a different approach to their work. And in the end, it’s all about collaboration and the team work!

 

References:

- https://uxdesign.cc/why-every-developer-should-understand-user-centered-design-7f34e5a32240

- ttps://medium.com/compassionate-coding/user-tests-will-make-you-a-better-developer-ab30c719e2b

- https://www.invisionapp.com/blog/how-to-start-user-testing/

- https://www.nngroup.com/articles/ux-prototype-hi-lo-fidelity/

- https://www.nngroup.com/articles/designers-developers-doing-usability/

- https://www.usertesting.com/blog/2015/11/05/why-you-should-user-test-your-prototype-before-coding-anything/