How to Use Ethnio for User Testing
At Lucid Agency we perform user tests at critical junctures in the web development process. At the project outset, we complete an analysis of the competitive landscape and do a deep dive into the current site analytics in order to gain a high-level understanding of our client. While this is a great jumping off point, it’s really not enough to understand a client’s businesss. If you want a site that meets the needs of a target audience, you need to talk to that target audience! To this end, we develop and administer surveys to obtain quantitative date about our client’s site users. But again, quantitative data by itself is only half of the story. The rest comes from getting face to face (or at least voice to voice) with a user, asking them about their needs, their goals, their experience, etc. We typically continue to perform user testing throughout the project: quick-and-dirty validation tests in the wireframing stage and development stages and thorough validation tests upon project completion.
Anyone with experience in user testing will tell you that the hardest part is actually getting users to talk to you. Luckily, we have found a great tool that allows us to pinpoint real site users: Ethnio.
What is ethnio?
Ethnio is a service that allows you to:
- Recruit users to participate in testing. This is done via a screener, explained below.
- Schedule tests.
- Perform the user tests. These Can be done via a phone call or through the join.me integration.
- Incentivize participants for performing user tests.
To recruit testers, you must first build a screener. A screener might look something like this:
The screener consists of three building blocks:
- The “invite” – this is the initial pop up. Rather than just starting with the form itself, you can give some context as to what it is, and give the user incentive to participate. We typically say something like: “Have a few minutes? Fill out this form and we’ll contact you to set up a 60 minute interview. You will be compensated with a $___ gift card
- The actual form with questions, in “form” or “story” format, as shown above
- A thank you message
Ethnio even allows you to insert logic into your screeners, such as“disqualifier” questions. For example, if the target audience for user testing is “business owners,” you could add a “business owner or consumer?” dropdown to your sceener, and configure the logic to send “business” users to a “success” thank you page, with further instructions and send “consumer” users to a “failure” thank you page with no further follow-up.
Screener Display Options
There are multiple options for publishing a screener. If your client’s site has high traffic, you can restrict the screener to display to any percentage of pageviews from .001% – 100%. You can add a delay to the screener, so it shows up immediately, after 5 seconds, or after 10 seconds. You can target specific users only, by region, device, or browser. You can shut the screener off automatically once a certain threshold is reached. Finally, you can publish the screener on a timer, so that it only displays during certain hours of the day.
Ethnio has a pretty robust scheduling mechanism. Sometimes I like to use it, but sometimes I schedule with my users “offline” – it really just depends on the type of user test I’m running (whether in-person, phone call, web survey, etc) and the number of users I’m trying to get.
Within the Ethnio platform, you can choose to auto-schedule users who fill out the screener, in which case every user who fills out the screener and meets your “success” criteria will automatically receive a follow-up email allowing them to schedule a time to perform the actual test. The available time slots are based on a calendar that you configure Once a user claims a timeslot on the calendar, this timeslot becomes unavailable.
Alternatively, Ethnio offers “picky” scheduling. With picky scheduling, you can look through the dashboard list of all screener respondents, and hand pick the ones with which you would like to schedule follow-up interviews.
For me, the biggest downfall of either scheduling method is that no matter which you choose (picky or automatic), you’re still following up with the user by email and scheduling a test for a “later” time – whether that’s an hour from now, a day from now, or 2 weeks from now.
In my user testing experience, I have not found this scheduling feature to be that useful. Typically, I perform user tests over the phone or via screenshare, so I find that I get much better results if I just call users directly rather than setting up a “future time.” This also gives me better data because the user is typically fresh off the site, and can remember their experience more clearly. To ensure I am only contacting users that are open to a phone call, I simply add a question to the screener itself, along the lines of “Would you be open to a 15 minute phone interview in the next hour?”
I do think the scheduling platform on Ethnio would be useful for projects where you need to test in-person, because obviously the logistics of getting someone “in the door” are more complicated than a phone call or remote screen-sharing session. I also think scheduling is useful and necessary for longer tests (an hour or longer).
Ethnio has a user testing module on the platform as well. You will see a list of all screener respondents on your Ethnio dashboard. Click a phone number to call any participant and automatically record the call. Of course, this feature only works if your device is equipped with a microphone and audio jack. Ethnio integrates directly with join.me to record screen-sharing sessions. Read ethnio’s detailed user testing guide for more information.
For my latest user teting project, I did not have the appropriate audio equipment on my computer, so I used freeconferencecall.com to record the conversations from my land-line desk phone. I will post a future blog with instructions for free land-line recordings soon!
Finally, you can pay your participants an honorarium directly through Ethnio. If you actually went through the steps of “scheduling” a user or performing a test within the Ethnio platform, that user will automatically show up on your “incentives” dashboard. If you scheduled manually or performed the test manually you will have to manually add users to the “incentives” dashboard.
From the incentives dashboard, you have two gift card options:
Send your participants incentives in any denomination. Note that ethnio adds $5 to each gift card purchase to cover their logistics cost, so keep this in mind when budgeting your user testing. To complete this step, simply verify the user’s email address, customize the message, and hit “send” to distribute the incentive. You can of course choose to incentivize users in another way if you do not want to use Ethnio’s incentive platform, but I found this section of Ethnio to be particularly useful and streamlined.
Overall, I’d say that ethnio has some really useful features, and will make sense for user testing on larger projects where in -person testing, long tests, or several testers are involved. For smaller projects that rely primarily on phone interviews, I think you could honestly accomplish the same thing with a Typeform or Surveymonkey survey as a screener, if you don’t mind doing a little extra set-up work. On the up-side, the support from ethnio is unbelievable. I have actually gotten to speak with Nate Bolt, the founder of ethnio on on several occasions. He answers support questions instantly, and obviously knows absolutely everything about the platform.