In planning the initial stages of a user-centered design project, I was recently asked to provide a summary of the relative advantages and disadvantages of various data capture methods. As others might find this list useful, I thought I'd post it here.

SURVEYS

Advantages:

  • Fast and cheap, cast wide net quickly
  • Easy analysis (except for open-ended questions)

Disadvantages:

  • Focus on preference rather than what works best
  • Don't reveal performance
  • Rely on memory
  • Usually no opportunity to probe or follow up
  • Motivation and attention span of participants is critical
  • Difficult to achieve representative sample
  • Reliability may be questionable

INTERVIEWS

Advantage: Provide in-depth data. Help with issues like:

  • What type of information are you looking for?
  • Are you able to find it? How?
  • What is missing?
  • How do you use this information?
  • How critical is this information?
  • What format do you want it in?
  • What works well on the existing site?
  • What would you change?

Pitfalls of interviews:

  • Subjects may be intimidated by or try to impress interviewer
  • Subjects may assume interviewer knows things he doesn't
  • Subjects may have incomplete knowledge or faulty memories
  • Subjects may not be articulate
  • Time-consuming and expensive (especially when interviewees must be compensated)

FOCUS GROUPS

Advantages include:

  • Rapid feedback from users
  • Provide window into users' stated priorities
  • Generates product possibilities that stakeholders may have missed

Disadvantages include:

  • Participants may influence each other
  • Tend to be dominated by one or two especially vocal participants
  • Talking about something is not the same as doing it (Famously, the Edsel was the most heavily focus grouped car in history)

USER TESTS

Advantages include:

  • Reveals how users interact with site, where problems lie
  • Great for evaluating existing websites
  • Can be done quickly and cheaply if you're willing to sacrifice careful selection and screening of participants, and eliminate “frills” like video editing and detailed reports.

Disadvantages:

  • Does not always reveal root causes of problems, or how to fix them
  • You can observe what users do, but may not always get the reasons behind their actions
  • Results are limited to the assigned tasks. Can't be generalized to entire site.
  • Very dependent upon skill and experience of moderator
  • Can get expensive, especially where participants are highly paid professionals and where video editing and detailed reporting are required.

CARD SORTING EXERCISES

Best conducted in-person, as you can ask participants to think out loud and learn not only how they organize content, but why. (i.e. deeper insight into mental models and priorities.)

Can be conducted remotely, online. (But then all you get is the “what”, not the “why”.)

Advantages:

  • Reveals logical groupings and labels, from users' point of view
  • Easy to implement
  • If done remotely, can involve large number of users worldwide

Limitations:

  • Sometimes, results are not logical. Need to ask questions.
  • Users might misinterpret card labels, contaminating results
  • Does not reveal how users would complete a task
  • Does not rank items by importance or frequency
  • Does not reveal page/task flows
  • If performed on existing users, they may simply replicate existing system

Message Sent

Thank you for registering.

Cardinal Path hosted a live session to connect with you and answer all your questions on Google Analytics.
Get all the expertise and none of the consultancy fees in this not-to-be-missed, rapid-fire virtual event.

Thank you for submitting the form.

Thank you for submitting the form.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you.

Click here to download access the tool.

Message Sent

Thank you for registering.

Message Sent

Thank you.

Message Sent

Thank you.

Message Sent

Thank you

Message Sent

Thank you

Message Sent

Thank you.

Message Sent

Thank you

Message Sent

Thank you.

Message Sent

Success!
Your message was received.

Thank you.

Thank you for registering.

Cardinal Path is continuing with its series of free training. Next we are conducting training on Google Data Studio. Check it out here.

Message Sent

Thank you for registering.

Thank you for your submission.

Your request has been submitted and a rep will reach out to you shortly.

Message Sent

Thank you for your interest.

Thank you for registering.

You should receive a confirmation email from GoToWebinar with your unique webinar login information. If you do not receive this email or have trouble logging in to the event, please email asmaa.mourad@cardinalpath.com.

Thank you for subscribing!

You're now looped into the world's largest GMP resource hub!

Thank you for your submission.

Thank you for your submission.

Thank you for your submission.

Thank you for your submission.

Message Sent

Thank you for registering.

Message Sent

Thank you for your submission.

Thank you for your submission.

Message Sent

Thank you for registering.

Thank you for registering.​

Paid media spend by Government websites increased a whopping 139% YoY in 2020.

2020 Online Behavior Live Dashboard

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

2020 Online Behavior Live Dashboard

Thank you for your submission.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Thank you for registering.

Message Sent

Success! Thank you
for reaching out.