Lecture 5: Expert Evaluation

목차

 

Evaluation

Evaluation is any type of test/measure to see how well something performs

Most evaluations involve people, but some not (e.g. ML algorithms)

Evaluation is part of the process, not just about the final product → Iterative Development

 

Is it good enough with respect to X?

How good is X?

Is X better in system A or B?

What effect does it have?

What is wrong with respect to X?

 

Evaluation done by expert and user

 

Qualitative vs Quantitative

 

Expert Evaluation

Evaluation with experts 

- Usability experts 

- Experts about your end-users (e.g. Teachers for application for kids)

 

Advantages:

- Relatively cheap

- Relatively quick

- But also when:

→ Minimally 'good' system needed

→ Vulnerable users

→ Confounding factors

 

Disadvantages:

- Simulated, so not real life

- Not actual user group

 

Examples:

1. Questions to help recollect trauma  → therapists

2. Smart watch design → Design expert

 

 

Cognitive Walkthrough (인지적 시찰법)

인지적 시찰법: 제품 또는 서비스(시스템)에 관한 사전 설명, 안내, 훈련 과정이 없어도 사용하는 것이 편리한지의 여부를 평가하는 방법.

Cognitive walkthrough is an evlauation method for manually simulating user's cognitive processes as the user interacts with an user interface in an effort to accomplish a specific task.

- Early in the design cycle

- Evaluate the ease with which users can perform a task with little or no formal instruction of informal coaching

 

Step 1: Define goals and steps & user

Step 2: Ask questions about design (will the user be able to execute the steps on their own)

 

Preparation:

1. Task description

2. Action sequence

3. Anticipated users

4. User's initial goals 

 

Every step: answer the 4 questions

1. Will people try to achieve the right effect? 

2. Will they notice the correct action?

3. Will they associate the action with the effect?

4. Will people see progress towards goal after action?

 

 

 

Cognitive Walkthrough - Log In

 

 

Cognitive Walkthrough - Enter Credentials

 

 

Cognitive Walkthrough - Scan Colors

 

 

Cognitive Walkthrough - Enter Login Code

 

 

Cognitive Walkthrough - Log In

 

 

 

Heuristic Evaluation (휴리스틱 평가)

휴리스틱 평가: 인터페이스 디자인에서 사용성 문제를 찾기 위한 사용성 엔지니어링 방법론.

 

Experts independently evaluate the user interface whether it conforms to established usability principles (heuristics).

Method can be applied on paper prototypes

Output of the evaluation is a list of usability problems in the interface with reference to usability principles that were violated.

 

Design Principles:

1. Visibility of system status (시스템 상태의 시각화)  

2. Match between system and the real world (시스템과 실제 세계의 일치) 

3. User control and freedom (제어의 자유) 

4. Consistency and standards (일치와 표준) 

5. Error prevention (에러 방지) 

6. Recognition rather than recall (기억보다 인식)

7. Flexibility and efficiency of use (유연성과 효율성)

8. Aesthetic and minimalistic design (심플하고 아름다운 디자인)

9. Help users with errors (에러 인식, 진단, 복구)

10. Help and documentation (도움말과 문서화)

 

 

Evaluators: Does not help to raise the number of evaluators (they may miss some problems), won't find new problems 

5 Experts find about 75% of problems that 15 experts find (so 3-5 experts sufficient)

 

Severity matrix helps you to prioritize the usability problems on severity

 

 

 

User Evaluation

Evaluating with the people the system is aimed at

In different stages of the design:

- Design: Based on ideas/sketches

- Prototype: Based on working system

- Use: Based on daily use in real context

 

 

When to Evaluate?

 

 

Cooperative Evaluation 

Include users in early stage

Participants work alone, but in cooperation with evaluator

Reliable but less economical

 

Steps:

1. Using the scenarios, write a draft list of tasks - Tasks must be realistic, doable with the software, and explore the system thoroughly

2. Try out the tasks and estimate how long they will take a participant to complete - Allow 50% longer than the total task time 

3. Prepare a task sheet for the participants - Be specific and explain the tasks it's understandable

4. Get ready for the test session - Have the prototype ready with a list of prompt questions with recording materials, e.g. notebook and pen, audio recorder, etc

5. Introduce participants to tasks - Participants should work individually.

6. Participants do task while giving running commentary - take notes of where participants find problems 

7. Encourage participants to keep talking 

8. When finished: brief interview about the usability of the prototype

9. Wrap up notes and write report

 

 

Participatory Heuristic Evaluation

Users independently evaluate the user interface whether it conforms to established usability principles = Heuristic evaluation for the end users

User should be well briefed about process and heuristics

Advantage: Evaluation by actual end users

Disadvantage: users might not be familiar with heuristics

 

 

Co-Discovery (공동 발견)

Naturalistic & informal

Capture first impression

Later stage of design

Thinking aloud

Pairs / groups

Interaction between users

 

 

Thinking Aloud

Say whatever comes to mind while doing the task

Record & transcribe to analyze

Also record what they were doing/where they were when saying something

 

 

Interviews

Best for open and complicated questions 

 

 

Overview

 

 

Analyzing Qualititative Studies

Focus on experience: why and how

Open questions and smaller sample size

More formative & Mid-process

 

 

Reporting

Methods:

1. Participants

2. Procedure

3. Materials

4. Measures

5. Data preparation & analysis

 

 

Methods - Participants

Type of user group

Recruited where/how

Number of participants

Mean age + SD

Gender balance

Other relevant user data - How user data was measured

 

 

Methods - Procedure

What will participants do, step by step:

- What is the physical setup

- What is the content of tasks

- How are tasks ordered/repeated

- When are things measured

 

Based on this, someone else should be replicate all the steps

 

 

Methods - Materials

What system did people use?

What did the prototype look like?

What hardware was used? (e.g. robot, VR)

 

 

Methods - Measures

What do you measure and how do you measure?

e.g.

- What questionnaire do you use?

- What are the questions?

- How do you collect data?

 

 

Methods - Data Preparation & Analysis