Research Project

Extending the NASA Task Load Index (NASA-TLX) for Human-Autonomy Team Decision Making

Motivation/Research Problem

The NASA Task Load Index (TLX; Hart & Staveland, 1988) measures workload using a self-report assessment. The six subscales participants are asked to rate include: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration. These subscales are then compared with one another to associate a weight with each score that is used to generate a weighted workload score. We know from previous research that there is a significant negative correlation between workload measured by the TLX and objective measures of team performance such as detection of target events and percentage of correct operator actions (Lang et al, 2002). Researchers have found that there is a significant positive correlation between workload measured by the TLX and objective measures of poor performance such as operationally important action deviations (Lang et al, 2002). These correlations indicate that as workload scores increase, performance tends to decrease, and that it is important to recognize when workload is too high. The NASA-TLX serves as an excellent measure for workload due to its exceptional level of reliability and validity (Rubio, Diaz, Martin, & Puente, 2004). However, the tool has a few limitations, one of which is the lack of a red line to indicate that workload is too high (Hart, 2006). Anecdotally speaking, users of the NASA-TLX identified that there are practical issues with the tool and that it could be missing some necessary components. Harris, Wilson, & Vine, (2019) explain that the NASA-TLX was designed for pilots and may not reflect the unique demands posed by some tasks. Researchers have attempted to address this limitation by developing variants of the NASA-TLX for specific occupations, activities, user communities and locales. A variant of the NASA-TLX specific for robot operation is not yet available.

Research Team

Lead Researchers:

Ellie Kazemi: ARCS Co-PI, CSUN professor, directly supervise ARCS Fellows
Nhut Ho: ARCS Director, CSUN Professor, advise team
Li Liu: ARCS Associate Director, CSUN Assistant Professor, advise team
Amiel Harman: ARCS/CSUN faculty

Collaborators:

Amir Rahmani: primary JPL contact, directly involved in the development of work
Suzanne Sheld: ARCS Co-PI, collaborator
Kevin Zemlicka: ARCS Co-PI, collaborator
Gina Masequesmay: ARCS Co-PI, collaborator

Student Team:

Eric McCoy
Helina Mekonnen
Adisa Ptah

Funding

Funding Organization: NASA
Funding Program: MIRO

Alignment, Engagement and Contributions to the priorities of NASA’s Mission Directorates

As NASA continues to develop new autonomous systems to complete their current mission objectives, they will need to evaluate the workload placed on the robot operators in order to determine an ideal level of workload that promotes higher performance. The development of a domain specific variant of the NASA-TLX will enable NASA to identify and manipulate specific sources of workload in robot operation tasks.

Research Questions and Research Objectives

We ask the following research question in project 1: “How can we extend the NASA Task Load Index to be a more effective tool in making decisions regarding the development of autonomous systems and the structure of human-robot teams?” The primary objective of this study is to gather data from subject matter experts (i.e. NASA scientists) to answer this question and develop a domain specific variant of the NASA-TLX.

Research Methods

After a thorough literature review, we will conduct a large-scale survey of NASA/JPL scientists who use the NASA Task Load Index. We will then follow up with a small sample of the individuals we surveyed using semi-structured phone interviews. The final phase will be a structured focus group of subject matter experts that have agreed to help extend the NASA Task Load.

Research Deliverables and Products

Anticipated deliverables

A list of features of interest identified by key stakeholders
A focused list, generated by subject matter experts at NASA/JPL, for extending the NASA Task Load
Research results of survey, interviews, and focus group
Research results of the experimental study
Extension of NASA Task Load
Conference presentations
Publication

Anticipated Publications

We are still considering appropriate outlets but will likely submit to the journals that we most reference. We are also looking into appropriate outlets for convergence of psychology and autonomy.

Conference Presentations

Same as above. Additionally, we will likely present at the California Association for Behavior Analysis and the Association for Behavior Analysis International.

Research Timeline

Start Date: January 2020
End Date: May 2021