Research Project

Extending the NASA Task Load Index (NASA-TLX) for Human-Autonomy Team Decision Making

Motivation/Research Problem

The NASA Task Load Index (TLX; Hart & Staveland, 1988) measures workload using a self-report assessment. The six subscales participants are asked to rate include: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration. These subscales are then compared with one another to associate a weight with each score that is used to generate a weighted workload score. We know from previous research that there is a significant negative correlation between workload measured by the TLX and objective measures of team performance such as detection of target events and percentage of correct operator actions (Lang et al, 2002). Researchers have found that there is a significant positive correlation between workload measured by the TLX and objective measures of poor performance such as operationally important action deviations (Lang et al, 2002). These correlations indicate that as workload scores increase, performance tends to decrease, and that it is important to recognize when workload is too high. The NASA-TLX serves as an excellent measure for workload due to its exceptional level of reliability and validity (Rubio, Diaz, Martin, & Puente, 2004). However, the tool has a few limitations, one of which is the lack of a red line to indicate that workload is too high (Hart, 2006). Anecdotally speaking, users of the NASA-TLX identified that there are practical issues with the tool and that it could be missing some necessary components. Harris, Wilson, & Vine, (2019) explain that the NASA-TLX was designed for pilots and may not reflect the unique demands posed by some tasks. Researchers have attempted to address this limitation by developing variants of the NASA-TLX for specific occupations, activities, user communities and locales. A variant of the NASA-TLX specific for robot operation is not yet available. 

Research Team
Collaborators:

  • Amir Rahmani: primary JPL contact, directly involved in the development of work
  • Suzanne Sheld: ARCS Co-PI, collaborator
  • Kevin Zemlicka: ARCS Co-PI, collaborator
  • Gina Masequesmay: ARCS Co-PI, collaborator

Faculty: 

  • Nhut Ho: ARCS Director, CSUN Professor, advise team
  • Li Liu: ARCS Associate Director, CSUN Assistant Professor, advise team
  • Amiel Harman: ARCS/CSUN faculty
  • Ellie Kazemi: ARCS Co-PI, CSUN professor, directly supervise ARCS Fellows

Student Team:

  • Adisa Ptah
  • Eric McCoy
  • Helina Mekonnen
Alignment, Engagement and Contributions to the priorities of NASA’s Mission Directorates

As NASA continues to develop new autonomous systems to complete their current mission objectives, they will need to evaluate the workload placed on the robot operators in order to determine an ideal level of workload that promotes higher performance. The development of a domain specific variant of the NASA-TLX will enable NASA to identify and manipulate specific sources of workload in robot operation tasks.

Research Questions and Research Objectives

We ask the following research question in project 1: “How can we extend the NASA Task Load Index to be a more effective tool in making decisions regarding the development of autonomous systems and the structure of human-robot teams?” The primary objective of this study is to gather data from subject matter experts (i.e. NASA scientists) to answer this question and develop a domain specific variant of the NASA-TLX.

Research Methods

After a thorough literature review, we will conduct a large-scale survey of NASA/JPL scientists who use the NASA Task Load Index. We will then follow up with a small sample of the individuals we surveyed using semi-structured phone interviews. The final phase will be a structured focus group of subject matter experts that have agreed to help extend the NASA Task Load.

Research Deliverables (Publications, Presentations and other Products)
Anticipated deliverables

  • A list of features of interest identified by key stakeholders
  • A focused list, generated by subject matter experts at NASA/JPL, for extending the NASA Task Load
  • Research results of survey, interviews, and focus group
  • Research results of the experimental study
  • Extension of NASA Task Load
  • Conference presentations
  • Publication

Anticipated Publications

  • We are still considering appropriate outlets but will likely submit to the journals that we most reference. We are also looking into appropriate outlets for convergence of psychology and autonomy.

Conference Presentations

  • Same as above. Additionally, we will likely present at the California Association for Behavior Analysis and the Association for Behavior Analysis International.
Research Timeline

Project 1. NASA Task Load Tx

January 2020  —

  • Research plan, literature review

February 2020  —

  • Literature review, poster session (ARCS Lift Off)

April 2020  —

  • Colloquia #1, Submit for IRB approval

May 2020  —

  • Implement IRB Feedback, prepare online survey and phone interview data collection materials

June 2020 —

  • Implement IRB Feedback, continue revising online survey and phone interview data collection materials

July 2020   —

  • Implement IRB Feedback, continue revising online survey and phone interview data collection materials

August 2020  —

  • Receive IRB Approval, Begin Sending Project Invitations Seminar #2

September 2020  —

  • Sending Project Invitations. Follow up regarding project invitations.

October 2020  —

  • Submit for NSTGRO21 Grant

November 2020  —

  • Complete online survey. Compile results of online survey.

December 2020  —

  • Revise phone interview questions and focus group guide

January 2021  —

  • Complete phone interviews

February 2021  —

  • Complete focus groups

March 2021  —

  • Analyze data and develop Extension of the NASA-TLX

April 2021  —

  • Begin writing manuscript and submit for publication

May 2021  —

  • Conference presentation
  •  

Are there other activities (e.g., proposals or additional projects) that you have developed or anticipate based on your NASA ARCS project?

Because of COVID-19, we have been very busy revising the projects we have. Both fellows applied for NSTGRO21 grant. Our collaborator, Dr. Rahmani, is invested in our collaborations.

Impact of Project Partnership with NASA:

The name recognition and prestige associated with NASA and JPL is attractive for recruiting strong students that would have historically not considered pursuing research careers or growth in STEM.

I continue to be invited by journals, conferences, and podcasters in the field of behavior analysis to discuss my efforts on this project because the convergence of behavior analysis and autonomy is new and exciting for expanding the reach of behavioral science.