Research Project

Extending the NASA Task Load Index (NASA-TLX)

Research Team

Lead Researchers

Ellie Kazemi: ARCS Co-PI, CSUN professor, directly supervise ARCS Fellows
Nhut Ho: ARCS Director, CSUN Professor, advise team
Li Liu: ARCS Associate Director, CSUN Assistant Professor, advise team
Amiel Harman: ARCS/CSUN faculty

Collaborators:

Amir Rahmani: primary JPL contact, directly involved in the development of work
Suzanne Sheld: ARCS Co-PI, collaborator
Kevin Zemlicka: ARCS Co-PI, collaborator
Gina Masequesmay: ARCS Co-PI, collaborator

Student Team:

An An Chang
Mackenzie McSherry
Eric McCoy
Helina Mekonnen
Adisa Ptah

Funding

Funding Organization: NASA
Funding Program: MUREP-MIRO

Abstract

Abstract: (approximately 250 words): The current research identified limitations of the NASA Task Load Index (TLX) by evaluating literature surrounding the workload measurement tool. We then gathered NASA scientists to participate in an online survey and semi-structured interviews to identify what limitations of the NASA TLX were the most important to them. We found that the practicality limitations of the NASA-TLX were most inhibiting the use of the NASA-TLX and developed a potential extension of the NASA TLX that would be able to mitigate the scientists’ barriers.

Motivation/Research Problem

The NASA Task Load Index (TLX; Hart & Staveland, 1988) measures workload using a self-report assessment. The six subscales participants are asked to rate include: Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration. These subscales are then compared with one another to associate a weight with each score that is used to generate a weighted workload score. We know from previous research that there is a significant negative correlation between workload measured by the TLX and objective measures of team performance, such as detection of target events and percentage of correct operator actions (Lang et al, 2002). Researchers have found that there is a significant positive correlation between workload measured by the TLX and objective measures of poor performance such as operationally important action deviations (Lang et al, 2002). These correlations indicate that as workload scores increase, performance tends to decrease, and that it is important to recognize when workload is too high. The NASA-TLX serves as an excellent measure for workload due to its exceptional level of reliability and validity (Rubio, Diaz, Martin, & Puente, 2004). However, the tool has a few limitations, one of which is the lack of a red line to indicate that the workload is too high (Hart, 2006). Anecdotally speaking, users of the NASA-TLX identified that there are practical issues with the tool and that it could be missing some necessary components. Harris, Wilson, & Vine, (2019) explain that the NASA-TLX was designed for pilots and may not reflect the unique demands posed by some tasks. Researchers have attempted to address this limitation by developing variants of the NASA-TLX for specific occupations, activities, user communities and locales. There is currently no research on how these limitations affect the work being done at NASA.

Alignment, Engagement and Contributions to the priorities of NASA’s Mission Directorates

As NASA continues to develop new autonomous systems to complete their current mission objectives, they will need to evaluate the workload placed on the robot operators in order to determine an ideal level of workload that promotes higher performance. Evaluating the perspectives of NASA scientists’ will allow us to better propose what changes to the tool will best fit their needs. We will then develop a tool based on their needs that should be able to better identify sources of workload in their task environments.

Research Questions and Research Objectives

We ask the following research question in Project 1: “What are the current perspectives of the NASA scientists on NASA-TLX limitations?” The primary objective of this study is to gather data from subject matter experts (i.e. NASA scientists) to answer this question and develop a solution for the proposed limitations.

Research Methods

After a thorough literature review, we will conduct a survey of NASA/JPL scientists who use the NASA Task Load Index. We will then follow up with a small sample of the individuals we surveyed using semi-structured phone interviews. The final phase will be a proposal solution that may be evaluated by scientists who find it beneficial.

Research Deliverables and Products

Internal to the project, we have already produced:

A list of features of interest identified by key stakeholders
The Proteus compiler, including example usage and documentation
Some existing HSM-based software ported to Proteus
Next steps for extending Proteus

From the last reflective journal, the following internal deliverables were planned:

Use cases derived from ScalaHSM and TextHSM implemented in Proteus, along with the results of their application on Proteus relative to ScalaHSM and TextHSM (carry-over of 3 from before)
A built-in system for runtime fault monitoring and response, which is to handle faults that either cannot be, or cannot easily be, statically prevented
Support for typeclasses
Support for basic data structures
Next steps for extending Proteus (a perpetual process, carried over from the last time)

With #1, we changed the deliverable a bit to include any HSM-based software (instead of just ScalaHSM and TextHSM), which widens scope of what is relevant. We are actively working on this now, and have already ported some software from JPL. With #2, a runtime fault monitoring system has been completed and is part of Proteus. With #3 and #4, these portions are partially implemented, but incomplete. #3 and #4 need project leads, and #1 currently has higher priority. #5 is ongoing.

Publications

Since the last reflective journal, SMC-IT has been identified as an appropriate peer-reviewed venue. Our prior AIAA ASCEND conference paper was reworked for SMC-IT, and was ultimately accepted. We are working on the camera-ready version of the SMC-IT paper now.

Presentations

Since the last reflective journal, there have been no conference presentations. A conference presentation is planned for SMC-IT, possibly by Brian McClelland. Some students who were previously on the project (Simran Gill, Eileen Quiroz, and Frank Serdenia) gave a presentation at the AIMS^2 Student Research Symposium regarding work they performed on Proteus during Summer 2020 (internal to CSUN). Additionally, Rebecca Carbone and Kennedy Johnson have presented on their Proteus work twice internal to ARCS, once during the ARCS seminar series and once with JPL stakeholders.

Other

Brian McClelland completed his MS thesis, which specifically concerned Proteus.

Research Timeline

Start Date: January 2020
End Date: July 2021