Research Project
Multimodal AI for Digital Twins of Collaborative Lunar Exploration
Research Team
Lead Researchers:
-
Dr. Bingbing Li, Manufacturing Systems Engineering
Student Team:
- Roger Xu
Funding
- Funding Organization:
- Funding Program:
SYNOPSIS
- Using multimodal AI tools such as Meta’s Video Joint Embedding Predictive Architecture (V-JEPA) and other vision language models/large language models(VLMs/LLMs) to support Digital Twin systems for space exploration.
- Explore long-term goal of supporting safe and effective human-robot collaboration during future missions to the Moon.
Abstract
This project investigates how multimodal AI can understand and predict interactions within a 3D simulated lunar environment such as the Digital Twins-a virtual representation of physical world, as a tool for simulation, planning, and real-time mission support. This would allow autonomous robots to collaborate with humans more safely and effectively in environments where communication and sensory data may be limited.
Research Objectives
-
Assessing potential applications of multimodal AI in supporting autonomous robotic agents for lunar exploration missions.
-
Evaluate the ability of AI systems to predict human and robot actions in different scenarios, improving safety and efficiency.
Research Methods
-
Utilize multimodal architectures such as V-JEPA and fine-tuned LLMs to process and interpret visual inputs.
-
Train models to recognize key actions and environmental features; evaluate predictive accuracy, decision-making performance, and response time.
Research Results and Deliverables
-
Demonstrated ability to predict and interpret human and robotic actions based on visual and contextual data.
-
Technical report documenting methodology, model performance, and use-case analysis of LLMs working with a simulated lunar environment via Digital Twin systems.
Lead Researchers:
-
Dr. Bingbing Li, Manufacturing Systems Engineering
Student Team:
- Roger Xu
Funding
- Funding Organization:
- Funding Program: