Publications

2024

1. FoundLoc: Vision-based Onboard Aerial Localization in the Wild

   He, Yao, Cisneros, Ivan, Keetha, Nikhil, Patrikar, Jay, Ye, Zelin, Higgins, Ian, Hu, Yaoyu,  Kapoor, Parv, and Scherer, Sebastian

   Robotics: Science and Systems (Rebuttal) 2024

   Abs HTML

   Robust and accurate localization for Unmanned Aerial Vehicles (UAVs) is an essential capability to achieve autonomous, long-range flights. Current methods either rely heavily on GNSS, face limitations in visual-based localization due to appearance variances and stylistic dissimilarities between camera and reference imagery, or operate under the assumption of a known initial pose. In this paper, we developed a GNSS-denied localization approach for UAVs that harnesses both Visual-Inertial Odometry (VIO) and Visual Place Recognition (VPR) using a foundation model. This paper presents a novel vision-based pipeline that works exclusively with a nadir-facing camera, an Inertial Measurement Unit (IMU), and pre-existing satellite imagery for robust, accurate localization in varied environments and conditions. Our system demonstrated average localization accuracy within a 20-meter range, with a minimum error below 1 meter, under real-world conditions marked by drastic changes in environmental appearance and with no assumption of the vehicle’s initial pose. The method is proven to be effective and robust, addressing the crucial need for reliable UAV localization in GNSS-denied environments, while also being computationally efficient enough to be deployed on resource-constrained platforms.

2. Safe Planning through Incremental Decomposition of Signal Temporal Logic

   Kapoor, Parv, Meira-Goes, Romulo, and Kang, Eunsuk

   Nasa Formal Methods 2024

   Abs HTML PDF

   Trajectory planning is a critical process that enables au- tonomous systems to safely navigate complex environments. Signal tem- poral logic (STL) specifications are an effective way to encode complex, temporally extended objectives for trajectory planning in cyber-physical systems (CPS). However, the complexity of planning with STL using existing techniques scales exponentially with the number of nested op- erators and the time horizon of a given specification. Additionally, poor performance is exacerbated at runtime due to limited computational bud- gets and compounding modeling errors. Decomposing a complex speci- fication into smaller subtasks and incrementally planning for them can remedy these issues. In this work, we present a method for decomposing STL specifications to improve planning efficiency and performance. The key insight in our work is to encode all specifications as a set of basic constraints called reachability and invariance constraints, and schedule these constraints sequentially at runtime. Our experiment shows that the proposed technique outperforms the state-of-the-art trajectory planning techniques for both linear and non-linear dynamical systems.

3. Tolerance of Reinforcement Learning Controllers against Deviations in Cyber Physical Systems

   *Zhang, Changjian, * Kapoor, Parv, Meira Goes, Romulo, Garlan, David, Kang, Eunsuk, Ganlath, Akila, Mishra, Shatadal, and Ammar, Nejib

   In 26th International Symposium on Formal Methods (FM24) 2024
4. Logically Constrained Robotics Transformers for Enhanced Perception-Action Planning

   Kapoor, Parv, Vemprala, Sai, and Kapoor, Ashish

   Robotics Science and Systems Towards Safe Autonomy 2024

   Abs

   With the advent of large foundation model based planning, there is a dire need to ensure their output aligns with the stakeholder’s intent. When these models are deployed in the real world, the need for alignment is magnified due to the potential cost to life and infrastructure due to unexpected faliures. Temporal Logic specifications have long provided a way to constrain system behaviors and are a natural fit for these use cases. In this work, we propose a novel approach to factor in signal temporal logic specifications while using autoregressive transformer models for trajectory planning. We also provide a trajectory dataset for pretraining and evaluating foundation models. Our proposed technique acheives 74.3 % higher specification satisfaction over the baselines.

2023

1. Follow The Rules: Online Signal Temporal Logic Tree Search for Guided Imitation Learning in Stochastic Domains

   Aloor, Jasmine Jerry, Patrikar, Jay,  Kapoor, Parv, Oh, Jean, and Scherer, Sebastian

   In 2023 IEEE International Conference on Robotics and Automation (ICRA) 2023

   Abs HTML

   Seamlessly integrating rules in Learning-from-Demonstrations (LfD) policies is a critical requirement to enable the real-world deployment of AI agents. Recently, Signal Temporal Logic (STL) has been shown to be an effective language for encoding rules as spatio-temporal constraints. This work uses Monte Carlo Tree Search (MCTS) as a means of integrating STL specification into a vanilla LfD policy to improve constraint satisfaction. We propose augmenting the MCTS heuristic with STL robustness values to bias the tree search towards branches with higher constraint satisfaction. While the domain-independent method can be applied to integrate STL rules online into any pre-trained LfD algorithm, we choose goal-conditioned Generative Adversarial Imitation Learning as the offline LfD policy. We apply the proposed method to the domain of planning trajectories for General Aviation aircraft around a non-towered airfield. Results using the simulator trained on real-world data showcase 60 percent improved performance over baseline LfD methods that do not use STL heuristics.

2. Investigating Robustness in Cyber-Physical Systems: Specification-Centric Analysis in the face of System Deviations

   *Changjian, Zhang, *Kapoor, Parv, Meira-Goes, Romulo, Kang, Eunsuk, Garlan, David, Ganlath, Akila, Mishra, Shatadal, and Ammar, Nejib

   Under Submission 2023

   Abs HTML

   The adoption of cyber-physical systems (CPS) is on the rise in complex physical environments, encompassing domains such as autonomous vehicles, the Internet of Things (IoT), and smart cities. A critical attribute of CPS is robustness, denoting its capacity to operate safely despite potential disruptions and uncertainties in the operating environment. This paper proposes a novel specification-based robustness, which characterizes the effectiveness of a controller in meeting a specified system requirement, articulated through Signal Temporal Logic (STL) while accounting for possible deviations in the system. This paper also proposes the robustness falsification problem based on the definition, which involves identifying minor deviations capable of violating the specified requirement. We present an innovative two-layer simulation-based analysis framework designed to identify subtle robustness violations. To assess our methodology, we devise a series of benchmark problems wherein system parameters can be adjusted to emulate various forms of uncertainties and disturbances. Initial evaluations indicate that our falsification approach proficiently identifies robustness violations, providing valuable insights for comparing robustness between conventional and reinforcement learning (RL)-based controllers

2022

1. Challenges in Close-Proximity Safe and Seamless Operation of Manned and Unmanned Aircraft in Shared Airspace

   Patrikar, Jay, Dantas, Joao, Ghosh, Sourish,  Kapoor, Parv, Higgins, Ian, Aloor, Jasmine J, Navarro, Ingrid, Sun, Jimin, Stoler, Ben, Hamidi, Milad, and others,

   IEEE International Conference on Robotics and Automation (ICRA) 2022

   Abs HTML

   We propose developing an integrated system to keep autonomous unmanned aircraft safely separated and behave as expected in conjunction with manned traffic. The main goal is to achieve safe manned-unmanned vehicle teaming to improve system performance, have each (robot/human) teammate learn from each other in various aircraft operations, and reduce the manning needs of manned aircraft. The proposed system anticipates and reacts to other aircraft using natural language instructions and can serve as a co-pilot or operate entirely autonomously. We point out the main technical challenges where improvements on current state-of-the-art are needed to enable Visual Flight Rules to fully autonomous aerial operations, bringing insights to these critical areas. Furthermore, we present an interactive demonstration in a prototypical scenario with one AI pilot and one human pilot sharing the same terminal airspace, interacting with each other using language, and landing safely on the same runway. We also show a demonstration of a vision-only aircraft detection system.

2021

1. Predicting food insecurity

   Feffer, Michael,  Kapoor, Parv, and Dodt, Sebastian

   2021

   Abs PDF

   In its longest drought in four decades, Somalia and other countries in the Horn of Africa are currently facing an un- precedented humanitarian crisis. 40 percent of the Somali popu- lation (six million people) are estimated to be impacted by severe acute food insecurity (WFP 2022c). The United Na- tions Office for the Coordination of Humanitarian Affairs (UN OCHA) has released additional funds for humanitarian aid in the region based on predictive analytics of its novel An- ticipatory Action (AA) Team. Releasing “trigger warnings” when food insecurity levels are predicted to reach critical lev- els, AA employs data from partners that leverages manual and qualitative methods for food insecurity classification and prediction. The frequency of food insecurity prediction is 2-4 times per year making any subsequent modelling ill-equipped to account for the dynamic nature of natural catastrophes and conflicts. Additionally, the prediction accuracy has been sub- ject to scrutiny. In this paper, we describe an AI-based ap- proach that can automate this system and more accurately forecast levels of hunger in real-time. We hope that our results serve as a foundation for the use of AI in humanitarian relief and lead to more targeted aid for vulnerable populations.

2020

1. Model-based Reinforcement Learning from Signal Temporal Logic Specifications

   Kapoor, Parv, Balakrishnan, Anand, and Deshmukh, Jyotirmoy V.

   CoRR 2020

   Abs HTML

   Techniques based on Reinforcement Learning (RL) are increasingly being used to design control policies for robotic systems. RL fundamentally relies on state-based reward functions to encode desired behavior of the robot and bad reward functions are prone to exploitation by the learning agent, leading to behav- ior that is undesirable in the best case and critically dangerous in the worst. On the other hand, designing good reward functions for complex tasks is a challenging problem. In this paper, we propose expressing desired high-level robot behavior using a formal specification language known as Signal Temporal Logic (STL) as an alternative to reward/cost functions. We use STL specifications in conjunction with model-based learning to design model predictive controllers that try to optimize the satisfaction of the STL specification over a finite time horizon. The proposed algorithm is empirically evaluated on simulations of robotic system such as a pick-and-place robotic arm, and adaptive cruise control for autonomous vehicles

2. Predicting Time to Contact Across the Visual Image

   Marshall, David, Rushton, Simon K, Redfern, Joseph,  Kapoor, Parv, and Moran, Rosalyn J

   In PERCEPTION 2020