Yunuo Zhang is a graduate student in the Department of Electrical Engineering and Computer Science at Vanderbilt University and works as a research assistant at the Institute for Software Integrated Systems. He received his Bachelor’s degree in Computer Science and Applied Mathematics from Vanderbilt University in December 2021.

Yunuo Zhang Publications

1. N. S. Keplinger, B. Luo, I. Bektas, Y. Zhang, K. H. Wray, A. Laszka, A. Dubey, and A. Mukhopadhyay, NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes. 2025.
   Bibtex Abstract

   @misc{keplinger2025nsgymopensourcesimulationenvironments,
     author = {Keplinger, Nathaniel S. and Luo, Baiting and Bektas, Iliyas and Zhang, Yunuo and Wray, Kyle Hollins and Laszka, Aron and Dubey, Abhishek and Mukhopadhyay, Ayan},
     title = {NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes},
     year = {2025},
     archiveprefix = {arXiv},
     contribution = {colab},
     eprint = {2501.09646},
     primaryclass = {cs.AI},
     url = {https://arxiv.org/abs/2501.09646}
   }
   

   In many real-world applications, agents must make sequential decisions in environments where conditions are subject to change due to various exogenous factors. These non-stationary environments pose significant challenges to traditional decision-making models, which typically assume stationary dynamics. Non-stationary Markov decision processes (NS-MDPs) offer a framework to model and solve decision problems under such changing conditions. However, the lack of standardized benchmarks and simulation tools has hindered systematic evaluation and advance in this field. We present NS-Gym, the first simulation toolkit designed explicitly for NS-MDPs, integrated within the popular Gymnasium framework. In NS-Gym, we segregate the evolution of the environmental parameters that characterize non-stationarity from the agent’s decision-making module, allowing for modular and flexible adaptations to dynamic environments. We review prior work in this domain and present a toolkit encapsulating key problem characteristics and types in NS-MDPs. This toolkit is the first effort to develop a set of standardized interfaces and benchmark problems to enable consistent and reproducible evaluation of algorithms under non-stationary conditions. We also benchmark six algorithmic approaches from prior work on NS-MDPs using NS-Gym. Our vision is that NS-Gym will enable researchers to assess the adaptability and robustness of their decision-making algorithms to non-stationary conditions.

2. R. Sen, Y. Zhang, F. Liu, J. P. Talusan, A. Pettet, Y. Suzue, A. Mukhopadhyay, and A. Dubey, Online Decision-Making Under Uncertainty for Vehicle-to-Building Systems, in Proceedings of the ACM/IEEE 16th International Conference on Cyber-Physical Systems (ICCPS), New York, NY, USA, 2025.
   Bibtex Abstract PDF

   @inproceedings{sen2025iccps,
     author = {Sen, Rishav and Zhang, Yunuo and Liu, Fangqi and Talusan, Jose Paolo and Pettet, Ava and Suzue, Yoshinori and Mukhopadhyay, Ayan and Dubey, Abhishek},
     booktitle = {Proceedings of the ACM/IEEE 16th International Conference on Cyber-Physical Systems (ICCPS)},
     title = {Online Decision-Making Under Uncertainty for Vehicle-to-Building Systems},
     year = {2025},
     address = {New York, NY, USA},
     publisher = {Association for Computing Machinery},
     series = {ICCPS '25},
     acceptance = {28.4},
     category = {selective},
     contribution = {lead},
     location = {California, USA},
     numpages = {10},
     ranking = {rank1}
   }
   

   Vehicle-to-building (V2B) systems combine physical infrastructure such as smart buildings and electric vehicles (EVs) connected to chargers at the building, with digital control mechanisms to manage energy use. By utilizing EVs as flexible energy reservoirs, buildings can dynamically charge and discharge EVs to effectively manage energy usage, and reduce costs under time-variable pricing and demand charge policies. This setup leads to the V2B optimization problem, where buildings coordinate EV charging and discharging to minimize total electricity costs while meeting users’ charging requirements. However, the V2B optimization problem is difficult due to: 1) fluctuating electricity pricing, which includes both energy charges (/kWh) and demand charges (/kW); 2) long planning horizons (usually over 30 days); 3) heterogeneous chargers with differing charging rates, controllability, and directionality (unidirectional or bidirectional); and 4) user-specific battery levels at departure to ensure user requirements are met. While existing approaches often model this setting as a single-shot combinatorial optimization problem, we highlight critical limitations in prior work and instead model the V2B optimization problem as a Markov decision process, i.e., a stochastic control process. Solving the resulting MDP is challenging due to the large state and action spaces. To address the challenges of the large state space, we leverage online search, and we counter the action space by using domain-specific heuristics to prune unpromising actions. We validate our approach in collaboration with an EV manufacturer and a smart building operator in California, United States, showing that the proposed framework significantly outperforms state-of-the-art methods.

3. Y. Zhang, B. Luo, A. Mukhopadhyay, and A. Dubey, Observation Adaptation via Annealed Importance Resampling for POMDPs, in Proceedings of the 35th International Conference on Automated Planning and Scheduling (ICAPS), 2025.
   Bibtex Abstract

   @inproceedings{zhang2025observation,
     author = {Zhang, Yunuo and Luo, Baiting and Mukhopadhyay, Ayan and Dubey, Abhishek},
     booktitle = {Proceedings of the 35th International Conference on Automated Planning and Scheduling ({ICAPS})},
     title = {Observation Adaptation via Annealed Importance Resampling for {POMDPs}},
     year = {2025},
     note = {accepted as oral presentation},
     publisher = {{AAAI} Press},
     acceptance = {22.8},
     category = {selective},
     contribution = {lead},
     url = {https://openreview.net/forum?id=KxxxyuOFg2&referrer=%5Bthe%20profile%20of%20Baiting%20Luo%5D(%2Fprofile%3Fid%3D~Baiting_Luo1)}
   }
   

   Partially observable Markov decision processes (POMDPs) are a general mathematical model for sequential decision-making in stochastic environments under state uncertainty. POMDPs are often solved online, which enables the algorithm to adapt to new information in real time. Online solvers typically use bootstrap particle filters based on importance resampling for updating the belief distribution. Since directly sampling from the ideal state distribution given the latest observation and previous state is infeasible, particle filters approximate the posterior belief distribution by propagating states and adjusting weights through prediction and resampling steps. However, in practice, the importance resampling technique often leads to particle degeneracy and sample impoverishment when the state transition model poorly aligns with the posterior belief distribution, especially when the received observation is noisy. We propose an approach that constructs a sequence of bridge distributions between the state-transition and optimal distributions through iterative Monte Carlo steps, better accommodating noisy observations in online POMDP solvers. Our algorithm demonstrates significantly superior performance compared to state-of-the-art methods when evaluated across multiple challenging POMDP domains.

4. B. Luo, Y. Zhang, A. Dubey, and A. Mukhopadhyay, Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processes, in Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, Richland, SC, 2024, pp. 1301–1309.
   Bibtex Abstract PDF

   @inproceedings{baiting2024AAMAS,
     author = {Luo, Baiting and Zhang, Yunuo and Dubey, Abhishek and Mukhopadhyay, Ayan},
     booktitle = {Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems},
     title = {Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processes},
     year = {2024},
     address = {Richland, SC},
     pages = {1301–1309},
     acceptance = {20},
     publisher = {International Foundation for Autonomous Agents and Multiagent Systems},
     series = {AAMAS '24},
     contribution = {colab},
     isbn = {9798400704864},
     keywords = {monte carlo tree search, non-stationary environments, online planning, sequential decision-making},
     location = {Auckland, New Zealand},
     numpages = {9}
   }
   

   A fundamental challenge in sequential decision-making is dealing with non-stationary environments, where exogenous environmental conditions change over time. Such problems are traditionally modeled as non-stationary Markov decision processes (NS-MDP). However, existing approaches for decision-making in NS-MDPs have two major shortcomings: first, they assume that the updated environmental dynamics at the current time are known (although future dynamics can change); and second, planning is largely pessimistic, i.e., the agent acts "safely” to account for the non-stationary evolution of the environment. We argue that both these assumptions are invalid in practice-updated environmental conditions are rarely known, and as the agent interacts with the environment, it can learn about the updated dynamics and avoid being pessimistic, at least in states whose dynamics it is confident about. We present a heuristic search algorithm called Adaptive Monte Carlo Tree Search (ADA-MCTS) that addresses these challenges. We show that the agent can learn the updated dynamics of the environment over time and then act as it learns, i.e., if the agent is in a region of the state space about which it has updated knowledge, it can avoid being pessimistic. To quantify "updated knowledge,” we disintegrate the aleatoric and epistemic uncertainty in the agent’s updated belief and show how the agent can use these estimates for decision-making. We compare the proposed approach with multiple state-of-the-art approaches in decision-making across multiple well-established open-source problems and empirically show that our approach is faster and more adaptive without sacrificing safety.

5. A. Pettet, Y. Zhang, B. Luo, K. Wray, H. Baier, A. Laszka, A. Dubey, and A. Mukhopadhyay, Decision Making in Non-Stationary Environments with Policy-Augmented Search, in Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, Richland, SC, 2024, pp. 2417–2419.
   Bibtex Abstract PDF

   @inproceedings{pettet2024decision,
     author = {Pettet, Ava and Zhang, Yunuo and Luo, Baiting and Wray, Kyle and Baier, Hendrik and Laszka, Aron and Dubey, Abhishek and Mukhopadhyay, Ayan},
     booktitle = {Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems},
     title = {Decision Making in Non-Stationary Environments with Policy-Augmented Search},
     year = {2024},
     address = {Richland, SC},
     acceptance = {36},
     pages = {2417–2419},
     publisher = {International Foundation for Autonomous Agents and Multiagent Systems},
     series = {AAMAS '24},
     contribution = {lead},
     isbn = {9798400704864},
     note = {extended abstract},
     keywords = {mcts, non-stationary environments, sequential decision-making},
     location = {Auckland, New Zealand},
     numpages = {3}
   }
   

   Sequential decision-making is challenging in non-stationary environments, where the environment in which an agent operates can change over time. Policies learned before execution become stale when the environment changes, and relearning takes time and computational effort. Online search, on the other hand, can return sub-optimal actions when there are limitations on allowed runtime. In this paper, we introduce Policy-Augmented Monte Carlo tree search (PA-MCTS), which combines action-value estimates from an out-of-date policy with an online search using an up-to-date model of the environment. We prove several theoretical results about PA-MCTS. We also compare and contrast our approach with AlphaZero, another hybrid planning approach, and Deep Q Learning on several OpenAI Gym environments and show that PA-MCTS outperforms these baselines.

6. Y. Zhang, B. Luo, A. Mukhopadhyay, D. Stojcsics, D. Elenius, A. Roy, S. Jha, M. Maroti, X. Koutsoukos, G. Karsai, and A. Dubey, Shrinking POMCP: A Framework for Real-Time UAV Search and Rescue, in 2024 International Conference on Assured Autonomy (ICAA), 2024, pp. 48–57.
   Bibtex Abstract DOI PDF

   @inproceedings{zhang2024,
     author = {Zhang, Yunuo and Luo, Baiting and Mukhopadhyay, Ayan and Stojcsics, Daniel and Elenius, Daniel and Roy, Anirban and Jha, Susmit and Maroti, Miklos and Koutsoukos, Xenofon and Karsai, Gabor and Dubey, Abhishek},
     booktitle = {2024 International Conference on Assured Autonomy (ICAA)},
     title = {Shrinking POMCP: A Framework for Real-Time UAV Search and Rescue},
     year = {2024},
     pages = {48--57},
     contribution = {lead},
     doi = {10.1109/ICAA64256.2024.00016},
     keywords = {Three-dimensional displays;Navigation;Markov decision processes;Urban areas;Probabilistic logic;Real-time systems;Trajectory;Maintenance;Time factors;Optimization;Search and Rescue;POMDP;MCTS}
   }
   

   Efficient path optimization for drones in search and rescue operations faces challenges, including limited visibility, time constraints, and complex information gathering in urban environments. We present a comprehensive approach to optimize UAV-based search and rescue operations in neighborhood areas, utilizing both a 3D AirSim-ROS2 simulator and a 2D simulator. The path planning problem is formulated as a partially observable Markov decision process (POMDP), and we propose a novel "Shrinking POMCP" approach to address time constraints. In the AirSim environment, we integrate our approach with a probabilistic world model for belief maintenance and a neurosymbolic navigator for obstacle avoidance. The 2D simulator employs surrogate ROS2 nodes with equivalent functionality. We compare trajectories generated by different approaches in the 2D simulator and evaluate performance across various belief types in the 3D AirSim-ROS simulator. Experimental results from both simulators demonstrate that our proposed shrinking POMCP solution achieves significant improvements in search times compared to alternative methods, showcasing its potential for enhancing the efficiency of UAV-assisted search and rescue operations.