Yunuo Zhang is a graduate student in the Department of Electrical Engineering and Computer Science at Vanderbilt University and works as a research assistant at the Institute for Software Integrated Systems. He received his Bachelor’s degree in Computer Science and Applied Mathematics from Vanderbilt University in December 2021.

Yunuo Zhang Publications

1. A. Pettet, Y. Zhang, B. Luo, K. Wray, H. Baier, A. Laszka, A. Dubey, and A. Mukhopadhyay, Decision Making in Non-Stationary Environments with Policy-Augmented Search, Proceedings of the 23rd Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2024, Auckland, New Zealand. 2024.
   Bibtex Abstract PDF

   @misc{pettet2024decision,
     title = {Decision Making in Non-Stationary Environments with Policy-Augmented Search},
     author = {Pettet, Ava and Zhang, Yunuo and Luo, Baiting and Wray, Kyle and Baier, Hendrik and Laszka, Aron and Dubey, Abhishek and Mukhopadhyay, Ayan},
     year = {2024},
     booktitle = {Proceedings of the 23rd Conference on Autonomous Agents and MultiAgent Systems, {AAMAS} 2024, Auckland, New Zealand},
     location = {Auckland, New Zealand},
     numpages = {9}
   }
   

   Sequential decision-making under uncertainty is present in many important problems. Two popular approaches for tackling such problems are reinforcement learning and online search (e.g., Monte Carlo tree search). While the former learns a policy by interacting with the environment (typically done before execution), the latter uses a generative model of the environment to sample promising action trajectories at decision time. Decision-making is particularly challenging in non-stationary environments, where the environment in which an agent operates can change over time. Both approaches have shortcomings in such settings – on the one hand, policies learned before execution become stale when the environment changes and relearning takes both time and computational effort. Online search, on the other hand, can return sub-optimal actions when there are limitations on allowed runtime. In this paper, we introduce \textitPolicy-Augmented Monte Carlo tree search (PA-MCTS), which combines action-value estimates from an out-of-date policy with an online search using an up-to-date model of the environment. We prove theoretical results showing conditions under which PA-MCTS selects the one-step optimal action and also bound the error accrued while following PA-MCTS as a policy. We compare and contrast our approach with AlphaZero, another hybrid planning approach, and Deep Q Learning on several OpenAI Gym environments. Through extensive experiments, we show that under non-stationary settings with limited time constraints, PA-MCTS outperforms these baselines.

2. B. Luo, Y. Zhang, A. Mukhopadhyay, and A. Dubey, Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processes, in Proceedings of the 23rd Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2024, Auckland, New Zealand, 2024.
   Bibtex Abstract PDF

   @inproceedings{baiting2024AAMAS,
     title = {Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processes},
     author = {Luo, Baiting and Zhang, Yunuo and Mukhopadhyay, Ayan and Dubey, Abhishek},
     year = {2024},
     booktitle = {Proceedings of the 23rd Conference on Autonomous Agents and MultiAgent Systems, {AAMAS} 2024, Auckland, New Zealand},
     location = {Auckland, New Zealand},
     numpages = {9}
   }
   

   A fundamental (and largely open) challenge in sequential decision- making is dealing with non-stationary environments, where exoge- nous environmental conditions change over time. Such problems are traditionally modeled as non-stationary Markov decision pro- cesses (NSMDP), which can account for a non-stationary environ- mental distribution during planning. However, existing approaches for decision-making in NSMDPs have two major shortcomings: first, they assume that the updated environmental dynamics at the current time are known (although future dynamics can change); and second, planning is largely pessimistic, i.e., the agent acts “safely” to account for the non-stationary evolution of the environment. We argue that both these assumptions are invalid in practice—updated environmental conditions are rarely known, and as the agent inter- acts with the environment, it can learn about the updated dynamics and avoid being pessimistic, at least in states whose dynamics it is confident about. We present a heuristic search algorithm called Adaptive Monte Carlo Tree Search (ADA-MCTS) that addresses these challenges. We show that the agent can learn the updated dynamics of the environment over time and then act as it learns, i.e., if the agent is in a region of the state space about which it has updated knowledge, it can avoid being pessimistic. To quantify “updated knowledge,” we disintegrate the aleatoric and epistemic uncertainty in the agent’s updated belief and show how the agent can use these estimates for decision-making. We compare the proposed approach with the multiple state-of-the-art approaches in decision-making across multiple well-established open-source problems and empirically show that our approach is faster and highly adaptive without sacrificing safety.