NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes

Nathaniel S. Keplinger, Baiting Luo, Iliyas Bektas, Yunuo Zhang, Kyle Hollins Wray, Aron Laszka, Abhishek Dubey, Ayan Mukhopadhyay

Proceeding of the 39th Conference on Neural Information Processing Systems (NeurIPS'25) 2025

PDF

Why This Matters

Many real-world decision-making problems involve non-stationary environments where the reward structure or transition dynamics change over time, yet most research assumes stationary conditions. The lack of standardized benchmarks and simulation interfaces has hindered systematic progress in non-stationary decision-making. NS-Gym is innovative because it provides the first comprehensive toolkit specifically designed for NS-MDPs, enabling researchers to evaluate algorithm robustness and adaptability under realistic environmental change patterns.

What We Did

NS-Gym is an open-source simulation toolkit for non-stationary Markov decision processes that segregates environmental parameter evolution from agent decision-making. The toolkit provides standardized interfaces for defining NS-MDPs, benchmark problems across different environmental change types, and implementations of state-of-the-art algorithmic approaches. NS-Gym enables systematic evaluation of decision-making algorithms under dynamic environments, addressing the gap in standardized benchmarks for non-stationary problems in fields like autonomous driving and resource optimization.

Key Results

Benchmark results comparing six algorithmic approaches across multiple NS-MDP problem types demonstrate clear performance differences in handling environmental changes. The toolkit enables reproducible evaluation of both model-based and model-free approaches under various environmental conditions.

Full Abstract

Cite This Paper

@inproceedings{keplinger2025nsgym,
  author = {Keplinger, Nathaniel S. and Luo, Baiting and Bektas, Iliyas and Zhang, Yunuo and Wray, Kyle Hollins and Laszka, Aron and Dubey, Abhishek and Mukhopadhyay, Ayan},
  title = {NS-Gym: Open-Source Simulation Environments and Benchmarks for Non-Stationary Markov Decision Processes},
  year = {2025},
  abstract = {In many real-world applications, agents must make sequential decisions in environments where conditions are subject to change due to various exogenous factors. These non-stationary environments pose significant challenges to traditional decision-making models, which typically assume stationary dynamics. Non-stationary Markov decision processes (NS-MDPs) offer a framework to model and solve decision problems under such changing conditions. However, the lack of standardized benchmarks and simulation tools has hindered systematic evaluation and advance in this field. We present NS-Gym, the first simulation toolkit designed explicitly for NS-MDPs, integrated within the popular Gymnasium framework. In NS-Gym, we segregate the evolution of the environmental parameters that characterize non-stationarity from the agent's decision-making module, allowing for modular and flexible adaptations to dynamic environments. We review prior work in this domain and present a toolkit encapsulating key problem characteristics and types in NS-MDPs. This toolkit is the first effort to develop a set of standardized interfaces and benchmark problems to enable consistent and reproducible evaluation of algorithms under non-stationary conditions. We also benchmark six algorithmic approaches from prior work on NS-MDPs using NS-Gym. Our vision is that NS-Gym will enable researchers to assess the adaptability and robustness of their decision-making algorithms to non-stationary conditions.},
  booktitle = {Proceeding of the 39th Conference on Neural Information Processing Systems (NeurIPS'25)},
  archiveprefix = {arXiv},
  contribution = {colab},
  eprint = {2501.09646},
  primaryclass = {cs.AI},
  url = {https://arxiv.org/abs/2501.09646},
  keywords = {non-stationary environments, Markov decision processes, benchmark problems, decision-making under change, algorithm evaluation, reinforcement learning}
}
Quick Info
Year 2025
Keywords
non-stationary environments Markov decision processes benchmark problems decision-making under change algorithm evaluation reinforcement learning
Research Areas
POMDP planning scalable AI
Search Tags

Open, Source, Simulation, Environments, Benchmarks, Stationary, Markov, Decision, Processes, non-stationary environments, Markov decision processes, benchmark problems, decision-making under change, algorithm evaluation, reinforcement learning, POMDP, planning, scalable AI, 2025, Keplinger, Luo, Bektas, Zhang, Wray, Laszka, Dubey, Mukhopadhyay