Adaptive Estimation and Optimal Control in Offline Contextual MDPs without Stationarity. - Riddhiman Bhattacharyya; Sayak Chakrabarty; Imon Banerjee (CoRR, 2026)