Little Exploration is All You Need. - Henry H. H. Chen; Jiaming Lu (CoRR, 2023)