Reinforcement learning has achieved remarkable success in landmark applications such as AlphaGo and AlphaFold, and is increasingly expected to play a central role in emerging domains such as autonomous driving. A substantial body of work has established theoretical foundations for classical reinforcement learning algorithms, including temporal-difference learning, policy iteration, and Q-learning. By contrast, policy gradient descent, despite its widespread use, remains less well understood from a theoretical perspective, largely because of its non-convex structure and the complexity of the underlying functional space. Existing convergence analyses typically require uniform regularity assumptions on the policy, viewed as a feedback control function, along the gradient flow; moreover, the resulting convergence rates depend on these regularity bounds. In this work, we significantly strengthen the existing convergence theory. Our key insight is to relate policy gradient descent to a mirror flow on the space of probability measures over controlled trajectories, where the Bregman divergence is induced by the convex control cost. We rigorously prove a JKO-type convergence result showing that the discrete-time mirror flow converges to its continuous-time counterpart, and we identify the limiting dynamics as a preconditioned policy gradient descent flow on the space of control processes. Leveraging this mirror-flow perspective, we establish an exponential convergence rate that is independent of policy regularity. The convergence holds both in Bregman divergence and in the value of the associated control problem.
