We consider a class of stochastic control problems which have been widely used in optimal foraging theory and financial modeling. The optimal state process has two distinct dynamics, characterized by two pairs of drift and diffusion coefficients, depending on whether it takes values bigger or smaller than a threshold value. Adopting a perturbation type approach, we find an expression for potential measure of the optimal state process. We then obtain an expression for the transition density of the optimal state process by inverting the associated Laplace transform. Properties including the stationary distribution of the optimal state process are discussed. Finally, an expression of the value function and the optimal control are given for such stochastic control problems. As an application, we transform the continuous-time two-armed bandit problem in finance into our control framework to derive the optimal investment strategy which maximizes the probability of reaching a specified threshold.
