Optimal Control of Markov Chains Admitting Strong and Weak Interactions

Sheldon Ira Wolk

Doctoral Dissertation, Year: 1988, Advisor: John S. Baras

Abstract

Measurements of the state of a continuous time Markov Chain having an infinitesimal generator that is a function of a small parameter, E, are corrupted by additive white noise. Two cases are considered: (1) Regulated perturbations, in which the functional dependence of the infinitesimal generator on E models the effects of a weak signal in noise, and (2) Singular perturbations, in which the small parameter E is introduced to model the behavior of a Markov Chain having two time scales.

The problem of finding approximations to the optimal control of these systems under an integral cost criterion is described. Chaging probability measure, the problem is transformed into the equivalent problem of optimal control on a fully observed state space under a linear cost criterion. The dynamics of the transformed problem are described by the Zakai equation for the unnormalized conditional probability which is bilinear in form.

The regular perturbation problem is shown to decompose into a family of optimization problems, the optimizing control for each providing higher order approximations to the optimal control for the original problem. The first problem in this family of optimization problem is deterministic. All other problems have quadratic costs and linear dynamics.

The singular perturbation problem is shown to decompose into two optimization problems. The first of these optimization problems, the "limit problem", captures the effect of show process in the orignal system. The solution to this limit problem approximates the value function for the original problem by O(E). The rest of the decomposition is the so-called "fast problem", which captures effects of processes that change quickly with respect to those in the limit problem.