Lending direction to neural networks

We present a general formulation for a network of stochastic directional units. This formulation is an extension of the Bolzmann machine in which the units are not binary, but take on values on a cyclic range, between 0 and 2Pi radians. This measure is appropriate to many domains representing cyclic or angular values, e.g., wind direction, days of the week, phases of the moon. The state of each unit in a directional-unit Boltzmann machine (DUBM) is described by a complex variable, where the phase component specifies a direction; the weights are also complex variables. We associate a quadratic energy function, and corresponding probability, with each DUBM configuration. The conditional distribution of a unit's stochastic state is a circular version of the Gaussian probability distribution, known as the von Mises distribution. In a mean-field approximation to a stochastic DUBM, the phase component of a unit's state represents its mean direction, and the magnitude component specifies the degree of certainty associated with this direction. This combination of a value and a certainty provides additional representational power in a unit. We prove that the settling dynamics for a mean-field DUBM cause convergence to a free energy minimum. Finally, we describe a learning algorithm and simulations that demonstrate a mean-field DUBM's ability to learn interesting mappings.

Retrieve Paper (pdf)