The cost of control in optimal perceptual decision making
Reinforcement Learning (RL) can specify the best course of action (optimal policy) that an agent should follow, given certain contingencies, to maximize long-term reward. However, agents in RL are conceived as a tabula rasa without any existing behaviors or response tendencies that may interfere with the optimal policy. In the field of optimal control theory, on the other hand, optimal control laws are obtained that steer the existing dynamics of an agent in a way that is compatible with her current goals, but taking into account the cost of control on the optimization problem. We have generalized ideas from Optimal Control (Todorov, PNAS, 2009) to study the cost of control in the context of a familiar task in Systems Neuroscience: Perceptual Decision Making. We derive the optimal course of action for an agent that has to performa a binary choice to determine the nature of a noisy sensory stream, as typical for instance, in the Random Dot Task devised by Newsome and colleagues. The goal of the subject is to maximize long-term reward subject to a control cost that penalizes deviations from a default tendency to make random choices at a particular rate. As in previous studies that did not consider the cost of control (Drugowitsch et al., J. Neurosci., 2012), the agent needs to perform bayesian inference on upcoming observations to update her beliefs about the environment. However, we show that the cost of control changes how beliefs about the sensory world are linked to action. We make and test predictions about behavioral phenotypes under different costs of control and, in particular, we provide a unified account of decision confidence (see e.g., Kepecs, Neuron, 2016) in perceptual decision making tasks.
Institut Champalimaud Centre for the Unknown
Av. Brasília s/n (Doca de Pedrouços)
1400-038 Lisboa, Portugal
Invité par David ROBBE
Salle de conférence INMED, vendredi 2 juin 2017 à 11h