What's Inside

Research on human and animal behavior has long emphasized its hierarchical structure, according to which tasks are comprised of subtask sequences, which are themselves built of simple actions. The hierarchical structure of behavior has also been of enduring interest within neuroscience, where it has been widely considered to reflect prefrontal cortical functions. In recent work, we have been reexamining behavioral hierarchy and its neural substrates from the point of view of recent developments in computational reinforcement learning. Specifically, we've been considering at a set of approaches known collectively as hierarchical reinforcement learning, which extend the reinforcement learning paradigm by allowing the learning agent to aggregate actions into reusable subroutines or skills.

A close look at the components of hierarchical reinforcement learning suggests how they might map onto neural structures, in particular regions within the dorsolateral and orbital prefrontal cortex. It also suggests specific ways in which hierarchical reinforcement learning might provide a complement to existing psychological models of hierarchically structured behavior. A particularly important question that hierarchical reinforcement learning brings to the fore is that of how learning identifies new action routines that are likely to provide useful building blocks in solving a wide range of future problems. Here and at many other points, hierarchical reinforcement learning offers an appealing framework for investigating the computational and neural underpinnings of hierarchically structured behavior. In addition to introducing the theoretical framework, I'll describe a first set of neuroimaging and behavioral studies, in which we have begun to test specific predictions.

The human capacity to “go beyond the information given” arguably marks a discontinuity between human and non-human intelligence. By exploiting various types of bootstrapping operations, human children are apparently able to transform perceptual inputs into explicit relational structures that distinguish the roles of relations from the objects that fill them. The capacity for what I term “role-based” relational reasoning is revealed in the ability to see analogies and understand abstract causal relations. I will review evidence for an apparent inter-species gap in relational abilities, and discuss the possible neural basis of role-based relational reasoning. Using comparative relations as an example, I will sketch a new computational model based on Bayesian inference that demonstrates how statistical learning, bootstrapped by subsymbolic analogical mapping, might allow role-based relational reasoning to emerge in human children.