It is thought that the brain’s judicious reuse of previous computation underlies our ability to plan flexibly, but also that inappropriate reuse gives rise to inflexibilities like habits and compulsion. Yet we lack a complete, realistic account of either. Building on control engineering, here we introduce a model for decision making in the brain that reuses a temporally abstracted map of future events to enable biologically-realistic, flexible choice at the expense of specific, quantifiable biases. It replaces the classic nonlinear, model-based optimization with a linear approximation that softly maximizes around (and is weakly biased toward) a default policy. This solution demonstrates connections between seemingly disparate phenomena across behavioral neuroscience, notably flexible replanning with biases and cognitive control. It also provides insight into how the brain can represent maps of long-distance contingencies stably and componentially, as in entorhinal response fields, and exploit them to guide choice even under changing goals. Models of decision making have so far been unable to account for how humans’ choices can be flexible yet efficient. Here the authors present a linear reinforcement learning model which explains both flexibility, and rare limitations such as habits, as arising from efficient approximate computation Exhaustive, “model-based” computation of action values is time-consuming; thus, it is deployed only selectively (such as early in learning a new task), and when possible, the brain instead bases choices on previously learned  decision variables4. This strategy saves computation, but gives rise to slips of action when cached values are out-of-date.The brain exhibits a remarkable range of flexibility and inflexibility. A key insight from reinforcement learning (RL) models is that humans’ ability flexibly to plan new actions—and also our failures sometimes to do so in healthy habits and disorders of compulsion—can be understood in terms of the brain’s ability to reuse previous computations6. For flexibility, we still lack a tractable and neurally plausible account how the brain accomplishes the behaviors associated with model-based planning. Conversely, the reuse of completely formed action preferences can explain extreme examples of habits (such as a rat persistently working for food it does not want), but fails fully to explain how and when these tendencies can be overridden, and also many subtler, graded or probabilistic response tendencies, such as Pavlovian biases or varying error rates, in cognitive control tasks.However, while the basic concept of adaptive recomputation seems promising, this class of models—even augmented with refinements such as prioritized replay, partial evaluation, and the successor representation—has so far failed fully to account either for the brain’s flexibility or its inflexibility7. However, it also greatly complicates planning, replanning, task transfer, and temporal abstraction in both artificial intelligence and biological settings8.Here, we introduce a new model that more nimbly reuses precursors of decision variables, so as to enable a flexible, tractable approximation to planning that is also characterized by specific, graded biases. The model’s flexibility and inflexibility (and its ability to explain a number of other hitherto separate issues in decision neuroscience) are all rooted in a new approach to a core issue in choice. In particular, we argue that the central computational challenge in sequential decision tasks is that the optimal decision at every timepoint depends on the optimal decision at the next timepoint, and so on. In a maze, for instance, the value of going left or right now depends on which turn you make at the subsequent junction, and similarly thereafter; so, figuring out what is the best action now requires, simultaneously, also figuring out what are the best choices at all possible steps down the line. This interdependence between actions is a consequence of the objective of maximizing cumulative expected reward in this setting and is reflected in the Bellman equation for the optimal values12. It has been argued that the brain does so by some variant of model-based planning; that is, solving the Bellman equation directly by iterative search4. However, we lack a biologically realistic account how this is implemented in the brain3; indeed, because of the interdependence of optimal actions, exhaustive search  seems infeasible for most real-world tasks due to the exponentially growing number of future actions that must each be, iteratively and nonlinearly optimized. It has thus also been suggested that the brain employs various shortcuts that rely on reusing previously computed (“cached”) quantities, notably model-free long-run values15.How, then, can the brain produce flexible behavior? Humans and animals can solve certain replanning tasks, such as reward devaluation and shortcuts, which require generating new action plans on the fly16, which by leveraging cached expectations about which states will be visited in future, can efficiently solve a subset of tasks traditionally associated with model-based planning6. However, it simply assumes away the key interdependent optimization problem by evaluating actions under a fixed choice policy (implied by the stored state expectancies) for future steps. This policy dependence makes the model incapable of explaining how the brain can solve other replanning tasks, in which manipulations also affect future choices17. In general, the precomputed information stored by the SR is only useful for replanning when the newly replanned policy remains similar to the old one. For instance, a change in goals implies a new optimal policy that visits a different set of states, and a different SR is then required to compute it. This is just one instance of a general problem that plagues attempts to simplify planning by temporal abstraction , again due to the interdependence of optimal actions: if my goals change, the optimal action at future steps  likely also change.One such proposal, which is perhaps the most promising step toward a neurally realistic planning algorithm is the successor representation (SR)22 to propose a new model for decision-making in the brain that can efficiently solve for an approximation to the optimal policy, jointly across all choices at once. It does so by relying on a precomputed, temporally abstract map of long-run state expectancies similar to the SR, but one which is, crucially, stable and useful even under changes in the current goals and the decision policy they imply. We term the model linear RL, because it is based on Todorov’s work with a class of linearly solvable Markov decision processes (MDPs)22. It provides a common framework for understanding different aspects of animals’ cognitive abilities, particularly flexible planning and replanning using these temporally abstract maps, but also biases in cognitive control and Pavlovian influences on decision-making, which arise directly from the strategy of reuse.Here, we adopt and build on recent advances in the field of control engineeringThe model is based on a reformulation of the classical decision problem, which makes “soft” assumptions about the future policy (in the form of a stochastic action distribution), and introduces an additional cost for decision policies which deviate from this baseline. This can be viewed as an approximation to the classic problem, where soft, cost-dependent optimization around a baseline, which we hereafter call the default policy, stands in for exact optimization of the action at each successor state. This enables the model efficiently to deal with the interdependent optimization problem. Crucially, the form of the costs allows the modified value function to be solved analytically using inexpensive and biologically plausible linear operations. In particular, the optimal value of any state under any set of goals depends on a weighted average of the long-run occupancies of future states that are expected under the default policy. Therefore, we propose that the brain stores a map of these long-run state expectancies across all states , which gives a metric of closeness of states under the default policy. Because the DR depends only on the default policy, and can be used to compute a new optimal policy for arbitrary goals, the model can solve a large class of replanning tasks, including ones that defeat the SR.23), and could provide some sort of mechanism for spatial24 and mental navigation26. However, it has been unclear how this and similar long-run temporal abstractions are actually useful for planning or navigation, because as mentioned long-run (low-frequency) expectancies over task space are not stable across tasks due to the interdependence of policy, goals, and trajectories28. For instance, because the SR only predicts accurately under the training policy, to be even marginally useful for replanning the SR theory predicts grid fields must continually change to reflect updated successor state predictions as the animal’s choice policy evolves, which is inconsistent with evidence31. The linear RL theory clarifies how the DR, a stable and globally useful long-run map under a fixed default policy, can serve flexible planning. Our theory also provides a new account for updating maps in situations which actually do require modification—notably, the introduction of barriers. We show how these give rise to additional, separable basis functions in the corresponding DR, which we associate with a distinct class of entorhinal response fields, the border cells. This aspect of the work goes some way toward delivering on the promise of such response as part of a reusable, componential code for cognitive maps25.Our modeling approach also addresses a number of seemingly distinct questions. First, the stability of the DR across tasks makes it a candidate for understanding the role in decision making of multiscale, temporally abstract representations in the brain, notably grid cells in the medial entorhinal cortex. These cells show regular grid-like firing patterns over space, at a range of frequencies, and have been argued to represent something akin to a Fourier-domain map of task space , there are many other phenomena which appear as more graded or occasional biases, such as Stroop effects33, Pavlovian tendencies35, slips of action36, and more sporadic failures of participants to solve replanning tasks6. The default policy and cost term introduced to make linear RL tractable offers a natural explanation for these tendencies, quantifies in units of common-currency reward how costly it is to overcome them in different circumstances, and offers a rationale and explanation for a classic problem in cognitive control: the source of the apparent costs of “control-demanding” actions.Finally, linear RL addresses the flip side of how the brain can be so flexible: why, in some cases it is inflexible. We suggest that this is simply another aspect of the same mechanisms used to enable flexible planning. While it has long been suggested that fully model-free learning in the brain might account for extreme cases of goal-inconsistent habits  a biologically realistic, efficient and flexible account of decision making; (ii) a computational understanding of entorhinal grid and border cells that explains their role in flexible planning, navigation, and inference; (iii) a soft and graded notion of response biases and habits; (iv) an understanding of cognitive control that naturally links it to other aspects of decision systems; and (iv) a normative understanding of Pavlovian-instrumental transfer (PIT).s, and at each they receive some reward or punishment r and choose among a set of available actions a, which then affects which state they visit next8. The objective in this setting is typically to maximize the expected sum of future rewards, called the “value” function. Formally, the optimal value 7:st, rt, and at denote the state, reward, and action at time t, respectively. Across all states, this results in a set of interdependent optimization problems, which can be solved, for instance, by iterative search through the tree of future states, computing the maximizing action at each step8. However, in realistic tasks with large state spaces, this iterative, nonlinear computation may be intractable.In Markov decision tasks, like mazes or video games, the agent visits a series of states π*; for instance, we can evaluate the max in Eq. , then Eq.  in general does not equal the value, π and S until they eventually converge to π*, i.e., the classic policy iteration algorithm.If we instead assumed that we were going to follow some given, not necessarily optimal, action selection policy then Eq.  would beentclass1pt{minimalinear RL, greatly simplifies the Bellman equation22. In this paper, we build on this foundation to revisit questions of flexibility and inflexibility in biological learning. To derive this simplification, we first assume a one-to-one, deterministic correspondence between actions and successor states  = 1, which is simply denoted by its destination, s′). This assumption, which we relax later, fits many problems with fully controllable, deterministic dynamics, such as spatial navigation . Second, linear RL seeks to optimize not a discrete choice of successor state (action), but a stochastic probability distribution π over it22. Finally, it redefines the value function to include not just the one-step rewards r but also at each step a new penalty22, called a “control cost,” KL(π||πd), which is increasing in the dissimilarity (Kullback–Leibler (KL) divergence) between the chosen distribution π and some default distribution, πd.A line of research in control engineering has shown that a change in the formulation of this problem, which we refer to as 22: the default policy πd represents a set of soft assumptions about which actions will be taken later, which are optimized into an optimal stochastic distribution π* that is approximately representative of the optimal (deterministic) subsequent choices in the original problem.Linear RL is most naturally a formalism for modeling tasks in which there are some default dynamics  and costly actions to modify them . Alternatively, here we view it as an approximation to the original value function, where the additional penalty terms modify the original problem to a related one that can be more efficiently solved. This is because linear RL deals with the problem of the interdependence of the optimal actions across states22:v* is a vector of the optimal values (now defined as maximizing cumulative reward minus control cost) for each state; r is a vector of rewards at a set of “terminal” states ; P is a matrix containing the probability of reaching each goal state from each other, nonterminal, state; and the key matrix M, which we call the DR, measures the closeness of each nonterminal state to each other nonterminal state  under the default policy. This is similar to the SR , and v* is systematically related to optimal values as defined in the original problem . Thus, while M does depend on the default policy πd, it is stable over changes in goals and independent from π* in the sense that it can usefully find optimized policies π*(solving the interdependent optimization problem) even when these are far from πd. In comparison, vπ (computed from the SR: Sπ) is only a useful approximation to v* (and thus only helpful in finding a new π*) when the SR’s learned policy π is near the target policy π*. Effectively, linear RL works by introducing a smooth approximation of the “max” in Eq. (πd) of a set of values approximates the maximum. The control costs, then, simply capture the difference between the original solution and the smooth approximate one. Note that distinguishing between terminal and nonterminal states is necessary, as only for this type of finite decision problem are the optimal values linearly computable; however, this places few limits on the flexibility of the model (see “Discussion”).Efficient solution is possible because, substituting the penalized rewards into the Bellman equation, the optimal value function is now given by a non-recursive, linear equation, exceptmentclass2pt{minim14 that optimally evaluate the tree up to a certain depth, then “prune” the recursion at that leaf by substituting the exact average value over the remaining subtree  in a way that facilitates finding the optimal values, but is independent of the goal values r, and the resulting optimized value and policy  or transition map  of a previously learned taskems Fig. .Fig. 3Lir in Eq. , because the SR, Sπ, even though learned under the original policy π, is for typical tasks good enough to compute usable new values from the new reward vector5. However, there are many other, structurally similar revaluation tasks—in particular, those with several stages of choices—that defeat the SR. We considered a slightly different revaluation task, which Russek et al.5 and Momennejad et al.6 termed “policy revaluation” that has this property. Here human subjects were first trained to navigate a three-stage sequential task leading to one of the three terminal states , because under the original training policy, the cached successor matrix does not predict visits to the previously low-valued state17. That is, it computes values for the top-level state .Importantly, if the SR is learned with respect to the policy used during the training phase, then it will imply the wrong choice in the test phase . In fact, the rank of matrix MB is equal to the number of states whose transition has changed. With these in place, the linear RL model can solve this task efficiently and computes the modified values and optimized policy using the old DR after updating it with simple operations  that could be stored in hippocampal–stratial projections. But using this information for planning requires exhaustive evaluation, e.g. by replay13, and strongly suggesting a role for map-like representations of longer-scale relationships (aggregating multiple steps) to simplify planning38.The linear RL model also highlights, and suggests a resolution for, a central puzzle about the neural representation of cognitive maps or world models. It has long been argued that the brain represents a task’s structure in order to support planning and flexible decision makingentclass1pt{minima24 and abstract25 state spaces. However, the specific computational role of these representations in flexible planning is still unclear. A key concept is that they represent a set of basis functions for quickly building up other functions over the state space, including future value predictions like 23 and also future state occupancy predictions like the SR39. By capturing longer range relationships over the map, such basis functions could facilitate estimating or learning these functions23. In particular, the graph Laplacian  generalizes Fourier analysis to an arbitrary state transition graph, and produces a set of periodic functions similar to grid fields28, including potentially useful low-frequency ones. Although there are clearly many different decompositions possible, this basic approach seems applicable to neural representations of long-run maps like the SR and DR, and potentially to compressed or regularized learning of them23.Indeed, grid cells in entorhinal cortex represent long-range (low-frequency) periodic relationships over space, and theoretical and experimental work has suggested that they play a key role in representation of the cognitive map and support navigation in both physicalπ) are of limited utilty for transferring to new tasks17. Accordingly, algorithms building on these ideas in computer science (such as “representation policy iteration”27) iteratively update basis functions to reflect changing policies and values as each new task is learned. It has been unclear how or whether representations like this can usefully support more one-shot task transfer, as in the experiments discussed in the previous section.The puzzle with this framework is that, as mentioned repeatedly, the long-range transition map is not actually stable under changes in goals, since it depends on action choices (“max”) at each step of Eq. : in effe30  include information about local costs along a path; this implies the prediction that environmental features that make locomotion difficult, like rough terrain or hills, should modulate grid responses (see “Discussion”).As shown in the previous section, linear RL suggests a resolution for this problem, since the DR is similar to the SR but stably useful across different reward functions and resulting choice policies. In particular, the comparison between Eqs.  and 4)  4) shows30 Fig. . Howevers30 Fig. ; grid ce 31 Fig. . SimilarAs we have already shown, one aspect of the environment that does require updating the DR if it changes is the transition structure of the environment, such as barriers. In simulating the Tolman detour task Fig.  we solve41, such as the wall; and are generic in the sense that they retain this tuning at least across changes in the environment’s geometry. Assuming that the DR has been represented using a combination of features from a low-rank basis set, such as its eigenvectors, the columns of the matrix term for updating the DR show remarkable similarity to the border cells  representation for a baseline map, and other cells could represent the contribution of additional environmental features. Here, we highlight the relevance and importance of this computational approach in the context of entorhinal border cells Fig. . This islls Fig. . This brMB as separate additive components, e.g. border cells. The second is to adjust the baseline map  in place, e.g., via experiential learning or replay to incorporate the change. The latter approach implies that the geometry of the grid cells themselves would be affected by the barriers; the former that it would not be. There is some evidence that some grid cells show sensitivity to barriers and others are invariant to barriers, and that this might depend also on the extent of training in the environment29. Therefore, it might be the case that MB is initially represented separately and later integrated into the map if the environment is stable.In fact, our framework (Eq. ) impliesWe have so far focused on environments with deterministic transitions, such as mazes, in which each action reliably selects the next state. This includes many nontrivial sequential decision tasks but excludes other stochastically controllable domains that are also relevant to biology. The assumption of fully controllable dynamics is part of what enables linear RL to work, because it allows policy optimization to occur over the continuous, differentiable space of state transition probabilities. However, it is straightforward to extend this approach to stochastic tasks by adding an additional step of approximation. First, we solve linearly for the optimal transition dynamics as though the task were fully controllable; next, choose the action selection policy that comes closest to achieving these dynamics.  The question then arises to what extent this approach can account for planning in biological organisms. Here we exemplify this approach in the stochastic sequential decision task that has been most extensively studied in neuroscience, and then consider its limitations.42. Each trial of this task , then pursuing this reward using an updated policy implies increasing the chance of taking the same-first level action to reach the rewarded state if the experienced transition was common, but instead switching to the alternative first-level action if the experienced transition was rare. People and animals’ choices display this type of sensitivity to the transition model42; as expected, linear RL (extended to the stochastic case) also successfully produces this pattern  are similar enough to those actually achievable given stochastic control (to choose the action that leads there with 70% probability). However, it is possible to construct scenarios in which this is not the case, and the approximation strategy would fail. The main issue again comes down to the interdependence of policy across states: there are cases in which ignoring action stochasticity at some state has a dramatic effect on the optimal policy at other, predecessor states. For example, in the otherwise similar Markov task of Fig. We have stressed the usefulness of linear RL for enabling flexible behavior. However, because this is permitted by the default policy, the model also offers a natural framework for understanding biases and inflexibilities in behavior—and phenomena of cognitive control for overcoming them—as necessary consequences of the very same computational mechanisms. The default policy represents soft, baseline assumptions about action preferences, which (on this view) are introduced because they help efficiently though approximately to solve the problem of forecasting the trajectory of optimal future choices during planning. So far, we have simulated it as unbiased (uniform over successors), which works well because of the insensitivity of the algorithm to the default policy. However, the same insensitivity equally allows for other, non-uniform or dynamically learned default policies. In situations where action choice preferences exhibit stable regularities, it can be an even better approximation to build these in via a non-uniform default. A non-uniform default policy softly biases the model towards actions that are common under it. This aspect of the model naturally captures systematic biases in human behavior, such as habits, Stroop effects, and Pavlovian biases (next sections), and suggests a rationale for them in terms of the default policy’s role in facilitating efficient planning.In previous sections, we considered a uniform default policy that did not change in the course of decision making. Without contradicting these observations , one can elaborate this model by assuming that the default policy might itself change gradually according to regularities in the observed transitions . Of course, there are many ways to accomplish this; for concreteness, we use a simple error-driven delta rule with a small step-size parameter (i.e. learning rate) to train the default policy . Note that there is no need to compute the DR matrix from scratch with every step of policy update. The DR can be efficiently updated from the old DR using the same matrix inversion identity used in previous sections.In the long run, this procedure results in biases in the default policy, which then softly biases the decision policy; this produces both a higher probability of errors on individual choices  and a resulting, more global distortion of sequential choice trajectories. Even when the step-size is small, overtraining can ultimately substantially bias the default policy toward the status quo policy. The degree to which overtraining biases the decision policy also depends on a constant parameter in the model, which scales the control cost against rewards .Cognitive control has been defined as the ability to direct behavior toward achieving internally maintained goals and away from responses that are in some sense more automatic but not helpful in achieving those goalsThese control costs trade off in planning against the rewards for different actions, and lead (through the stochastic resulting policy) to biased patterns of errors. Figure 50 , but not contingent on the organism’s behavior. Famously, such associations when learned can trigger reflexive responses . More confusingly, such noncontingent experience can also affect later  choices over actions  which are otherwise thought to be controlled by the learned association between the actions and the outcomes. This phenomenon is known as PIT. Puzzlingly, this happens even though the Pavlovian cues are objectively irrelevant to the actions’ outcomes55, instead predicting that the presence of the stimulus should not influence behavior in the test phase, because actions predict the same outcome contingencies regardless of the stimulus.In a typical PIT task Fig. , animals53.Linear RL naturally explains PIT as another example of biases arising from a learned default policy, because during the Pavlovian phase the agent should learn that the reward outcome occurs more often in the presence of the conditioned stimulus, which is reflected in the default contingencies. Therefore, during the test phase, the presentation of a conditioned stimulus elicits a default policy biased toward the corresponding outcome occurring, which favors choosing the corresponding action Fig. . Further56 to neuroscience a computational model of decision making in the brain, called linear RL, which enables efficient (though approximate) global policy optimization by relying on soft relaxation away from default, stochastic policy expectations.A central question in decision neuroscience is how the brain can store cognitive maps or internal models of task contingencies and use them to make flexible choices, and more particularly how this can be done efficiently in a way that facilitates reuse of previous computations and leverages long-run, temporally abstract predictions without compromising flexibility. To help answer this question, we identify a core issue underlying many difficulties in planning, replanning, and reuse, which is the interdependence of optimal actions across states in a sequential decision task. To solve this problem, we import from control theory57. However, linear RL corrects serious problems that hobble the practical applicability of the SR. The DR, unlike the SR, exerts only a weak bias toward the default policy, and so delivers on the promise of a stable cognitive map11 that can reuse substantial computation to transfer learning across contexts without sacrificing flexibility. This allows the model to explain animals’ ability to solve reward and policy revaluation problems that otherwise would require exhaustive, biologically unrealistic model-based search. For the same reason, the model also helps to deliver on the idea that grid cells in entorhinal cortex could provide a broadly useful neural substrate for such a temporally abstract map. And the model’s remaining inflexibilities—in general, soft, stochastic biases rather than hard failures—connect naturally with phenomena of cognitive control and Pavlovian biases and provide a strong theoretical framework for understanding the role of many such biases in both healthy and disordered choice.This leverages the DR, a stored, long-run predictive map of state and cost expectancies under the default policy. The DR is closely related to the SR, and inherits many of the appealing features that have generated current excitement for it as a neuroscientific model58. First, the framework encompasses many different subprocesses that have been previously associated with different brain systems . The current framework suggests how these processes might interact, but we do not mean to imply that they are all the same thing. Furthermore, even though a particular subfunction—like map/model learning—may seem unitary in an abstract, computational sense, it may nonetheless be supported by different brain systems in different contexts, such as social vs spatial.This theory provides a unified and realistic computational framework for model-based planning in the brain and, therefore, provides a foundation for some suggestions here and much future work studying the neural substrates of different aspects of such planning. However, we should emphasize that unification at the computational level does not necessarily mean that a single neural system  governs all these computations22 and soft Q-learning59. Also, as discussed below, there exist alternative rationales that can motivate the same form of KL-divergence costs, where the default policy enters as a baseline expectation for efficient capacity-constrained value coding, or as a prior for Bayesian planning-as-inference. These perspectives are not necessarily mutually exclusive, but our proposal to view flexible planning as fundamental has the benefit of providing a unified view on two important but otherwise mostly separate areas of cognitive neuroscience, i.e., flexible planning and cognitive control.We motivated linear RL from a computational perspective, in which the central question is how the brain efficiently reuses previous computations for flexible replanning. Mathematically, this is enabled by introducing a control cost term, given by the dissimilarity (KL divergence) between a default policy, and the final, optimized decision policy. We argued that this penalty allows the model to explain a range of “model-based” planning and transfer phenomena, and simultaneously explain a separate set of inflexibilities in decision making, such as biased behavior, Pavlovian biases, and cognitive control Figs. –9, while17.The basic planning operation in linear RL is matrix-vector multiplication, which is easily implemented in a single neural network layer. The theory offers new insights into the systems-level organization of this computation. In particular, the model realizes the promise of a representation that factors a map representing the structure of environment, separate from an enumeration of the current value of the goals in the environment. This facilitates transfer by allowing update of either of these representations while reusing the other. Previous models, like the SR, nominally exhibit this separation, but the hard policy dependence of the SR’s state expectancies means that goal information, in practice, inseparably infects the map and interferes with flexible transferM) and a second matrix representing one-step connections from nonterminal states to goals (P). This is a necessary restriction, in that only for this type of finite decision problem are the optimal values linearly computable. However, this classification is not inflexible, because we also introduce new techniques (based on matrix inversion lemmas) that allow dynamically changing which states are classed as goals. This allows the model (for example) to plan the best route to any arbitrary location in a maze , and nonterminal states , and dividing the map into one matrix encoding long-run interconnectivity between nonterminal states  are represented as terminal goal states with corresponding value in r, and the rest of the  states contain only costs, constant everywhere, meant to capture the cost of locomotion. But, in general, state-dependent costs can be included for nonterminal states as well. These in effect modulate the “distance” between states represented within the DR (see “Methods”). Nevertheless, this leads to the testable prediction that to whatever extent state-specific costs are accounted for within nonterminal states, they should affect hypothetical neural representations of the DR, such as grid cells. For instance, unlike for the SR, the DR predicts that by increasing locomotion cost, hills or rough terrain should increase “distance” as measured in the grid map .This last point raises several interesting directions for future work. First, although there is evidence that humans choose their goal and plan towards that goal42. This approximation may be sufficient in practice for many realistic cases, though we also show that tasks can be constructed to defeat it  and instrumental lever-pressing, but does not exactly map to tasks that include irreducibly stochastic state transitions. We show, however, that it is possible to address the latter class of tasks by approximating them as controllable and producing an intermediate approximate solution via linear RL. Though extremely simple, this approach can solve tasks such as two-step noisy Markov decision tasks that we and others have used to study model-based planning in humans and animals it Fig. . It rema5 and one newly introduced here. First, like the SR, the DR can be learned gradually by actual or replay-based sampling of the environment, using a temporal difference rule16. Second, again like the SR, the DR can be constructed from the one-step transition matrix and costs (which can themselves be learned directly by Hebbian learning) by a matrix inversion, or equivalently a sum over a series of powers of a matrix. The latter form motivates attractor methods for computing the inverse iteratively by a simple recurrent network64.We have stressed how the DR can be used for planning, and also how it embodies substantial, reusable computation , relative to simpler, easy-to-learn map representations like the one-step state adjacency model A third possibility for learning the DR follows from the method we introduce for using matrix inversion identities to efficiently update the DR in place to add additional goals, barriers, or shortcuts (see “Methods”). This works by expressing the inverse matrix in terms of the inverses of simpler component matrices (one of which is the pre-update DR), rather than for instance by updating the transition matrix and then, expensively, re-inverting the whole thing. For instance, we used this to solve tasks, such as Tolman’s detour task, in which the transition structure of the environment changes. It could also be used, state by state or barrier by barrier, as a learning rule for building up the DR from scratch.65. Here, we showed that the term required to update the DR when encountering a wall remarkably resembles entorhinal border cells41. Therefore, our theory unifies the functional roles of entorhinal grid and border cells in planning and navigation, both as neural codes for making long-term maps that are useful for planning. We believe that this is the first step toward constructing a fully componential model of maps, although producing a border cell than can be translated and reused at different locations in compositional fashion will likely involve additional nonlinear operations. Beyond its direct consequences, our model also opens the way for future work giving a more detailed account of different patterns of change in entorhinal maps under different environmental changes. Empirically, such changes arise both gradually and abruptly. Although we have emphasized the robustness of planning to the choice of default policy, since the DR depends on the default policy, any situations such as overtraining that produce a biased default policy  into the sum of component maps, each associated with different sub-graphs of the transition space, offers a promising direction for a direct neural implementation for representing and constructing maps componentially: via summing basis functions, here represented by the low-rank Woodbury updates or some further function of them. This idea dovetails with—and may help to formalize and extend—the emerging idea that maps in the brain are built up by composing basis functions, such as those putatively represented in the grid cellsicy Fig.  could ul2 and used a graph Laplacian approach, in which the DR is represented using its eigen-decomposition. Although there are a number of reasons  to think that the brain represents maps via some decomposition similar to this, we are not committed to this specific decomposition as a mechanistic neural model. Instead, our main motivation for taking this approach was descriptive, to investigate the properties of the DR and to highlight its relationship with multiscale, periodic functions qualitatively similar to grid cells. We believe that this approach is revealing, despite its many limitations, including the fact that eigenvectors of the DR (and the SR) show a wide range of frequencies, not only hexagonally symmetric fields65. Notably, eigenvectors can also be learned using biologically plausible networks trained by Oja’s rule70, and it has been suggested that since eigen-decomposition is commonly used for compression, this approach could also be used to allow a regularized SR or DR to be learned and represented more efficiently2. However, this results in a loss of information if only a subset of eigenvectors is used. Nevertheless, the eigen-decomposition is by no means the only possible approach for efficient representation of the DR. In fact, the DR, at least in open fields with a fixed cost, has a redundant structure that can be exploited for efficient and exact representation, and a fully compositional account of border cells would require additional nonlinearities to account for their translation to different locations in space. This is a topic that goes beyond the scope of the current work and which we plan to pursue in future work.For making the connection between the DR and entorhinal grid fields, we followed Stachenfeld et al.λ, that scales the control costs relative to rewards and corresponds to the temperature in the softmax approximation to the optimization. In the limits of zero, or respectively infinite, control cost scaling, the approximation to the optimum becomes exact, or the default policy dominates completely. How should this parameter be set, and why not always take it near zero to improve the fidelity of the approximation? Linear RL works by multiplying very small numbers (future occupancies) times very large numbers  to approximate the maximum expected value. Making this work effectively across different decision situations in the brain requires careful control of scaling to manage limits on precision . This suggests fruitful connections (for future work) with research on gain control and normalization71, and rational models for choice using noisy representations73.Our model is based on the notion of the default policy, which is a map of expected state-to-state transition probability regardless of the current goals. Unlike previous RL models, such as the SR, linear RL does not entirely rely on the default policy and instead optimizes the decision policy around the default policy. This means that the final optimized policy is between the exact, deterministic optimized policy, and the default. The degree of this bias is controlled by a free parameter, 74 and from the perspective of Bayesian planning as inference76. Here, the default policy plays the role of prior over policy space and rewards play the role of the likelihood function. In this case, the decision policy is the posterior that optimally combines them56. Then, how much the default should influence the decision policy depends on how informative a prior it is (e.g. how reliable or uncertain it has been previously). This also suggests another distinct perspective on the default policy’s role, in the model, in producing prepotent biases that can be overcome by cognitive control46. On this view, it serves to regularize behavior toward policies that have worked reliably in the past; and deviations from this baseline are presumptively costly. This perspective also provides the theoretical basis to exploit the machinery of probabilistic graphical modeling for unifying models of planning and inference in neuroscience.The same tradeoff can also be understood from principles of efficient information theoretic codingv*) is minimized whenever the future π* is equal to the default πd. Any algorithm for learning policies might be appropriate, then, for finding a πd that is likely to be near-optimal in the future. We exhibit one simple policy-learning algorithm  but other habit-learning models including model-free actor-critic learning78 are equally applicable. A related idea has also been recently proposed in the context of a more explicitly hierarchical model of policy learning: that a default policy (and control-like charges for deviation form it) can be useful in the context of multitask learning to extract useful, reusable policies79. Separately, an analogous principle of identification of task structure that generalizes across tasks in a hierarchical generative model has also been proposed as a model of grid and place cell responses that shares some similarities with our account26. Future work remains to understand the relationship between the considerations in both of these models—which involve identifying shared structure across tasks—and ours, which are motivated instead more by efficient planning within a task.Indeed, our framework can encompass many different possibilities not just for how strongly the default policy is emphasized but also how it is learned or chosen. In general, while the model provides a good approximation to the true optimal values independent of which default policy is used , we can also ask the converse question—which default policy should be chosen to allow for the best approximation and thereby obtain the most  reward? The answer is of course, that the cost term 82, the default policy offers a clear and normative lever for influencing behavior on the basis of emotional (and other) context.The role of the default policy, finally, points at how the linear RL framework provides a richer, more nuanced view of habits and pathologies of decision making than previous computational theories. Although a learned default policy biases behavior, and may modulate accuracy or speed of performance, it trades off against rewards in the optimization. This give and take stands in contrast to much previous work, especially in computational psychiatry, which has often assumed a binary model of evaluation: either flexible values are computed  or they are not . The latter, acting rather than thinking, has been taken as a model of both healthy and unhealthy habits, and especially of compulsive symptoms such as in drug abuse and obsessive compulsive disordersT; second, we initially consider deterministic environments, such as mazes, in which there is a one-to-one map between actions and successor states .In this work, we focus on MDPs with two conditions. First, we assume that there is one or a set of terminal states, 22, in which the agent controls the probabilistic distribution over successor states  and pays an additional control cost quantified as the dissimilarity, in the form of KL divergence, between the controlled dynamics (i.e. decision policy), g(st), defined asλ > 0 is a constant and π = πd and otherwise is positive. We also require that π = 0 if πd = 0. Note that in the limit of zero, or respectively infinite, λ, the gain converges to pure reward (i.e. a standard MDP), or pure cost. Here, λ scales the relative strength of control costs in units of reward .The linear RL model is based on a modification to the value function for this settingv*, is analytically solvable22 . We first define the one-step state transition matrix T, whose  element is equal to the probability of transitioning from state i to state j under the default policy (i.e. probability of the action under the default policy that makes i→ j transition). This contains subblocks, TNN, the transition probability between nonterminal states, and TNT = P, the transition probabilities from nonterminal to terminal states. Then:v* is the vector of optimal values at nonterminal states, r is the vector of rewards at terminal states, and M is the DR matrix defined below. Note that Eq. .The DR matrix D, defined over all states  asrA is the reward vector across all states. Note that since matrix M can be easily computed from D , we refer to both of them as the DR unless specified otherwise. Also note that for defining D, we assumed, without loss of generality (since this assumption does not affect M), that reward at terminal states are not 0.For flexibility in updating which states are viewed as goal states, it is helpful to define a second, more general version of the DR matrix, v* further implies that the policy takes the form of a weighted softmax, where the weights are given by the default policysa is the successor state associated with action a. Thus, for a uniform default policy, the optimal policy is simply given by the softmax over optimal values with the temperature parameter λ. Note also that in the limit as λ → 0, the problem becomes the classical MDP , i.e., we wish to add j to the set of terminal states. Here, we aim to develop an efficient method to plan towards j by using the cached D0, without re-inverting the matrix.Consider an environment with L and L0 are only different in their jth row (because T and T0 are only different in their jth row). We define d, a vector corresponding to the difference in jth row of the two matrices, e is a binary column-vector that is one only on jth element. Using the Woodbury matrix identity, L−1 is given byd and e are row and column vectors, respectively, and therefore D = L−1, we obtainm0 is the jth column of D0.If we define z = MP, which is what is needed for planning using Eq. , we may also further simplify the computation by focusing only on the product sing Eq.  in the nz is given by the submatrix of Finally, d and d is only nonzero on elements associated with immediate state of j (and jth element). If we assume that there is only one terminal state (i.e. j), then j.It is important to note that since  in Eqs.  and 15 aTNN. For example, if transition from state j to i has been blocked, new values for D and z can be computed efficiently using Eqs. TNN. For 0 in Eq. .T0 to T. Suppose Δ using rows of L0 and L corresponding to J:J denotes those states that their transition has been changed, LJ and LJ0, are, respectively, submatrices associated with rows of L and L0 corresponding to J. Using the Woodbury matrix identity  is given byWe employed a similar approach to account for border cells. Suppose that a wall has been inserted into the environment, which changes the transition matrix r to Eq. ), the DRsa denotes the corresponding state (among the set of successor states of s) to action a. In environments with stochastic dynamics, however, there is no such mapping between policy and dynamics, and therefore one additional step is required to extend the linear RL framework to stochastic environments. Here, s to s′ (it is only nonzero for successor states of s). We can then use the framework of linear RL to obtain the optimal transition, u. Therefore, it is easy to see that optimal u is given as before by Eq.  and available actions (columns), then we have π by minimizing the squared error between π is a probability distribution. In practice, in most situations, such as the two-step task , where we set λ = 10 to avoid overflow of the exponential due to large reward values.The only parameter of linear RL is 22. By substituting the gain defined in Eq. (sa denotes the corresponding state (among the set of successor states of st) to action a.For completeness, we present derivation of Eqs. –10) bas bas10) bd in Eq.  into thec:Note that the expectation in the Bellman equation is under the dynamics, which we have replaced it with the policy because they are equivalent here. The expression being optimized in this equation is akin to a KL divergence, except that the denominator in the argument of the log function is not normalized. Therefore, we define the normalization term c is independent of the distribution being optimized π. By multiplying and dividing the denominator of the log by c, we obtainNote that e to Eq. :24\documFurthermore, since the KL divergence is zero, optimal values satisfyv(sT) = r(sT), this system can be solved analytically, which can be written in the matrix Eq. (Across all states, this gives rise to a system of linear equations in the exponential space. Since at terminal states, trix Eq. .Further information on research design is available in the Supplementary InformationPeer Review FileReporting Summary


Background: Delay discounting (DD) and time perspective (TP) are conceptually related constructs that are theorized as important determinants of the pursuit of future outcomes over present inclinations. This study explores their predictive relationships for smoking cessation. Methods: 5006 daily smokers at a baseline wave provided 6710 paired observations of quitting activity between two waves. Data are from the International Tobacco Control (ITC) smoking and vaping surveys with samples from the USA, Canada, England, and Australia, across three waves conducted in 2016, 2018 and 2020. Smokers were assessed for TP and DD, plus smoking-specific predictors at one wave of cessation outcomes defined as either making a quit attempt and/or success among those who tried to quit which was ascertained at the subsequent survey wave. Results: TP and DD were essentially uncorrelated. TP predicted making quit attempts, both on its own and controlling for other potential predictors but was negatively associated with quit success. By contrast, DD was not related to making quit attempts, but high DD predicted relapse. The presence of financial stress at baseline resulted in some moderation of effects. Conclusions: Understanding the mechanisms of action of TP and DD can advance our understanding of, and ability to enhance, goal-directed behavioural change. TP appears to contribute to future intention formation, but not necessarily practical thought of how to achieve goals. DD is more likely an index of capacity to effectively generate competing future possibilities in response to immediate gratification. Smoking cessation is a behaviour that involves weighing a choice for an immediate reward  against some future benefit . Although this temporal tradeoff is often recognized as a major obstacle to smoking cessation, to date, there has been less research in this domain than would be suggested by its centrality.There are two general approaches to understanding the role of intertemporal choice and judgment. Delay reward discounting (DD) is a depreciation of the value of a reward as a function of the time that it is received . As typiCigarette smokers discount the future to a substantially greater extent than non-smokers ,7. DD haConceptually related to DD is the concept of time perspective ,14. HallThe distinction between beliefs and dispositions mirrors the distinction in social psychology between explicit and implicit attitudes . ImpliciOne way to analyze possible mechanisms for differences between beliefs and related dispositions comes from CEOS theory. CEOS is an acronym for context, executive, and operational systems ,26. CEOSCEOS theory treats dispositional measures as indicators of how the person behaves or will respond to contextual stimuli  that are measured independent of conscious beliefs. It locates dispositions within its operational system and explains their mode of influence in terms of the affective forces they generate towards particular action patterns. Operational forces can be inferred from actual behaviour or, in the absence of spontaneous behaviour, experienced feelings and urges to act (sometimes described as “what we want to do”). By contrast, CEOS theory treats explicit attitudes as reports on evaluative beliefs about the conceptual desirability or otherwise of the activity (“what we should do”). Behaviour-related beliefs can be grounded in an attempt to reconcile some or all factual information, social expectations, and relevant experiences into a coherent position, or simply be a rationalization of experienced desires. In the case of evidence-grounded beliefs, there can be a discord with experienced dispositional tendencies. CEOS defines hard-to-change behaviours as ones where there is a discord between desires based on existing dispositions and those based on evaluative beliefs .According to the theory, affective forces  and the tendencies towards action they produce (experienced as urges) are the primary determinants of behaviour  and thusCEOS theory also postulates that executive beliefs only influence behaviour (smoking in this case) when they are brought to mind. Because goal seeking is about future outcomes, it follows that the capacity of the person to consider and value future events relative to current ones should affect the potency of future-oriented beliefs on behaviour. However, it is unclear whether such tendencies would be incorporated within the plans people make, or independently affect the likelihood of plans translating into action.Most smokers have a strong emotional attachment to smoking, a conditioned response built up from smoking thousands of cigarettes ; that is, it has become a habit. The difficulty most smokers experience in trying to quit ,31 demonHarder to imagine scenarios are discounted more than ones that can be more vividly imagined ,38,39 deIn summary, because TP and DD are associated with different mechanisms ,44,45, tThere are several potential moderators of the associations between DD and quitting and to the extent to which they are related, may also apply to TP. Measures of DD involving money are likely to vary as a function of the person’s current economic circumstances. The predictive power of DD is also theorized to be on top of, and potentially interactive with the difficulty of the behaviour change task ,10,11. FThe association between TP and DD: we predict that TP and DD will be moderately negatively correlated; that is, high DD being associated with lower levels of TP.The predictive effects of TP and DD for making a quit attempt and for sustained smoking abstinence. We predict, based on past findings, that TP will predict making quit attempt, while DD will predict success among those who try. However, we are uncertain as to whether these two measures will add explanatory power to prediction in isolation or when major known determinants of each outcome are controlled for.Whether the relationship of TP and DD to quitting and sustaining abstinence from cigarettes is moderated by sociodemographic factors, financial stress, measures of dependence, and motivation. We make no specific predictions.Thus, the three aims and associated hypotheses of this paper are to explore:The current study combined and analysed three waves of the International Tobacco Control (ITC) Four Country Smoking and Vaping Survey . Full descriptions of the ITC Project conceptual framework and methods have been published elsewhere ,48. BrieDemographic covariates. Demographics included age at recruitment , gender, and country . Highest level of education attained was coded into low, medium, and high. Financial stress was assessed by the question “In the last 30 days, because of a shortage of money, were you unable to pay any important bills on time, such as electricity, telephone or rent bills?” We choose to use this variable over household income since it has less missing data (1.2% compared to 5.4% for income) and this variable was associated with income but is perhaps a better measure of current economic circumstances. We also adjusted for self-imposed smoking restrictions in the home since this has been found to explain some of the predictive value of time to first cigarette . All resDelay discounting. DD was assessed by asking respondents questions about which of two amounts of hypothetical money they would prefer, a delayed USD 1000 (the larger later option) or a reduced immediately available amount USD 500 . An example item is “Would you rather have USD 500 now or USD 1000 in 1 day?” In England Pounds replaced dollars), but no adjustment was made for currency values. The delays ranged from 1 h to 25 years and adjusted across five contiguous trials depending on responses to earlier questions ,52. PrioTime perspective. TP was assessed using a single item from the time perspective questionnaire (TPQ) . RespondIntention to quit: Intention to quit smoking was measured using the item “Are you planning to quit smoking…” with the response options “Not planning to quit”, “Sometime in the future, beyond 6 months”, “Within the next 6 months”, ”Within the next month” and “don’t know”. A binary version of this measure was constructed comparing “not planning to quit” with all other options.Wanting to quit: Wanting to quit, a measure of explicit motivation was measured by the item “How much do you want to quit smoking?” with the response options “not at all”, “a little”, “somewhat”, and “a lot” and “don’t know”. A binary version of this measure was also constructed with “not at all” compared to all other responses. Since there was only a moderate correlation between wanting to quit and intention to quit in this analytic sample (r = 0.45) this measure can be considered independent from intention to quit.Quitting self-efficacy: Self-efficacy was measured using the item “If you decided to give up smoking completely in the next 6 months, how sure are you that you would succeed?” and the 5-point response scale was “Not at all sure/Slightly sure/Moderately sure/Very sure/Extremely sure”. “Don’t know responses were coded as “Not at all sure”.Heaviness of smoking index (HSI) (continuous): To avoid restrictions of the imposed categorical classification of continuous data , we usedPerceived addiction to smoking: All regular smokers were asked “Do you consider yourself addicted to cigarettes?” with the response options: ‘not at all’, ‘yes, somewhat addicted’, ‘yes, very addicted’ and ‘don’t know’. This measure was adapted from an earlier five-category measure in the cigarette dependence scale . For preStrength of urges to smoke in past 24 h: At baseline all regular smokers were asked “In general, how strong have urges to smoke been in the last 24 h?” with responses forming a six-point Likert scale . This variable was treated as an ordinal measure with don’t know (less than one percent of responses) treated as missing.Reporting a quit attempt of any duration between Waves, and;Reporting successful abstinence from smoking for one month or longer, and separately for 6 months or longer, independent of current status, between baseline and follow-up, among those smokers who had made a quit attempt. Those quit for less than the criterion were treated as quit failures, and those still quit at follow-up but for less than the criterion were excluded from analysis. At least one month abstinence was chosen since short unsuccessful quit attempts may not be recalled successfully  and 6 moThe two outcomes assessed in this study were:Three main sets of analyses were conducted.Bivariate analyses evaluated the association of time perspective and delay discounting (tertiles) with baseline sociodemographic, dependence measures and motivational variables. We also evaluated the association of all variables with the follow-up outcomes . Pearson Chi-square was used to evaluate categorical variables and independent t-tests were used for continuous measures.Separate logistic regression models for estimating probability of follow-up quit attempts and ≥1 month smoking abstinence were estimated using generalised estimating equations (GEE) with an unstructured correlation matrix, robust standard errors and the binary logit link . The priWe tested for interactions of the time perspective and delay discounting measures by age group (split at 40) , countryAll confidence intervals were computed at the 95% confidence level. Analyses were conducted using Stata V16.1 .p = 0.001), unlike the hypothesized negative relationship . Higher TP was associated with younger age (p < 0.001), higher education level achieved (p < 0.001), reporting financial stress (p < 0.001), both planning and wanting to quit (p < 0.001), higher quit efficacy (p < 0.001), higher perceived addiction to smoking (p = 0.031) and higher mean strength of urges to smoke (p < 0.001), and lower mean HSI (p = 0.009). Higher DD was associated with younger age (p < 0.001), female sex (p < 0.001), lower education level achieved (p < 0.001), reporting financial stress (p < 0.001), not wanting to quit (p = 0.026), higher strength of urges to smoke (p < 0.001), and higher HSI (p < 0.001).The bivariate associations between time perspective, delay discounting and study covariates are presented in p < 0.001) with TP was highest in Canada and lowest in England, and DD highest in Australia and lowest in England  and wanting to quit , but DD was not significantly related to either variable: plan to quit ; want to quit   .Overall, 41.9% of the smokers (n = 2810) reported a quit attempt between baseline and follow-up. Of those smokers who made a quit attempt, 37.3% remained abstinent for at least one month and 17.8% remained abstinent for at least 6 months. p = 0.046) and for six months abstinence . Higher TP was associated with higher likelihood of achieving abstinence in financially distressed smokers but reduced likelihood of abstinence in the non-financially distressed. There was also a significant DD by financial stress interaction for one month abstinence  with lower DD associated with abstinence in the non-financially distressed smokers in contrast to DD being positively associated with abstinence in smokers reporting financial distress. This interaction was not evident for six months abstinence . No significant interaction was observed between TP and DD for one month abstinence  and six months abstinence . There were no other significant interactions for TP or DD with age, gender, country, education level attained, ethnicity and HSI for either quit attempts or smoking abstinence.Turning to the moderator analyses, individual tests for interactions in full models with all covariates  revealedp = 0.009; six months abstinence aOR = 0.93, 95% CI 0.94–0.99, p = 0.014). Similarly, for quit attempts addition of these variables did not significantly alter the adjusted odds ratios for high TP . We also tested for non-linearity of the effect of DD by substituting continuous DD with categorical DD tertiles in our models predicting smoking abstinence. With low DD as the reference category, medium DD was not significant  and high DD  was highly significant for one month abstinence. Similarly, for six months abstinence, medium DD failed to reach significance  and high DD was highly significant with an even lower aOR than for one month abstinence . Taken together the direction and strength of the aORs do indicate linearity. Substituting household income  for financial distress in all models did not significantly alter the aORs for the main study variables.We also conducted sensitivity analyses for the final GEE models by inclusion of previous wave measures of NRT use, quit fatigue, perceived damage of smoking to health, and worry smoking will damage your health. For both one month and six-month smoking abstinence the inclusion of these variables did not significantly affect the aOR for DD individually or when all of these measures were added to the model with  predicted relapse. By contrast, there were non-significant positive trends for high TP to predict success among those financially stressed for both cessation periods. It is also notable that those who were financially stressed reported higher levels of future thinking. Those who are in financial stress are facing situations in the present which threaten their future and working out what to do in the immediate future is practical and important and this may explain the increase in future thinking. However, whether this argument extends to simply thinking about any kind of future is unclear. We hypothesize that the increased focus on the future reported by those who are currently under financial stress is likely to be a focus on medium term issues around resolving the financial stress and/or not making the situation worse; that is, action to achieve goals over the near term; that is within weeks, or months. By contrast, thinking more generally about the future, particularly more distant futures without clear links to the current situation, may be more in the way of fantasizing ; that isIn previous work where we have found motivational measures strongly predictive of smokers making quit attempts, we have interpreted a reverse effect on sustained cessation among those who try as potentially being due to these smokers being more dependent . This anWhat does it mean to value the future? The future as we conceive of it is really just a collection of ideas about possibilities that may happen or that we might be able to facilitate happening. To value the future means at the very least to have the ability to imagine futures in such a way that generate sufficient affective force to influence choices. Our analysis suggests that DD among daily smokers is partly driven by impulsivity, that is a tendency to act on actions strongly cued in the moment, rather than being able to inhibit these actions to allow for other considerations, of which predicted future consequences would be important. Clearly an ability to imagine future scenarios with some degree of credibility is going to be critical to inhibiting tendencies to act for immediate rewards, but this imaginative activity is only likely to affect behavior if it actually occurs at the time the undesirable impulses to act occur. The findings point to the importance of general dispositions that are not adequately represented in assessments of specific behaviours, such as DD, influencing operationally dominant activities such as sustaining smoking abstinence. Overall, findings are broadly as would be expected from CEOS theory and do not fit as well with a conceptualization of DD as measuring some fundamental undervaluing of the future. It would be useful for future studies to explore the relationships between DD and more direct measures of impulsivity and capacity to imagine scenarios which would seem to be its more basic determinants and to explore relationships with other difficult to change behaviours.A recent large meta-analysis  concludeTurning to the implications for smoking cessation, both TP and DD had predictive value over and above smoking specific predictors. However, the implications are likely to differ. In the case of DD, if indeed it is composed of ability to imagine futures when needed and impulsivity, we should be exploring the potential to increase people’s general skills in this area so they can apply them to complex tasks such as smoking cessation. It would seem that prior to actually having quit, smokers do not take into account the potential effect of their impulsivity on their likelihood of success, but this becomes manifest as they face the challenges of resisting temptations to smoke. That we found essentially the same effects for 6 months as for 1 months sustained abstinence, suggests some causal impacts of impulsivity occur in the early weeks of the quit attempt when ability to overcome the more immediate barriers to maintenance of cessation such as overcoming nicotine withdrawal are most likely to be evident. That said, we have not tested for differential effects of time quit by conducting analyses of relapse between 1 and 6 months to see if the importance of DD changes over time quit. In contrast, for TP, our findings suggests that we should be assessing the extent to which people think about their smoking, not just what they want but also about mechanisms for achieving their goals. That is, we should be encouraging smokers to focus on both what they should do and how they should do it if they are to increase their chances of turning desires to quit into successful cessation. Our findings also provide support for behaviour change interventions, that are designed to reduce the impacts of impulsiveness, such as trying to discourage major decision making at times when urges are strong, a strategy used in some effective smoking cessation interventions .Limitations. This study should not be used to estimate the magnitude of the predictive relationships because the measures were taken some time from the behaviours they are designed to predict (quit attempts and success) which would have attenuated these relationships. Further, some of the measures used are theorized to have large situational determinants , and all are subject to some situational variability, so we would expect much greater predictive utility if the gap between the measures and the outcomes was shorter. While both TP and DD are conceived as largely stable characteristics (more similar to traits than states), both constructs are potentially modifiable through situational factors. It would be interesting to assess them throughout a quit attempt to see if increases in DD or reductions in TP might predict relapse. In addition, for quit success, the predictor questions are asked while the person was smoking so the smokers had no current direct experience of the challenges of cessation, so a high level of prediction is never going to be likely.Another major limitation is that for TP, we only used a single item from the TP scale developed by Fong and Hall [2003], so there is a higher degree of unreliability . As we chose the “best” item in terms of item total correlation, our item is clearly in part measuring the core construct and, as noted, we see capacity to refine the measures to include consideration of pathways to future possibilities. Finally, longitudinal associations do not demonstrate causality, they still need to be argued for. We think causal mechanisms are plausible, but as should be clear from the above discussion, the plausibility of causation varies by what we think the measures are actually assessing. Given that changing DD has been shown experimentally to change choices, we think a causal role is plausible for its underlying mechanisms, at least to be affecting smoking cessation outcomes.In conclusion, TP and DD are distinctly different concepts even though they both relate to effects of the future on choices and behaviours, in this case around smoking cessation. Our analyses support TP as being a measure of future thinking, but one that currently fails to differentiate possible futures from thinking about methods for pursuing them. In contrast, DD may be better thought of as indexing a capacity to evoke future possibilities, perhaps as a function of capacity to imagine them and to act in opposition to impulses to immediate gratification. That is, it is not so much about how often futures are considered but the ability to take them seriously when making choices that can have foreseeable future consequences.


The present study aimed to identify barriers and enablers for the implementation of artificial intelligence (AI) in dental, specifically radiographic, diagnostics. Semi-structured phone interviews with dentists and patients were conducted between the end of May and the end of June 2020 . A questionnaire developed along the Theoretical Domains Framework (TDF) and the Capabilities, Opportunities and Motivations influencing Behaviors model (COM-B) was used to guide interviews. Mayring’s content analysis was employed to point out barriers and enablers. We identified 36 barriers, conflicting themes or enablers, covering nine of the fourteen domains of the TDF and all three determinants of behavior (COM). Both stakeholders emphasized chances and hopes for AI. A range of enablers for implementing AI in dental diagnostics were identified . Barriers related to reliance on AI and responsibility for medical decisions, as well as the explainability of AI and the related option to de-bug AI applications, emerged. Decision-makers and industry may want to consider these aspects to foster implementation of AI in dentistry. The usage of artificial intelligence (AI) for assisting medical diagnostics has been exploding over the last three years , and espIn dental radiology, AI has been used for a range of purposes. Detection of dental caries ,15,16,17Notably, medical and dental AI is a complex technology which needs to be properly integrated and handled in the daily workflow. For instance, practitioners need to understand the concept behind most AI tools—supervised machine learning—and they have to be able to interpret AI findings as probabilistic. Aspects such as ethical, safe and explainable AI need to be considered. A range of studies deals with these topics, demonstrating risks of bias and criticizing AI algorithms as black boxes ,25, whicTo understand how all these different factors may act as barriers or enablers to implement AI in care, a systematic and comprehensive understanding of stakeholders’ expectations, needs and fears is essential. Qualitative studies can yield such insights but require a theoretical underpinning to ensure the required rigor and to yield the desired understanding ,30. MoreIn the present qualitative study, we conducted semi-structured qualitative interviews with dentists and patients and used the Theoretical Domains Framework (TDF)  to strucOne author  conducted the interviews. AM has a PhD with a background in biology, has been working in healthcare for eight years and has experience in the organization, conduction and analysis of qualitative studies. Together with a second author , she conceived the study. There was no relationship established with participants prior to the study. Interview dates were scheduled by a third assisting person. Interviews were carried out from the end of May to the end of June 2020.The present qualitative study builds on thirteen semi-structured individual interviews with dentists and patients. Interview guides were developed based on TDF . TDF facEight dentists (D1–D8) from eight different German dental practices and five patients (P1–P5) were interviewed. Dentists were identified non-systematically via Internet research while patients were randomly selected from the attending patient pool visiting the Department of Oral Diagnostics, Digital Health and Health Services Research, Charité–Universitätsmedizin Berlin, Germany, for a first assessment, triage or treatment. The dentists, two females and six males, were between 37 and 65 years old. The patients, one female and four males, were between 29 and 38 years old. One interviewer (AM) conducted all thirteen telephone interviews, with the dentists and patients being at home or, for dentists, at their practice.Interview guides covering the fourteen domains of the TDF were developed by two authors (AM and FS): (1) knowledge; (2) skills; (3) social/professional role and identity; (4) beliefs about capabilities; (5) optimism; (6) beliefs about consequences; (7) reinforcement; (8) intentions; (9) goals; (10) memory, attention and decision processes; (11) environmental context and resources; (12) social influences; (13) emotions; and (14) behavioral regulation. Each domain was addressed by a minimum of one question. Interview guides were pilot tested within the research group.Following written (e-mail) and verbal information about the study, the interviewees signed a declaration of consent. All interviews were conducted in German. Interviewees could freely discuss any topics directly or indirectly related to the question. Each interview took approximately 30 min and was recorded using a digital voice recorder. In addition to recording, notes were made. Transcripts were not returned to participants for comments or corrections.The interview was structured in two sections. In the first part, dentists and patients explained how they perceived radiographic diagnostics in dentistry. In the second, main part, the interviewer provided a brief overview about the current state-of-the-art of AI in dental diagnostics and described how an AI-based radiographic diagnostics assistance tool could be used in the daily workflow. From there on, interviewees could express their apprehensions, expectations and thoughts in a dialogue fashion, with the interviewer probing for an understanding of facilitators and barriers towards AI applications in dental practice.Interviews were transcribed and analyzed using f4 . The coding tree was derived according to the TDF domains. To increase inter- and intra-coder reliability, AM performed transcription and analysis independently and FS reviewed the results. An inductive and deductive content analysis was conducted using Mayring’s principles . The ideWe collected statements from the interviewees and coded them according to the TDF. The identified themes were then organized along the COM-B model. Relevant themes were found in nine domains of the TDF . Figure For dentists, we identified 10 themes as enablers, 5 as conflicting themes and 5 as barriers . For patients, we identified 10 enablers, 4 conflicting themes and 2 barriers : Obviously, the availability of digital radiographs is required to seamlessly use an AI-based diagnostic radiographic assistance tool. Six of the eight interviewed dentists had such digital radiographs available. Digitalization via scanning of analog films was found too time consuming.Time pressure  and legal requirements (Motivation–Reinforcement): Three dentists highlighted the legal need for a full systematic report, which is time consuming to generate and hence often abbreviated or written-up after the clinical work at the end of the day. AI applications may relieve dentists from this by automating the report generation and were expected to be time saving.D1: “On the other hand, it also makes work easier, right./So, I mean, diagnosing of an OPG (panoramic radiograph)/, if you do it properly, you can sit for a quarter of an hour/, if you then put it down in writing/.”D4: “If you could do that with one “click”, that would be good, right./because I know that from colleagues who studied together with me and who work as employed dentists/, there it is the rule/./and who then always groan that they have to sit down to write down these X-ray findings and diagnosis even after work.”D4: “Yes, because I have to say/, it is actually a requirement to write down the detailed X-ray diagnosis. We just do not do that/, not out of laziness/, we just write the file briefly: why we did something and what can be seen, right. But with an OPG (panoramic radiograph) in particular, a lot more has to be diagnosed and put down in writing, right. And maybe that (the AI-tool) would be a support, because I know that you actually have to do that. And I actually think it is stupid that we do not do that.”Improved accuracy (Motivation–Beliefs about Consequences): Three dentists reported on the difficulties of radiographic diagnostics and the associated uncertainty, especially towards caries diagnostics on bitewings. Improving and standardizing this accuracy was seen as a benefit.D1: “/so I am not claiming that I always see everything. I am only human too. As a human being, you overlook things, and such an incorruptible algorithm is sometimes very useful. There are situations/, a patient comes along and has problems/, and then you can interpret a little more in such a picture than there is. So also, not wrong for something like that.”D4: “Well that always happens/, I already had. I do bitewings/, you do not see anything, but in the mouth, I have a suspicion/, then I drilled into it and there was caries.”D5 : “For us that would be, so to speak, always in relation to the question: ‘Are there any additional findings?’/, because that is not the focus we are looking at.”D1: (Concerning AI) “/which of course means that there is a constant standard of diagnosis. And I think that makes perfect sense, right.”Responsibility : One dentist highlighted that it is in his responsibility to provide a sound and detailed diagnosis and found that AI could help him reach this goal.D5 : “/we also have dentists who send us patients just to take an X-ray because they do not have the appropriate equipment/, and of course then it would be smart, when the software could analyze the picture and you just have to look over it again/. That would certainly make work a bit easier.”Impact of AI : Two dentists were not sure about the impact of AI on their own diagnostic approaches and feared to be biased by it.D6: “The question is, will I be influenced in my decisions by the given suggestion. Do I want to control myself or does the program controls me?”Reliance on AI and responsibility : Along the same line, four dentists expressed apprehensions that staff or dentists may (possibly long-term) rely too much on the AI. They were unsure if this was a good thing, supporting and relieving them, or not.D3: “/that you can rely on it completely/, and only focusses on these marked areas/, and then maybe lose track of it yourself/somehow rest a bit on it (the program), saying: ‘Well, the program did not show me that and therefore I could not see it.’”D4: (concerning reliance on AI-diagnose) “Yes you should not do that. So that is the responsibility of every individual dentist”.D5: (concerning reliance on AI-diagnose) “/so the legal situation is clear/, and also the need to look at it (X-ray) again for yourself/, is also clear.”One dentist highlighted the importance of remaining responsible and critical towards AI and requiring time to doublecheck AI findings.D8: “Having time/, look up and thinking. Can there be something or is that a software error after all, or something that the software does not know.”Fears of malfunction (Motivation–Emotion): Two dentists commented that they were afraid of malfunction of the AI. One dentist indicated that it would be relevant to spot such errors.D4: “I would rather be afraid that the program might interpret something that is not there or something like that.”D8: “/the question is whether the program is able to diagnose overlays as overlays and not as caries/. This could be a problem.”D1: “I mean the question is: ‚What when you get false-positive results?’”D8: “/having time/, look up and think. Can it be or is it a software error after all/, or something that the software does not know.”Technological hurdles and fear : One dentist thought the technology may not be easy to maintain and compared the AI tool to his usual radiographic equipment.D2: “Documentation and maintenance and cost structure associated with it/. Beginning with the daily analysis of the monitor/, which probably nobody does/, but what is prescribed/, and so on, right. Until all the check-ups which are necessary are done/.”Moreover, another noted that patients may fear AI as part of the dental diagnosis.D7: “/when an apparatus, a machine, a computer program says: ‘That and that is going on, and that and that should be done, or that is the therapy recommendation.’ Then it is like in a science-fiction movie. And I mean, people are not ready for that yet.”I: “So, you mean the dentist can use it for its own support, but not in front of the patient.”D7: “Exactly.”Digital literacy (Capability–Knowledge): Patients with an engineering background, who understood the technology, showed high confidence into AI and its usefulness.P4: “I see everything that offers automated support in a relatively positive way. So, I also work with programs that somehow support you/, why should not you use that. I think support is always good.”Understanding the reasoning behind the AI  was also found an enabler.P2: “But if you are prepared for the fact that there can be mistakes/, and then asks again why he shows that/, and then maybe you can see for yourself why he is displaying it/, that understands. Maybe a program could write that too: ‘Because of this thing I marked it that way’. That would be ideal.”Social experience with AI : Four of five patients found any experience of their dentists or their family with AI to be enablers, strengthening their trust.P5: “If there is any suspicion and the dentist would like to check the X-ray also with the help of the AI, then I would follow the dentist recommendation, of course.”All interviewed patients were positive towards practices providing AI, seeing the potential for improving care. They also found that practices taking up this technology may be seen as more innovative and up to current standards.I: “Would you prefer a dental practice using the AI-tool?” P1: “Potentially yes, because it shows that the doctor is ready to innovate and up to date.”Costs and communication (Motivation–Reinforcement): Four of five patients stated that they would pay for an AI tool be used for their radiographic diagnostic independently from any health insurance. Four of five patients would also like to receive an AI-augmented, comprehensive medical report, as a “second opinion” .P1: “I had to request it (the X-ray pictures). Although my former practice also caused problems. They only printed the X-ray picture, what is not sufficient for a second opinion at another dentist. That was a pretty heck of a mess.”P4: “Yes of course. So that you can understand that better as a patient. That would be very helpful.”P5: “Exactly, with such a report you could go to another dentist and get a second opinion. This would be fantastic, right.”Improved accuracy (Motivation–Beliefs about Consequences): Four of five patients described negative experiences with dentists before, facilitating their hope that AI-based diagnostics would benefit them in the future by increasing diagnostic accuracy.I: “And then the dentist could not really identify where the problem is (on the X-ray picture)?”P2: “No. What you might have seen on the X-ray. But the pictures are sometimes not so easy to interpret.”I: “Has it already happened that you had complaints, and nothing could be determined?”P3: “Yes, that happens a lot/, there are also many mistakes. When they say: ‘You need two or three more root canal treatments’ or ‘here you have a problem in addition. You need a filling too.’/fifty percent (of the dentists) lie. That is just the way it is.”Responsibility for an adequate diagnosis: Three patients understood the technology behind AI and that AI can fail (Capability–Knowledge). They hence demanded dentists to not fully rely on the AI only.P1: “/Nowadays it doesn’t quite get across that the computer can also make mistakes. And that should be clear.”P2: “/it has to be clear for all persons that deep learning programs also like to make mistakes that a person would not make.”P4: “/you cannot rely on it one hundred percent because they are just machines. We (place of work from P4) still have an authority over it, a person who then counterchecks it.”Three of five patients indicated that it would be unsettling if staff or dentists rely too much on the software. One patient mentioned it would also be unsettling to not know if dentists spot possible malfunctions and are able to correct them .P2: “/it must be clearly stated (to dentists who use the AI-tool) that you always have to think again about what you are doing/, that you should not rely blindly on the program./and then it should be clear: ‘Here the computer is usually a little bit better than our dentists. The program marks it.’ But it should just be clear to everyone that there might be mistakes/, and do not take it as face value directly.”P1: “Of course, it can be that patients are unsettled (by the use of AI)./because things are potentially marked that may not need treatment after all or mistakes occur by the program. When mistakes happen, patients could say: ‘But why aren’t you doing anything now, everything is red. ‘To argue against the program is difficult then, because for the patient, the dentist gives a subjective and the software an objective opinion.”Validation and training (Capability–Knowledge): One patient mentioned that extensive regulatory validation of any AI and training of dentists and staff would be needed prior to it being used.P1: “When a practice test has taken place and the dentist is familiar with it, so to speak.”The presentation and juxtaposition of themes in Applications of AI are entering dental practice; providers and receivers of dental care will soon be confronted with AI. The aim of our study was to assess the attitude of dentists and patients towards AI for dental diagnostics, with a focus on radiography, as well as to identify barriers and enablers for implementing AI in clinical dentistry. A multi-perspective qualitative study was conducted and established theoretical instruments were employed for comprehensive and systematic analysis. In total, 36 enablers, barriers and conflicting themes were identified, covering nine of the domains of the TDF, and touching all three relevant behavior determinants, capability, opportunity and motivation. Overall, we found both stakeholder groups, dentists and patients, to be positive and open towards AI for dental diagnostics, noting the chances for a higher diagnostics quality, a relieved daily workload and a better communication and decision-making. Notably, dentists did not show any worries that AI applications could one day take over a significant amount of work and responsibility. This may well be as dentists’ tasks are heterogeneous, with only a minority allowing automation using AI, at least with the current state-of-the-art technologies. Some medical specialist such as radiologists, who do not directly communicate with patients (requiring empathy which machines do not have), do not contrast their findings against clinical signs and symptoms  and do not perform the subsequent therapy, are expected to undergo significant changes in their daily work induced by AI . Fewer cSeveral aspects were identified as enablers for implementing AI. First, and from a dentist’s perspective, the option to generate an automated diagnostic report after interaction with AI findings, thereby saving time for reporting, was a relevant enabler. This was associated with interviewees acknowledging this task as legally required but labor-intensive and automatable. Patients found the generation and receipt of such report desirable to obtain a full documentation of their oral health status, also for communication with other healthcare professionals. Second, both stakeholders expected AI to increase the diagnostic accuracy. Such expectations were more pronounced in patients, especially those with experience in invasive dental work. Important enabler for patients is that AI provides a second independent opinion and gives more confidence into the diagnosis of the dentist. Third, this trust into the AI was enhanced by the option to improve communication. Prior to the interviews, we demonstrated how current software frontends look and work. Patients and dentists further evaluated exemplary images of radiographs which had been assessed by AI models and which highlighted pathological findings on the radiograph in colors. For patients, this greatly increased their understanding of the radiograph and their trust into the subsequent diagnostic and therapeutic steps.All these enablers were closely related to a general technological openness. Individuals with technological expertise or background showed a more positive attitude and expected AI applications to mature with time. The identified enablers align well with those found for AI applications in radiology, for instance the work in , where tAs conflicting themes, we identified those around issues of reliance, responsibility and safety. Obviously, all parties were aware that AI applications will not be free of error potential and were interested to understand how practitioners would deal with this. If an erroneous AI finding could be spotted and corrected, this greatly increased the trust of patients. Our findings highlight the importance of explainable AI in the medical domain. In line with this, patients expected their dentists to maintain control and carry the final responsibility for any diagnostic finding, being able to de-bug or override the AI. This is notable, as the same was demanded by dentists, who found this responsibility sitting with them as part of how they perceive their professional role. Both sides also agreed that technological competency but also training with each specific AI tool was required to cover this requirement. If such training was not granted or an AI tool used which has not been sufficiently validated , these conflicting themes were acting as barriers. The relevance of understanding the basic principles, potentials and limits of AI to ensure quality and safety assessment has been found in other study, as has the importance of ensuring or scrutinizing AI results and reliability ,48,49,50A similar intermediate position was found for data protection aspects. If these were addressed sufficiently and in an accepted manner, data protection was not necessarily a barrier. If, however, dentists or patients perceived an AI application as not fully safeguarded against data protection breaches or legal conflicts, this was a clear barrier.Another barrier was related to the previous point. Both patients and dentists feared errors of the AI not being identifiable and leading to wrong therapeutic decisions. The importance of explainability has been confirmed previously . CertainGenerally, this study comes with a range of strengths and limitations. First, we assessed the attitudes and expectations of two major stakeholder groups, dentists and patients, towards dental AI diagnostics, a timely field with growing research and clinical relevance. Contrasting insights from different stakeholder is useful and yields a more comprehensive picture of possible barriers and enablers and also facilitates triangulation. Second, we employed the TDF and associated instruments, strengthening the foundation of this study and increasing our analytic comprehensiveness and rigor. The option to deduce interventions may be used by future studies or stakeholders such as policymakers or industry in the AI field. Third, and as a limitation, our sample was small and, as usual for qualitative studies, not representative. Despite sampling care providers and patients from different backgrounds and of different age or gender, this sample cannot claim representativeness. Our findings will not necessarily be generalizable. In addition, our interviews focused on AI for 2D radiography; we did not explicitly ask for 3D radiographs and hence cannot infer in detail on this topic. This was done as very few German dentists own or use a 3D radiographic device or have the license to operate one. Notably, dentists have expressed a high need for support especially for this type of imagery, and our findings may have differed if particularly inquiring users of 3D radiographic devices ,56. LastBoth stakeholders emphasized the chances and hopes for AI. A range of enablers for implementing AI in dental diagnostics were identified, e.g., the chance for higher diagnostic accuracy, a reduced workload, more comprehensive reporting and better patient–provider communication. Barriers related to reliance on AI and responsibility for medical decisions, as well as the explainability of AI and the related option to de-bug AI applications, emerged. Decision-makers and industry may want to consider these aspects to foster implementation of AI in dentistry.