A delta learning rule feeding the drift rate of a two-boundary Wiener
diffusion, so that the choice and the response time are one
likelihood rather than two. The learning half is
bandit2arm_delta()'s exactly:
Arguments
- subject
The column separating one learner's trial sequence from the next, given unquoted.
- trial
The column giving trial order within a subject, given unquoted.
NULLuses the order the rows appear in.- session
The column naming the session each trial belongs to, given unquoted.
NULL, the default, is one session per subject. Every subject's value store starts again from its initial values at the first trial of each of its sessions, so nothing learned in one session carries into the next. The subject stays the unit a random effect andfrm(importance =)group on: the sessions of one subject share that subject's effects. Trial numbers need to be unique only within a session. A label reused in two runs that are not adjacent in trial order is refused. See the Sessions section ofbandit2arm_delta().- max_ndt
Upper bound for the non-decision time.
NULL, the default, uses the fastest response in the data. A value above it is refused, because it admits parameters at which the fastest trial has no likelihood. It cannot be combined withndt_group(), which sets the same bound per group. It is also whatbf(ndt = )now needs: see below.
Details
and the choice half replaces the softmax with a first-passage density. Evidence accumulates at a rate proportional to how far apart the two value estimates have grown,
between boundaries bs apart, starting a fraction bias of the way
up, and the response time is ndt plus the time the accumulator
takes to touch a boundary. A trial's contribution is then the joint
density of WHICH boundary was touched and WHEN, which is what makes
this a model of response times rather than a choice model with a
timing footnote: a learning rate estimated from choices alone cannot
tell a subject who deliberated from one who guessed, and this one
can.
The model is Pedersen, Frank and Biele (2017). hBayesDM does not carry it; see the section below for what it does carry.
What it corresponds to in hBayesDM
Nothing, and that is worth stating plainly because every other family
in this package has a counterpart. hBayesDM ships choiceRT_ddm,
which is this family's choice rule with NO learning: one drift rate
per subject and no value store. It also ships bandit2arm_delta,
which is this family's learning rule with a softmax instead of a
diffusion. rlddm() is the two joined, and the join is the model
rather than a renaming of either.
Where the names do map, they map as hBayesDM's choiceRT_ddm spells
the diffusion: its alpha is the boundary separation, which is bs
here, its beta is the start point, which is bias, its delta is
the drift rate, which is drift times the value difference, and its
tau is the non-decision time, which is ndt. alpha here is the
LEARNING RATE, as it is in every other family in this package. That
collision is why the diffusion parameters carry frmtmb.eam's names
rather than hBayesDM's: one convention across this package's families
is worth more than an inherited one that would make alpha mean two
different things two families apart.
The data a trial carries
The response is the response TIME. The boundary reached travels in
dec(), coded 0 for the lower boundary and 1 for the upper, which is
the term and the coding frmtmb.eam::wiener() uses. Arm 1 is the
lower boundary and arm 2 the upper, so reward(pay1, pay2) is in the
same arm order as every other family here.
Holding the start point at a half
Most of the reinforcement-learning literature fixes bias at 0.5,
because with two ARMS rather than a correct and an error response
there is no reason for the accumulator to start nearer one boundary.
It is an ordinary distributional parameter here, so it is estimated
by default and held with bf()'s constant form:
frm(bf(rt | dec(choice) + reward(pay1, pay2) ~ 1 + (1 | id),
drift ~ 1, bs ~ 1, ndt ~ 1, bias = 0.5), ...)Estimating it is the more general model and costs one parameter;
fixing it is what makes drift and bs easier to separate on a
short session.
The non-decision time is bounded, not logged
The density is zero at and below ndt, so the likelihood has a hard
edge at the fastest response and a log link would let the optimizer
walk over it. ndt therefore gets a logit scaled onto (0, ub),
which makes the constraint structural. The bound comes from
frmtmb.eam::ndt_bound(), so this family and frmtmb.eam::wiener()
derive it the same way and refuse the same things.
One bound, or one per learner
ub is one number for the whole data set unless the model says
otherwise, and the whole data set's fastest response is the wrong
ceiling for a hierarchical fit. The information about one subject's
non-decision time is that SUBJECT's own fastest response, and with
100 learners the global minimum is the fastest of all of them, so a
subject deviation on ndt is a deviation on a fraction of somebody
else's floor. The scale tier measured what that costs: the 100 by 200
design reached a maximum gradient of 6.36e+09, a Hessian that was not
positive definite and four NaN standard errors.
ndt_group() is the fix, and it is opt-in:
frm(bf(rt | dec(choice) + reward(pay1, pay2) + ndt_group(id) ~
1 + (1 | p | id),
drift ~ 1 + (1 | p | id), bs ~ 1 + (1 | p | id),
ndt ~ 1 + (1 | p | id), bias = 0.5),
family = rlddm(subject = id, trial = trial), data = d)With it, each row's bound is its own group's fastest response, ndt
is a FRACTION of that bound on a plain logit, and the density
multiplies it back out. frm_linpred(dpar = "ndt", type = "response")
then reports the fraction, and frmtmb.eam::ndt_time() reports the
non-decision time in seconds whichever parameterization a fit is in.
Without it nothing about the family moved.
The grouping need not be the learner: it is whatever shares a floor.
max_ndt and ndt_group() together are refused, because they set
one bound to two different things.
What the per-learner spread of ndt does and does not say
Under ndt_group() a learner's non-decision time is a FRACTION times
that learner's own floor, so the fitted times differ between learners
even when the random effect on ndt is exactly zero, purely because
the floors differ. A spread across learners is therefore not evidence
that the variance component recovered anything.
Measured on the 100 by 200 design of dev/rlddm-findings.md, where
the drawn truths have a spread of 0.0393041: the full model returns
0.0333713 and the same model with the random effect on ndt
switched OFF returns 0.0380049. On this statistic the random
effect is not neutral, it is worse.
What the random effect IS worth on the same fit is the per-learner
accuracy, and there it is not close: 13.996 ms of error against the
drawn truths and a correlation of 0.9404, where no estimator without
it can beat 21.075 ms or 0.8478, because the correlation of any
constant multiple of the floors with the truth is fixed at 0.847780.
So read the per-learner error and the correlation, not the spread.
frmtmb.eam::wiener() carries the same warning for the same reason.
Why the ndt variance component is not what it looks like
The section above says to read the per-learner error rather than the spread. This one says WHY, in one line of algebra, because the reason is exact rather than empirical.
A learner's floor is its own non-decision time plus the fastest
DECISION it happened to make, floor = ndt + m. So the quantity
ndt ~ (1 | p | id) estimates under ndt_group() is
identically. The fitted deviation is not the non-decision time on a link. It is the log non-decision time MINUS the log of an observed minimum, and that second term depends on how fast the learner happened to decide, which is a property of their drift, their boundary and the luck of their trials.
Measured over 60 replicates at 100 learners by 200 trials
(dev/learnhier-findings.md), where the design drew
sd(log ndt) = 0.15:
| term | value |
sd(log ndt), the random effect the design drew | 0.1494 |
sd(log m), the observed-minimum term it did not | 0.4404 |
sd of the fitted-scale deviation | 0.4264 |
| variance share from the observed minimum | 107 percent |
So sd(ndt) on the link scale is about 94 percent a property of
the data rather than of the learners, and more trials do not repair
it: over 100, 200 and 400 trials the share reads 107.9, 106.7 and
108.2 percent, a paired difference of +0.002 with a 95 percent
interval of (-0.035, +0.040).
The same term explains the correlation. log(m) correlates 0.872
with a learner's own bs deviation, so the parameterization presses
cor(bs, ndt) toward -1: the fit returns -0.940 and the realized
value is -0.789, while the correlation the design actually drew comes
back exactly once the nuisance term is removed, cor(bs, log ndt) of
+0.2989 against a drawn 0.3.
What this does NOT affect. Every parameter with a population
truth on the scale it is fitted on recovers at that design: over 60
replicates the three intercepts, the three other standard deviations
and the three other correlations all cover between 0.900 and 0.983
against a nominal 0.95. And the per-learner non-decision times
themselves are unaffected, because frmtmb.eam::ndt_time() reports
them on the response's own scale and never passes through the
nuisance term.
What it would take to change it is a deviation defined against something other than an observed minimum, and the cost is that the bound stops being structural, which is the reason it is a scaled logit in the first place. That trade is not made here.
Pinning the non-decision time needs max_ndt
bf(ndt = 0.2) on a bare rlddm() is refused, and the message says
to pass max_ndt. bf() transforms a pinned constant at PARSE time,
before the response has been seen, so a family with no bound yet has
no scale to transform it on. Write the bound down instead:
frm(bf(rt | dec(choice) + reward(pay1, pay2) ~ 1,
drift ~ 1, bs ~ 1, ndt = 0.2, bias = 0.5),
family = rlddm(subject = id, trial = trial, max_ndt = 0.26),
data = d)This is a change, and it is one thing that stops working. Through
0.3.0 bf(ndt = 0.2) on a bare family was accepted and fitted 0.2
exactly, so far as the constant was below the fastest response. What
it did NOT do is check the constant against the bound the fit would
actually use: bf(ndt = 0.3) with max_ndt = 0.26 was accepted too,
and reached the objective as NaN, surfacing as an optimizer failure
that named nothing. Both cases are now settled before the formula is
parsed, which is how frmtmb.eam::wiener() and its siblings have
behaved since frmtmb.eam 0.7.0.
Recovery, measured
dev/learn-recovery-round2.R runs it and ?frmtmb.learn carries the
table. All five parameters recover at 30 subjects by 100 trials with
a random intercept on the learning rate: every bias is inside its
Monte Carlo error, and 60 of 60 replicates produced a usable
interval. Coverage is at the nominal rate for drift, bs, ndt
and bias, and 0.87 for alpha, so a learning rate's Wald interval
is slightly optimistic here and should be read as such.
THE PAIR THAT TRADES OFF IS NOT THE ONE TO EXPECT, and this section
said otherwise before the study was run. A first-passage density is
driven largely by the RATIO of the drift to the boundary, so drift
and bs look like the pair at risk; measured, their estimates
correlate at 0.17 across replicates, which is nothing. What does
co-vary is bs with ndt, at -0.57, and drift with bias, at
+0.56. Both make sense after the fact: the boundary separation and
the non-decision time are the two ways to make responses slower, and
the drift and the start point are the two ways to favor a boundary.
Read those two pairs together.
Drawing data needs RWiener, and fitting does not
frm_task_simulate() on this family draws a boundary and a time
JOINTLY from the diffusion, through
frmtmb.eam::ddm_simulate(), and that route is exact only when
RWiener is installed. Without it the draw is refused rather than
approximated: frmtmb.eam's own fallback draws the boundary first
and then rejects paths until one reaches it, which fails outright on
the rows where one boundary is strongly favored, and a simulator
that works on most rows and errors on the rest is worse than one
that says what it needs. Nothing about FITTING needs RWiener; the
likelihood is frmtmb.eam::wiener_lpdf() and is exact either way.
Where the density and the bound come from
frmtmb.eam::wiener_lpdf() and frmtmb.eam::ndt_bound(), through
the exports rather than through a colon. The Navarro and Fuss (2009)
series pair, its smooth blend and its tape safety belong to that
package, and so does the bound, its ndt_group() behavior and every
refusal either of them owes. This family writes the learning rule and
one multiplication.
References
Pedersen, M. L., Frank, M. J. and Biele, G. (2017). The drift diffusion model as the choice rule in reinforcement learning. Psychonomic Bulletin and Review 24, 1234-1251.
Navarro, D. J. and Fuss, I. G. (2009). Fast and accurate calculations for first-passage times in Wiener diffusion models. Journal of Mathematical Psychology 53, 222-230.
See also
bandit2arm_delta() for the same learning rule under a
softmax, frmtmb.eam::wiener() for the diffusion without learning,
frm_value_trace() for the per-trial drift rates.
Examples
d <- frm_task_design("bandit2arm", n_subject = 6, n_trial = 40,
seed = 1)
s <- frm_task_simulate(rlddm(subject = id, trial = trial), d,
pars = list(alpha = 0.4, drift = 3, bs = 1.6,
ndt = 0.2, bias = 0.5), seed = 1)
fit <- frmtmb::frm(
frmtmb::bf(rt | dec(choice) + reward(pay1, pay2) ~ 1,
drift ~ 1, bs ~ 1, ndt ~ 1, bias ~ 1),
family = rlddm(subject = id, trial = trial), data = s[[1]])
frmtmb::fixef(fit)
#> Estimate Est.Error Q2.5 Q97.5
#> alpha_Intercept -0.12988490 0.23329041 -0.5871257 0.3273559
#> drift_Intercept 3.00339160 0.24136432 2.5303262 3.4764570
#> bs_Intercept 0.40327886 0.04090687 0.3231029 0.4834549
#> ndt_Intercept 1.30354587 0.20156860 0.9084787 1.6986131
#> bias_Intercept 0.05694329 0.08646527 -0.1125255 0.2264121
head(frm_value_trace(fit))
#> subject trial q1 q2 drift_t pe dens
#> 1 1 1 0.0000000 0 0.000000 0.0000000 0.76659084
#> 2 1 2 0.0000000 0 0.000000 1.0000000 0.31815357
#> 3 1 3 0.4675743 0 -1.404309 0.5324257 0.04651865
#> 4 1 4 0.7165229 0 -2.151999 -0.7165229 2.11932154
#> 5 1 5 0.3814952 0 -1.145779 0.6185048 1.63379607
#> 6 1 6 0.6706922 0 -2.014351 -0.6706922 2.42811813