My undergraduate and graduate training in psychology and cognitive
neuroscience focused on computational modeling and behavioral experimentation:
implementing concrete models to test cognitive theories by simulation and
evaluating predictions from those models with behavioral experiments. During
this time, good ol’ t-test was enough statistics for me. I continued this sort
of work during my post-doctoral fellowship, but as I became more interested in
studying the time course of cognitive processing, I had to learn about
statistical modeling, specifically, growth curve analysis (multilevel
regression) for time series data. These two kinds of modeling –
computational/cognitive and statistical – are often conflated, but I believe
they are very different and serve complementary purposes in cognitive science
and cognitive neuroscience.

It will help to have some examples of what I mean when I say that
statistical and cognitive models are sometimes conflated. I have found that
computational modeling talks sometimes provoke a certain kind of skeptic to ask “With a
sufficient number of free parameters it is possible to fit any data set, so how
many parameters does your model have?” The first part of that question is true
in a strictly mathematical sense: for example, a Taylor series polynomial
can be used to approximate any function with arbitrary precision. But this is
not how cognitive modeling works. Cognitive models are meant to implement
theoretical principles, not arbitrary mathematical functions, and although they
always have some flexible parameters, these parameters are not “free” in the
way that the coefficients of a Taylor series are free.

On the other hand, when analyzing behavioral data, it can be
tempting to use a statistical model with parameters that map in some simple way
onto theoretical constructs. For example, assuming Weber’s Law holds (a
power law relationship between physical stimulus magnitude and perceived
intensity), one can collect data in some domain of interest, fit a power law
function, and compute the Weber constant for that domain. However, if you
happen to be studying a domain where Weber’s law does not quite hold, your
Weber constant will not be very informative.

In other words, statistical and computational models have different, complementary goals. The point of statistical models is to

*describe*or*quantify*the observed data. This is immensely useful because extracting key effects or patterns allows us to talk about large data sets in terms of a small number of “effects” or differences between conditions. Such descriptions are best when they focus on the data themselves and are independent of any particular theory – this allows researchers to evaluate any and all theories against the data. Statistical models need to worry about number of free parameters and this is captured by standard goodness-of-fit statistics such as AIC, BIC, and log-likelihood.
In contrast, cognitive models are meant to test a specific theory, so fidelity to the theory is more important than counting the number of parameters. Ideally, the cognitive model’s output can be compared directly to the observed behavioral data, using more or less the same model comparison techniques (R-squared, log-likelihood, etc.). However, because cognitive models are usually simplified, that kind of quantitative fit is not always possible
(or even advisable) and a qualitative comparison of model and behavioral data must suffice. This qualitative comparison critically depends on an accurate – and theory-neutral – description of the behavioral data, which is provided by the statistical model. (A nice summary of different methods of evaluating computational models against behavioral data is provided by Pitt et al.,2006).

Jim
Magnuson, J. Dixon, and I advocated this kind of two-pronged approach – using
statistical models to describe the data and computational models to evaluate
theories – when we adapted growth curve analysis to eye-tracking data (Mirman etal., 2008). Then, working with Eiling Yee and Sheila Blumstein, we used this approach to study phonological competition in spoken word recognition in aphasia (Mirman etal., 2011). To my mind, this is the optimal way to simultaneously maximize accurate description of behavioral data and theoretical impact of the research.

I had a professor in a modeling class describe the value of computational models as being concrete manifestations of a verbal theory. Creating the model forces you to confront a theory's critical assumptions and highlights what should be tested experimentally. Clear theories and pointed research questions seem like a good thing for science, so it's good to see that you and other researchers are making resources for developing models more accessible.

ReplyDeleteThanks! I agree 100%. Having a concrete manifestation makes both the assumptions and the predictions concrete, which is critical for a theory to be falsifiable.

ReplyDeleteHi Dan,

ReplyDeleteso, in order to evaluate the plausibility of some putative mechanism underlying the data, we could, in principle (cf. the caveat regarding quantitative fit of cognitive models), compare a statistical model with parameters that map onto theoretical constructs (e.g., a 24 h-sine wave if we hypothesize circadian modulation in a variable) with a theory-neutral statistical model (orthogonal polynomials) in terms of R-squared, etc right?

And if the fit of the theoretically informed model is not substantially worse than that of the theory-neutral model, would this support our hypotheses regarding the mechanism?

Best wishes,

Jakub

In general, I think it is not a good idea to compare theoretical models with statistical models because they are fundamentally different kinds of models. That said, sine functions are fairly well-behaved, especially if you're willing to make the period constant across individuals (e.g., 24 hours). In this case, you might be able to use sine-time the same way that we often use orthogonal polynomial time, and test whether circadian cycles modulate behavior and whether other factor influence that modulation.

ReplyDelete