Thursday, October 6, 2016

New media and priorities

I was disappointed to read a (a draft of) a forthcoming APS Observer article by Susan Fiske in which she complains about how new media have allowed "unmoderated attacks" on individuals and their research programs. Other bloggers have written at some length about this (Andrew Gelman, Chris Chambers, Uri Simonsohn), I particularly recommend the longer and very thoughtful post by Tal Yarkoni. A few points have emerged as the most salient to me:

First, scientific criticism should be evaluated on its accuracy and constructiveness. Our goal should be accurate critiques that provide constructive ideas about how to do better. Efforts to improve the peer review process often focus on those factors, along with timeliness. As it happens, blogs are actually great for this: posts can be written quickly and immediately followed by comments that allow for back-and-forth so that any inaccuracies can be corrected and constructive ideas can emerge. Providing critiques in a polite way is a nice goal, but it is secondary. (Tal Yarkoni's post discusses this issue very well).

Second, APS is the publisher of Psychological Science, a journal that was once prominent and prestigious, but has gradually become a pop psychology punchline. Perhaps I should not have been surprised that they're publishing an unmoderated attack on new media.

Third, things have changed very rapidly (this is the main point of Andrew Gelman's post). When I was in graduate school (2000-2005), I don't remember hearing concerns about replication and standard operating procedures included lots of stuff that I would now consider "garden of forking paths"/"p-hacking". 2011 was a major turning point: Daryl Bem reported his evidence of ESP (side note: he was working on that since at least the mid-to-late 90's when I was undergrad at Cornell and heard him speak about it). At the time, the flaws in that paper were not at all clear. That was also the year a paper called “False-positive psychology” was published (in Psychological Science), which showed that “researcher degrees of freedom” (or "p-hacking") make actual false positive rates much higher than the nominal p < 0.05 values. The year after that, in 2012, Greg Francis's paper ("Too good to be true") came out showing that multi-experiment papers reporting consistent replications of small effect sizes are themselves very unlikely and may be reflecting selection bias, p-hacking, or other problems. 2012 also the year I was contacted by the Open Science Collaboration to contribute to their large-scale replication effort, which eventually led to a major report on the reproducibility of psychological research.

My point is that these issues, which are a huge deal now, were not very widely known even 5-6 years ago and almost nobody was talking about them 10 years ago. To put it another way, just about all tenured Psychology professors were trained before the term "p-hacking" even existed. So, maybe we should admit that all this rapid change can be a bit alarming and disorienting. But we're scientists, we're in the business of drawing conclusions from data, and the data clearly show that our old way of doing business has some flaws, so we should try to fix those flaws. Lots of good ideas are being implemented and tested -- transparency (sharing data and analysis code), post-publication peer review, new impact metrics for hiring/tenure/promotion that reward transparency and reproducibility. And many of those ideas came from those unmoderated new media discussions.

Thursday, September 15, 2016

Post-doctoral research position available

We are hiring a post-doctoral research fellow to start in 2017. Research in the lab focuses on spoken language processing and semantic memory in typical and atypical speakers. Current research projects investigate: (1) The processing and representation of semantic knowledge, particularly knowledge of object features and categories, and the events or situations in which they participate. (2) The organization of the spoken language system by mapping the relationships between stroke lesion location and behavioral deficits.

Research methods include:
  • behavioral and eye-tracking experiments
  • lesion-symptom mapping
  • computational modeling
  • non-invasive brain stimulation (tDCS)

  • Doctorate degree in Psychology, Cognitive & Brain Science, CSD/SHLS, or related discipline. Must be completed before starting post-doctoral fellowship.
  • Experience with one or more of the research methods and/or content domains.
  • Programming experience in R, Matlab, python, or similar language will be preferred.

The post-doctoral fellow will be expected to contribute to ongoing projects and to develop an independent line of research. Mentorship, training, and professional development opportunities will be provided to facilitate the fellow’s future career in academic, research, or industry settings.

LCDL has recently relocated to the Department of Psychology at the University of Alabama at Birmingham. UAB is a comprehensive, urban research university, ranked among the top 25 in funding from the NIH. Postdoctoral training at UAB is enhanced by the Office ofPostdoctoral Education. The medical school is routinely ranked among the top in the US, and interdisciplinary programs are a particular strength, including the Psychology Department’s undergraduate and graduate neuroscience programs. Birmingham is a growing, diverse, and progressive city located in the foothills of the Appalachians. It was recently rated #1 Next Hot Food City by Zagat, it is home to several world-class museums and performing arts venues, and the region offers excellent sites for hiking, camping, boating, swimming, and fishing.

To Apply, submit the following
  • A letter of interest that describes your training, research experience and interests, and career goals
  • CV
  • 2-3 letters of recommendation

Applications will be considered until the position is filled. For full consideration please apply by November 1, 2016. Only complete applications will be considered. Questions and applications can be addressed to LCDL Director Dan Mirman.

Tuesday, March 1, 2016


About 5 or 6 years ago my colleagues at Moss Rehabilitation Research Institute and I made public a large set of behavioral data from language and cognitive tasks performed by people with aphasia. Our goal was to facilitate larger-scale research on spoken language processing and how it is impaired following left hemisphere stroke. We are pleased to announce that we have completed a thorough redesign of Moss Aphasia Psycholinguistics Project Database site. The MAPPD 2.0 interface is much simpler and easier to use, geared toward letting users download the data they want and analyze it themselves. 

The core of this database is single-trial picture naming and word repetition data for over 300 participants (including 20 neurologically intact control participants) with detailed target word and response information. The database also contains basic demographic and clinical information for each participant with aphasia, as well as performance on a host of supplementary tests of speech perception, semantic cognition, short-term/working memory, and sentence comprehension. A more detailed description of the included tests, coding schemes, and usage suggestions is available in our original description of the database (Mirman et al., 2010) and in the site's documentation.

Friday, February 19, 2016

Acceptance and rejection rates

There was a recent blog post at Frontiers pointing out that journals' publicly-available rejection rates are not associated with their impact factors. Their post discusses several factors that contribute to this, but I've been thinking about how rejection rates are calculated, particularly publicly stated rejection rates. For example, the 2013 rejection rate for both JEP:LMC and JEP:HPP is 78% and JEP:General is slightly higher at 83%. These are top-tier experimental psychology journals and those rejection rates seem intuitively appropriate for selective outlets, but I think they might be inflated because many papers are rejected with an invitation to revise and resubmit.

Thursday, February 4, 2016

15th Neural Computation and Psychology Workshop

NCPW15 – August 8-9, 2016 – Philadelphia, PA, USA

Contemporary Neural Network Models:
Machine Learning, Artificial Intelligence, and Cognition

Thursday, September 3, 2015

Reproducibility project: A front row seat

A recent paper in Science reports the results of a large-scale effort to test reproducibility in psychological science. The results have caused much discussion (as well they should) in both general public and science forums. I thought I would offer my perspective as the lead author of one of the studies that was included in the reproducibility analysis. I had heard about the project even before being contacted to participate and one of the things that appealed to me about it was that they were trying to be unbiased in their selection of studies for replication: all papers published in three prominent journals in 2008.  Jim Magnuson and I had published a paper in one of those journals (Journal of Experimental Psychology: Learning, Memory, & Cognition) in 2008 (Mirman & Magnuson, 2008), so I figured I would hear from them sooner or later. 

Friday, June 19, 2015

Zeno's paradox of teaching

I've wrapped up my Spring term teaching and received my teaching evals. Now that I've (finally) had a chance to teach the same class a few times, I am starting to believe in what I call Zeno's Paradox of Teaching: every time I teach a class, my improvement in teaching quality is half the distance between quality of the last time I taught it and my maximum ability to teach that material.
If I'm right about this, then I think it means that it's important to think long-term when approaching teaching:
  1. New faculty (like me) should start by teaching primarily core courses, ones that are offered every year, have good support materials, and provide a consistent opportunity for improvement. Specialized seminars can be fun to teach, but if they're not going to be offered every year, then improvement will be slow.
  2. Don't drive yourself (myself) crazy trying to teach the "perfect" class on your (my) first time teaching. Try to do a good job and next time try to improve on it as much as possible.
  3. Zeno's paradox means that I'll never teach quite as well as I think I could teach. The positive message there is that one should continue trying to come up with creative ways to improve a course. The warning there is that perfection is not an appropriate standard and not to be too hard on oneself for failing to reach it.