Estimating the Reproducibility of Experimental Philosophy

After more than a decade of crisis replication in Psychological Science, different projects are being pursued to estimate the reproducibility rate of studies in various fields ranging from biology to medicine (surprisingly, sociology have been largely absent in that debate, though new effort had recently emerged, see Freese & Peterson, 2017). Even in economy, the tendency to replicate has started with only around half of the studies being replicated (see Chang & Li, 2015). This is concerning since even among scientists, only 52% considered there is a serious crisis (Baker, 2016) leading to wonder whether people differ on their definition of scientific crisis or if a non-trivial part of scientists and professors are just not following the dynamic of their respective fields.

In an effort to extend the replication project to other disciplines, Florian Cova and 20 research teams across 8 countries (which I had the pleasure of being part of), successfully replicated about 70% of 40 Experimental Philosophy (x-phi) studies ( The criteria to assess a successful replication were a significant replicated study (p < .05), subjective assessment of the replicating team (considering study designs, methodological details, 95% CIs, etc.) and comparison of the original and replicated effect sizes.

xphi replication



Is It a Dangerous World Out There? The Motivational Bases of American Gun Ownership

When trying to understand American gun policy, there is at least 2 sides of the story. One deals with the epidemiological data: is there a link between, for instance, gun availability and homicides/suicides? This type of analysis, with strong controls and replication, can eventually give the danger of having guns at home or gun shops around the corner.

But we usually omit the other part of the story: what motivates people to carry guns in the first place? Stroebe, Leander & Kruglanski (2017) recently published a paper in PSPB reporting a two pathways models (perceive threats/diffuse threats) pre- and post- the Orlando shooting (deadliest in US history, 49 dead and 58 wounded). The population-level data showed us that the level of danger of a State/City is not correlated with the number of people owning a guns. As a result, the need for a psychological model underlying the processes at play seem crucial to see which pathways are influencable and which ones are not.



Publication Bias & Heterogeneity

Among the myriad of goals that meta-analysis is prone to fulfill, estimating heterogeneity is a major one. Thanks to the Dutch team (Augusteijn, van Aert & van Assen, see, we have here a really good summarize of both classical heterogeneity tests as well as the effect of publication bias upon them. I’ll keep you in suspense for now but they basically showed that the effect of publication bias on the Q-test, for instance, is large, complex and non-linear (which would seems bad news as first glance). But what I like about this team is that they always propose tools trying to adresse the issue and this article was no exception.



Sexe, Hormones & Présidentielles

“Les récentes surprises électorales ont amplement mis en lumière l’imperfection des modèles censés prédire le comportement des électeurs. Le Brexit, l’élection de Donald Trump, les primaires françaises de gauche et de droite sont autant d’exemples que les experts et autres instituts de sondage ont mal anticipés. Même le modèle développé par le statisticien et journaliste américain Nate Silver, qui avait correctement prédit la victoire d’Obama en 2008, n’a pas su voir la victoire de Trump dans certains États-clés. Des problèmes statistiques liés aux modèles eux-mêmes compliquent la tâche : peut-on prévoir, en pondérant des centaines de sondages, les résultats (régression vers la moyenne) ou bien est-ce que les sondages disent seulement l’état du corps électoral à un instant t (marche aléatoire) ?”

Lire la suite :


Sous le joug du passé : Colonisation, Esclavage & Histoire Expérimentale

« La colonisation est plus que la domination d’un individu par un autre, d’un peuple par un autre ; c’est la domination d’une civilisation par une autre » dénonçait en son temps Léopold Sédar Senghor. Emmanuel Macron a, quant à lui, récemment qualifié, lors d’une visite à Alger, la colonisation de « crime contre l’humanité » provoquant les émois de la droite. Au-delà de ces postures morales, on est en droit de se poser la question de l’impact de la colonisation, et notamment de l’esclavage, sur le développement des pays africains. L’histoire expérimentale permet de comparer quantitativement, et à l’aide d’outils statistiques, différents pays qui se ressemblent sous beaucoup de points mais diffèrent quant à la variable étudiée (en l’occurrence, l’étendue de la traite négrière). Une étude de Nathan Nunn, Professeur d’Économie à l’Université de Harvard, révèle que l’importance de l’esclavage est directement corrélée à un faible développement économique issu d’une instabilité politique. Cette tradition de recherche appelle à une réflexion historique basée sur des inférences justifiées par des faits statistiques.



Publication Bias in Meta-Analysis: how to slay the Dragon?

McShane and his colleagues recently published a paper on publications bias and selection methods. Publications bias represent concerns one can have over the representativeness of a study or set of studies and has been subject of debates for centuries now (McShane even quoted Boyle, the Anglo-Irish chemist who was supposedly one of the first having shed a light on these concerns (I haven’t read the book but I let you check if this is true).

Publication bias not only raise the issues regarding the correct estimate of effect sizes, direction or statistical significance but also to accessibility, languages or familiarity. While it is generally viewed as a specific problem for meta-analysis, it is as much of a problem for a single study. If you generate an hypothesis based on a set of biased studies, you might very well end up not being able to reproduce any major findings and, a fortiori, finding a robust effect for your specific claim. This view generates questions about the nature of the hypothesis generation process, implying to get as much unpublished work possible before even starting to draw inferences and generate testable hypothesis.

null-results (more…)

Meta-analysis, p-curve, p-uniform… p-tastic!

If you open a Borenstein class on YouTube, you may end up on comments like this :

“A meta-analysis is NOT science.  It is meaningless garbage for people too lazy and too stupid to do their own randomized clinical study.”

Don’t get me wrong, this is more interesting that it seems at first glance. One of the goal of empirical science is actually to estimate effect size and so the first sentence does not make any sense of course. The second part of the second sentence comes from a hater that probably spent countless hours doing interesting and costly clinical studies that failed to reach significance (welcome to Science dude). But the first part of the second sentence is interesting as it contains the adjective “meaningless” and it is true that, recently, medical and social sciences are received severe critics from top experts (including the famous « Why Most Published Research Findings Are False ») concluding that results of many findings and thus of meta-analysis may be really difficult to interpret if not uninterpretable.