Science

Error message

  • Deprecated function: The each() function is deprecated. This message will be suppressed on further calls in _menu_load_objects() (line 579 of /var/www/drupal-7.x/includes/menu.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Notice: Trying to access array offset on value of type int in element_children() (line 6600 of /var/www/drupal-7.x/includes/common.inc).
  • Deprecated function: implode(): Passing glue string after array is deprecated. Swap the parameters in drupal_get_feeds() (line 394 of /var/www/drupal-7.x/includes/common.inc).

The Pandemic Timeline

Published by Anonymous (not verified) on Fri, 02/04/2021 - 7:00am in

Trump’s lies are like zombies. Fact-checkers keep killing them, but he keeps bringing them back to life — and repeating them over and over again. The only antidote is the truth — repeated over and over again. Steven Harper is following the pandemic for Moyers on Democracy. Continue reading

The post The Pandemic Timeline appeared first on BillMoyers.com.

A modest proposal for generating useful analyses of economies

Published by Anonymous (not verified) on Sun, 28/03/2021 - 11:35am in

[Published in Real World Economic Review #95, Davies, Geoff (2021) “A modest proposal for generating useful analyses of economies: a brief note.” real-world economics review, issue no. 95, 22 March, pp. 118-123, http://www.paecon.net/PAEReview/issue95/Davies95.pdf. Some other comments are at https://rwer.wordpress.com/comments-on-rwer-issue-no-95/ and https://rwer.wordpress.com/2021/03/25/a-modest-proposal-for-generating-u....

This is written for ‘heterodox’ economists, those who recognise mainstream (neoclassical) economics is nonsense, but who seem to flounder around not knowing what to do instead. It is a little more technical than my usual posts, but the message does not depend on the details.]

I propose that economists leave philosophy alone for a while and instead try analysing some actual economic observations.

I have observed much discussion among heterodox economists about what science comprises, whether one could do “scientific” economics, and what ontology, epistemology, etc, etc, might be involved. If, for example, economies are historically contingent, how could one hope to do a rigorous analysis. I have also observed much concern about the complications of people and societies and the resulting alleged need for elaborate statistical analyses to extract an object of interest, followed by the construction of an elaborate mathematical model that includes many nuances of human behaviour.

I think the challenge is not nearly so daunting. An economic analysis does not have to emulate the precision of (some) laboratory physics to be useful. It does not have to yield a literal prediction. If one steps out of the equilibrium mindset of the neoclassical mainstream one can find obvious phenomena crying out for explanation, a financial market crash for example.

It is not a great mystery how one might try to do some scientific economic analysis. Many kinds of scientist do science all the time, mostly without worrying about the philosophical nuances of precisely what kind of process they are engaged in.

The process I illustrate here is drawn from my experience studying Earth’s interior (Davies, 1999). It is a historical science. Earth’s processes are historically contingent and often very complicated. Observations of the interior are difficult, indirect and always incomplete. Yet we have developed considerable understanding of how the interior works to move continents and tectonic plates around the surface. Sometimes a very rough estimate can yield considerable insight.

What follows is expressed in terms that I think apply to many other kinds of science. Worrying about whether this process encompasses all kinds of science is precisely the sort of distraction I want to avoid. In avoiding the philosophy I do not mean to imply there is none involved, I simply want to get on with something that I know to work.

The process, in outline, is to seek some regularity or pattern or striking feature of an observable economy, and to propose a hypothesis that might account for the observed feature. The relationship of the hypothesis to the observation(s) ought to be explained. This might involve mathematics or it might not, and any mathematics used might be simple or sophisticated. Ideally some additional observations would be noted that are consistent with the implications of the hypothesis. One might then conclude by discussing whether the hypothesis appears to provide a useful description of the noted phenomenon and, if it does, how its usefulness might be further tested or enhanced. 

The currency of such enquiries is thus observations, hypotheses and useful resemblances. Nothing more obscure or contentious.

As an example of an observed phenomenon I offer the record of the US financial markets in the late 1980s (Figure 1). 

Figure 1. The Dow-Jones industrial average, 1986-1989

pastedGraphic.pdf

One can interpret this as a broad rising trend upon which is super-imposed a more rapid rise followed by a sudden drop. One might make other interpretations of a rather irregular plot, but that seems to be a useful description. 

This interpretation raises the question of what might cause the financial markets to rise unusually rapidly and then suddenly drop, i.e. the implied change of behaviour signals something that might be usefully investigated. An analogy can be made with population dynamics in ecosystems. A species, say rabbits, might find itself in a very favourable season, with a lot of food and few predators, and breed rapidly. If, however the rabbits eat the food faster than it can regenerate then they might find, rather suddenly, that there are lots of rabbits and hardly any food. Starvation might then ensue, with a sudden crash in rabbit numbers. Worse, they might have over-grazed the land and damaged its productive capacity.

This is known as an overshoot-and-crash phenomenon. It occurs because the feedback from food supply to breeding restraint is delayed. If there had been a wise old rabbit who persuaded the rabbits to slow their breeding as the carrying capacity of the land was approached then the population might have stabilised around a steady number. However rabbits are not so wise, and they keep breeding past the carrying capacity until the issue is forced by lack of food. By then it is too late to stabilise, and a disastrous crash is inevitable. 

So by analogy one might suppose that something triggers a burst of optimism among financial market traders and they borrow lots of money to “invest” (though really to bet on the market continuing to rise). This may continue until the amount of debt becomes more than can be reasonably repaid. There might then be a cascade of defaults that drive the market rapidly down. 

One can make a mathematical description of this process, in which the rate of growth of debt approaches zero as the “carrying capacity” of the economy is approached, but with a delay, so the amount of debt may overshoot the carrying capacity. How much overshoot will depend on the delay between the debt reaching the carrying capacity and the traders slowing their borrowing. It is assumed that during overshoot debt is extinguished, with a delay, by defaults at a rate that rises rapidly as the overshoot increases. Three solutions are shown in Figure 2, corresponding to delays of zero, 2 years and 3.5 years. The details of this model and further discussion of it are developed in my books Sack the Economists (Davies, 2013) and Economy, Society, Nature (Davies, 2019). 

Figure 2. Modelling debt as an overshoot and crash phenomenon

pastedGraphic_1.pdf

With no delay of feedback, the debt rises, exponentially at first, then smoothly levels off at the presumed carrying capacity. With a 2-year delay there is some overshoot, then quasi-periodic oscillating. With a 3.5-year delay there is a larger overshoot followed by a crash to very low levels of debt. The crash in the model is because debt is still being rapidly extinguished even after the level of debt has fallen back. It then takes a long time for the debt (and the economy that depends on it) to recover. 

This simple model seems to capture some of the essence of the market behaviour observed in Figure 1. It suggests that the core of the problem might be excessive optimism of traders and their resistance to warnings, combined with debt being too readily available. Clearly one could make more sophisticated models of such a process, but this simple model encourages further exploration of the main idea. 

A further encouragement is the solution with the 2-year delay, which yields behaviour reminiscent of the “business cycle”, too much money (and debt) alternating with not enough. The three kinds of behaviour illustrated in Figure 2 are obtained merely by changing one parameter, the delay. Thus in this sense the model has a generality extending beyond the immediate question addressed. 

This model is kept very simple in order to focus on the process of developing and using it, but you might think it is just too crude. It involves a single non-linear ordinary differential equation and a few plausible but debatable assumptions about parameters. The equation can readily be solved numerically. 

Yet consider a model of the Global Financial Crisis of 2007-8 by Eggertsson and Krugman (2012), the latter a pseudo-Nobel prize winner. They made two models, one for before and one for after a crash, with the difference between the models being effectively that the amount of available credit was presumed to be less in the second. Nothing in the model determined the amount of credit, it was imposed from the outside. Their equations of optimisation did require sophisticated, though old-fashioned, analytical methods to solve, but that says nothing about the usefulness of the models. 

Both models are equilibrium models. But if the “before” state of the market, with high prices, was an equilibrium state there would be no crash. Therefore the model must be missing the imbalance that drove the crash. It is therefore incapable of telling us why such a crash occurred. It cannot tell us anything about the dynamic process of boom and crash, the inflation and bursting of a debt bubble. It is not a useful model, it is a useless model, a dead end as far as understanding an observable economy is concerned. 

My simple model, on the other hand, can accommodate imbalances of the sort that must be involved in market crashes and the initial results are encouraging. It could be elaborated with more empirically-constrained input and, for example, a more explicit model of traders’ behaviour. In other words it has the potential to yield useful insight into a financial market crash. Steve Keen’s rather more elaborate models of finance are of the same general kind, so we have taken a small step in the direction of his instructive models (Keen, 2012). It may also turn out that this approach has limitations, when compared with more observations, and that other factors are important. That would still be a useful insight. 

Another virtue of my model is that it addresses a major and recurrent dysfunction of modern economies, their propensity for financial market crashes. The neoclassical tradition has been completely unable to address this rather fundamental issue, to the point that it was claimed no-one could have anticipated the Global Financial Crisis of 2007-8, though of course many outside the mainstream did. 

Let us look at what went into the models of Figure 2. First was a subjective interpretation of the graph in Figure 1, that the rise and fall during 1987 was a distinct episode with potentially identifiable causes. Then there was a model of one aspect of human behaviour: that financial traders tend to be over-optimistic and tend to ignore warning signs until they are very strong. This behaviour was then rendered into a rather simplified mathematical description with some plausible choices of parameters. These are all assumptions on which the model is based. Then deductive logic was invoked to deduce the implications of these assumptions – in other words the equation was solved, numerically. Then the results (Figure 2) were compared with observations – those of Figure 1 and also, qualitatively, with knowledge of “the business cycle”. 

This is a scientific process in operation: non-rational perceptions or interpretations of events or patterns in observations, the formulation of a hypothesis, deduction of implications of the hypothesis and further comparison with observations. 

Notice also that taking this small step in understanding did not require a comprehensive model of human behaviour and the many ways we interact. We do not have to be paralysed by the immense complication of human societies. 

Nor does one episode need to be identical to another for potentially useful insight to be found. An economy is a historically-contingent system, but we can still gain some understanding. The field of history would not exist if resemblances did not exist. You do not lie in bed despairing that today will not be just like yesterday, nor trying to model the coming day in detail. Rather, you get up and muddle through the new day as best your considerable understanding allows. 

My analysis did not yield a prediction of the future of the stock market. The event analysed is in the past, immutable, as the processes that formed a rock are in the past, yet we can draw insight from an analysis of how it might have come to pass. The solution of the equations expressing the assumed model allows us to deduce the implications of the model’s assumptions. It is more general and more useful to talk about implications rather than predictions.

By dealing with observations we avoid worrying about what “reality”, if any, might lie behind our perceptions. Observations will always have limited accuracy and will always be incomplete, and we have to contend with those limitations, but the existence of some observations is not in doubt.

The assumptions on which our model is built matter, contrary to the thoroughly muddled assertions of Milton Friedman (1953/1983). In fact assumptions are everything. Eggertsson and Krugman assumed equilibrium and thus emasculated their model at the beginning, making it incapable of addressing the issue. Scientifically it is a useless model, even though professionally it presumably further enhanced the reputations and power of its authors within their misguided profession. 

The model behind Figure 2 would be called a macro-economic model, in conventional mainstream terms. It is legitimate to address aggregate properties in this way because a complex self-organising system, as a modern economy plausibly is, has emergent behaviour that is not inherent in the behaviour of the system’s components (the traders and the objects of their trade). The behaviour emerges from the interactions among the traders and cannot be deduced, for example, from the behaviour of one representative trader. The well-developed science of fluid dynamics, for example, proceeds on the basis of the emergent macro behaviour of fluids, not by modelling all the atoms within the fluid. 

It is possible to do useful micro-economic models, but they must involve modelling the interactions of large numbers of traders and extracting their collective effects. Yoshinori Shiozawa and his colleagues are developing such an approach (Shiozawa et al., 2019). Other agent-based models, some associated with the Santa Fe Institute, have been reviewed by Eric Beinhocker (2006). 

This exercise did not involve statistics. It did not require elaborate massaging of data to extract the object of interest. That object is obvious enough in the raw data. Thus we need not be paralysed by the knowledge that there are many factors and variables at play in any given situation. There is a place for statistical analysis, but understanding of modern, disequilibrium economies is still so limited it is not hard to find rather obvious phenomena worth analysing. There is much low-hanging fruit to be harvested by those willing to venture into the field of far-from-equilibrium systems. 

That of course is a telling observation about a field that has been so misdirected it has failed to make substantial progress on basic questions (financial crashes, inequality, oligopoly, capture of markets and governments, destruction of our planetary life-support system, for example) after 150 years or more. 

One might reflect on the wisdom of humans relative to rabbits. We understand that rabbits do not have the foresight we do, and so they proceed to eat out their food supplies. We, on the other hand, do have foresight, and we have wise elders who have been warning us for decades, yet we persist with exactly the same over-exploitation of our planet. I think the problem is that in large societies our collective communication, verbal and non-verbal, is too limited and distorted for our foresight to regulate our baser impulses. Traditional societies living in smaller groups with mutual eye contact fare better in this regard – otherwise we would not be here. Until we reclaim and wisely use our means of communication in large societies we will remain collectively greedy and stupid. 

If at this point you feel the urge to further analyse the type of approach I used in the example presented above, please, resist. Do not tell us about the epistological, ontomagestic, negativist, empyreanist foundations of what I just did, nor even about the sadly disturbed state of my psyche. Instead, get out of bed, come outside into the messy world and join us in muddling along. Look for an interesting aspect of the observable behaviour of an economy and its society and either find or invent a possible explanation for it. Then tell us about it. We can then start to learn more about how modern economies and societies work. 

References 

Beinhocker, E.D. (2006) The Origin of Wealth. Boston: Harvard Business School Press. 

Davies, G. F. (1999) Dynamic Earth. Cambridge, UK: Cambridge University Press.

Davies, G. (2013) Sack the Economists. Canberra, ACT, Australia: BWM Books. 

http://betternaturebooks.net.au/my-books/sack-the- economists/ 

Davies, G. (2019) Economy, Society, Nature: An introduction to the new systems-based, life-friendly economics. Bristol, UK: World Economics Association, www.worldeconomicsassociation.org.  

Eggertsson, G.B. and P. Krugman (2012) “Debt, Deleveraging, and the Liquidity Trap: A Fisher-Minsky-Koo Approach.” Quarterly Journal of Economics, 127: 1469-1513  

Friedman, M. (1953/1984) “The methodology of positive economics” in B. Caldwell (ed) Appraisal and Criticism in Economics: A Book of Readings. Allen and Unwin: London. 

Keen, S. (2012) “A monetary Minsky model of the Great Moderation and the Great Recession.” Journal of Economic Behavior & Organization, 86: 221-235. 

Shiozawa, Y., M. Marioka, and K. Taniguchi (2019) Microfoundations of Evolutionary Economics. Springer.  

The Determinism of the Gaps

Published by Matthew Davidson on Sun, 31/03/2013 - 11:17am in

I'm surprised to find that it appears nobody's coined the phrase "the determinism of the gaps". So henceforth it's mine.

Just as the "God of the gaps" argument ("If [scientifically unexplained phenomena x] isn't the work of God, what else could it be?") is justly dismissed as an irrational argument from faith, so the scientistic claim that some imminent theory of everything will show that any observed phenomena is the result of the playing out of purely deterministic processes just because "What else is could it be?" should be dismissed for the same reason.

Unilaterally Raising the Scientific Standard

Published by Anonymous (not verified) on Sun, 03/02/2013 - 9:51pm in

For years, I and others have been arguing that the current system of publishing science is broken. Publishing and peer-reviewing work only after the study's been conducted and the data analysed allows bad practices - such as selective publication of desirable findings, and running multiple statistical tests to find positive results - to run rampant.

So I was extremely interested when I received an email from Jona Sassenhagen, of the University of Marburg, with subject line: Unilaterally raising the standard.

Sassenhagen explained that he's chose to pre-register a neuroscience study on a public database, the German Clinical Trials Register (DRKS).

His project, Alignment of Late Positive ERP Components to Linguistic Deviations ("P600"), is designed to use EEG to test whether the brain generates a distinct electrical response - the P600 - in response to seeing grammatical errors. The background here is that the P600 certainly exists, but people disagree on whether it's specific to language; Sassenhagen hopes to find out.

By publicly announcing the methods he'll use before collecting any data, Sassenhagen has, in my view, taken a brave and important step towards a better kind of science.

Already, most journals require trials of medical treatments to be publicly pre-registered, and the DRKS is one such registry. This study, however, is 'pure' neuroscience with nothing clinical about it, so it doesn't need to be registered - Sassenhagen just did it voluntarily.

Further, I should point out that he offered to pre-register his data analysis pipeline too by sending it to me. Unfortunately, I didn't reply to the email in time... but that was purely my fault.

I very much hope and expect that others will follow in his footsteps. Unilaterally adopting preregistration is one of the ways that I've argued reform could get started. As I said:

This would, at least at first, place these adopters at an objective disadvantage. However, by voluntarily accepting such a disadvantage, it might be hoped that such actors would gain acclaim as more trustworthy than non-adopters.

Pre-registration puts you at a disadvantage - insofar as it limits your ability to use bad practice to fish for positive results. It means you can't cheat, essentially, which is a handicap if everyone else can.

I don't know if this is the first time anyone's opted in to registering a pure neuroscience study, but it's certainly the first case I know of it being done for an entirely new experiment.

There have, however, recently been many pre-registered attempts to replicate previously published results e.g. the Reproducibility of Psychological Science; the 'Precognition' Replications; and an upcoming special issue of Frontiers in Cognition.

Replications are good, registered ones doubly so - but they're not enough to fix bad practice on their own. To do that we need to work on the source, original scientific research.

Is This How Memory Works?

Published by Anonymous (not verified) on Sun, 27/01/2013 - 8:46pm in

Tags 

papers, Science

We know quite a bit about how long-term memory is formed in the brain - it's all about strengthening of synaptic connections between neurons. But what about remembering something over the course of just a few seconds? Like how you (hopefully) still recall what that last sentence as about?

Short-term memory is formed and lost far too quickly for it to be explained by any (known) kind of synaptic plasticity. So how does it work? British mathematicians Samuel Johnson and colleagues say they have the answer: Robust Short-Term Memory without Synaptic Learning.

They write:

The mechanism, which we call Cluster Reverberation (CR), is very simple. If neurons in a group are more densely connected to each other than to the rest of the network, either because they form a module or because the network is significantly clustered, they will tend to retain the activity of the group: when they are all initially firing, they each continue to receive many action potentials and so go on firing.

The idea is that a neural network will naturally exhibit short-term memory - i.e. a pattern of electrical activity will tend to be maintained over time - so long as neurons are wired up in the form of clusters of cells mostly connected to their neighbours:


The cells within a cluster (or module) are all connected to each other, so once a module becomes active, it will stay active as the cells stimulate each other.

Why, you might ask, are the clusters necessary? Couldn't each individual cell have a memory - a tendency for its activity level to be 'sticky' over time, so that it kept firing even after it had stopped receiving input?

The authors say that even 'sticky' cells couldn't store memory effectively, because we know that the firing pattern of any individual cell is subject to a lot of random variation. If all of the cells were interconnected, this noise would quickly erase the signal. Clustering overcomes this problem.

But how could a neural clustering system develop in the first place? And how would the brain ensure that the clusters were 'useful' groups, rather than just being a bunch of different neurons doing entirely different things? Here's the clever bit:

If an initially homogeneous (i.e., neither modular nor clustered) area of brain tissue were repeatedly stimulated with different patterns... then synaptic plasticity mechanisms might be expected to alter the network structure in such a way that synapses within each of the imposed modules would all tend to become strengthened.

In other words, even if the brain started out life with a random pattern of connections, everyday experience (e.g. sensory input) could create a modular structure of just the right kind to allow short-term memory. Incidentally, such a 'modular' network would also be one of those famous small-world networks.

It strikes me as a very elegant model. But it is just a model, and neuroscience has a lot of those; as always, it awaits experimental proof.

One possible implication of this idea, it seems to me, is that short-term memory ought to be pretty conservative, in the sense that it could only store reactivations of existing neural circuits, rather than entirely new patterns of activity. Might it be possible to test that...?

ResearchBlogging.orgJohnson S, Marro J, and Torres JJ (2013). Robust Short-Term Memory without Synaptic Learning. PloS ONE, 8 (1) PMID: 23349664

Is Medical Science Really 86% True?

Published by Anonymous (not verified) on Fri, 25/01/2013 - 5:39am in

The idea that Most Published Research Findings Are False rocked the world of science when it was proposed in 2005. Since then, however, it's become widely accepted - at least with respect to many kinds of studies in biology, genetics, medicine and psychology.

Now, however, a new analysis from Jager and Leek says things are nowhere near as bad after all: only 14% of the medical literature is wrong, not half of it. Phew!

But is this conclusion... falsely positive?

I'm skeptical of this result for two separate reasons. First off, I have problems with the sample of the literature they used: it seems likely to contain only the 'best' results. This is because the authors:

  • only considered the creme-de-la-creme of top-ranked medical journals, which may be more reliable than others.
  • only looked at the Abstracts of the papers, which generally contain the best results in the paper.
  • only included the just over 5000 statistically significant p-values present in the 75,000 Abstracts published. Those papers that put their p-values up front might be more reliable than those that bury them deep in the Results.

In other words, even if it's true that only 14% of the results in these Abstracts were false, the proportion in the medical literature as a whole might be much higher.

Secondly, I have doubts about the statistics. Jager and Leek estimated the proportion of false positive p values, by assuming that true p-values tend to be low: not just below the arbitrary 0.05 cutoff, but well below it.

It turns out that p-values in these Abstracts strongly cluster around 0, and the conclusion is that most of them are real:

But this depends on the crucial assumption that false-positive p values are different from real ones, and equally likely to be anywhere from 0 to 0.05.

"if we consider only the P-­values that are less than 0.05, the P-­values for false positives must be distributed uniformly between 0 and 0.05."

The statement is true in theory - by definition, p values should behave in that way assuming the null hypothesis is true. In theory.

But... we have no way of knowing if it's true in practice. It might well not be.

For example, authors tend to put their best p-values in the Abstract. If they have several significant findings below 0.05, they'll likely put the lowest one up front. This works for both true and false positives: if you get p=0.01 and p=0.05, you'll probably highlight the 0.01. Therefore, false positive p values in Abstracts might cluster low, just like true positives.

Alternatively, false p's could also cluster the other way, just below 0.05. This is because running lots of independent comparisons is not the only way to generate false positives. You can also take almost-significant p's and fudge them downwards, for example by excluding 'outliers', or running slightly different statistical tests. You won't get p=0.06 down to p=0.001 by doing that, but you can get it down to p=0.04.

In this dataset, there's no evidence that p's just below 0.05 were more common. However, in many other sets of scientific papers, clear evidence of such "p hacking" has been found. That reinforces my suspicion that this is an especially 'good' sample.

Anyway, those are just two examples of why false p's might be unevenly distributed; there are plenty of others: 'there are more bad scientific practices in heaven and earth, Horatio, than are dreamt of in your model...'

In summary, although I think the idea of modelling the distribution of true and false findings, and using these models to estimate the proportions of each in a sample, is promising, I think a lot more work is needed before we can be confident in the results of the approach.

How (Not) To Fix Social Psychology

Published by Anonymous (not verified) on Fri, 18/01/2013 - 8:37pm in

British psychologist David Shanks has commented on the Diedrik Stapel affair and other recent scandals that have rocked the field of social psychology: Unconscious track to disciplinary train wreck,


Lots of people are chipping in on this debate for the first time at the moment, but peoples' initial reactions often fall prey to misunderstandings that can stand in the way of meaningful reform - misunderstandings that more considered analysis has exposed.

For example, Shanks writes:

[despite claims that] social psychology is no more prone to fraud than any other discipline, but outright fraud is not the major problem: the biggest concern is sloppy research practice, such as running several experiments and only reporting the ones that work.

It's true that fraud is not the major issue, as I and many others have said. But bad practice, such as p-value fishing, is in no way "sloppy" as Shanks says. Running multiple experiments to get a positive results is a sensible and effective strategy for getting positive results; that's why so many people do it. And so long as scientists are required to get such findings to get publications and grants, it will continue.

Behavior is the product of rewards and punishments, as a great psychologist said. We need to change the reinforcement schedule, not berate the rats for pressing the lever.

Earlier, Shanks writes that evidence of unconscious influences on human behaviour - a popular topic in Stapel's work and in social psychology generally -

is easily obtained because it usually rests on null results, namely finding that people's reports about (and hence awareness of) the causes of their behaviour fail to acknowledge the relevant cues. Null results are easily obtained if one's methods are poor.

Thus journals have in recent years published extraordinary reports of unconscious social influences on behaviour, including claims that people are more likely to take a cleansing wipe at the end of an experiment in which they are induced to recall an immoral act [etc]...

...failures to replicate the effects described above have been reported, though often papers reporting such failures are rejected out of hand by the journals that published the initial studies. I await with interest the outcome of efforts to replicate the recent claim that touching a teddy bear makes lonely people more sociable.

Here Shanks first says that null results can easily result from poorly-conducted experiments, and then criticizes journals for not publishing null results that represent failures to replicate prior claims! But null replications are very often rejected because a reviewer says, like Shanks, "This replication was just poorly-conducted, it doesn't count." Shanks (unconsciously no doubt) replicates the problem in his article.

So what to do? Again, it's a systemic problem. So long as we have peer-reviewed scientific journals, and the peer-review takes place after the data are collected, it will be open to reviewers to spike results they don't like - generally although not always null ones. If reviewers had to judge the quality of a study before they knew what it was going to find, as I've suggested, this problem would be solved.

Other people have great ideas for fixing science of their own. The problem is structural, not a failing on the part of individual scientists, and not limited to social psychology.

My Breakfast With "Scientism"

Published by Anonymous (not verified) on Mon, 17/12/2012 - 8:16pm in

One morning, I awoke convinced that science was the only source of knowledge. I had developed a case of spontaneous scientism.


The first challenge I faced was deciding what to eat for breakfast. Muesli, or cornflakes? Which would be the more scientific choice? I decided to go on the internet to look up the nutritional value of the different cereals, to see which one would be healthiest.

My computer was off. So first I'd need to turn it on - but how? From past experience, I suspected that pressing the big green power button on the front would do it - but then I remembered, that's merely anecdotal evidence. I needed scientific proof.

So I made a mental note to run a double-blind, randomized controlled trial of "turning my computer on" tomorrow.

Lacking nutritional data, I decided to pick a cereal by taste. I like muesli more than cornflakes. At last, a choice! Muesli it is, I thought - until I realized that I didn't actually know which one I preferred more. I had a gut feeling I liked muesli, but that's not science. What if, in fact, I hated muesli? Science couldn't tell me, at least not yet.

Another mental note: conduct cereal taste preference study, day after tomorrow. No breakfast for me, today.

By now, I was hungry, confused and annoyed. "This is getting ridiculous!", I tried to exclaim to no-one in particular - but then I realized - I could not even speak because I knew next to nothing scientific about the English language.

Sure, I had vague intuitions about how to put words together to express meaning, but that's just unscientific hearsay that I'd picked up as a child (no better than a religion, really!) In order to communicate, I'd need to study some proper science about semantics and grammar... but, oh no, how could I even read that literature?

Faced with the impossibility of doing anything whatsoever purely guided by science, I decided to go back to bed... yet with no scientific basis for controlling my own muscles, I collapsed where I stood, bashing my head on the breakfast table as I fell. 

Luckily, the bump on the noggin cured me of my strange obsession, and I lived to tell the tale.

---
Many people will tell you that "scientism", the belief that science is the only way to know anything, is a serious problem, a misunderstanding that threatens all kinds of nasty consequences.

It's not, because it doesn't exist - no-one believes that. If they did, they would end up like the unfortunate narrator in my story.

Everyday, we make use of many sources of information, from personal experience and learning to simply looking at things, whether they be right in front of your eyes or on TV. This is knowledge, and no-one thinks that we ought to replace it with "science", if that were even possible.

"Scientism" is a fundamentally unhelpful concept. Scientists are often wrong, and sometimes they're wrong about things that other non-scientists are right about. But each such case is different and must be judged on its own merits.

Pages