Posts
Women are grumpy during their period, and they have good reasons to be so. That this can impact some of their decisions should come as no surprise, yet it can be useful to determine how and how much.
Matthew Pearson and Burkhard Schipper do this by running an experiment that tries to tease out risky behavior and find that women bid higher in an auction when in the most fecund phase of their menstrual cycle or when they are on hormonal contraceptives. OK.
But wait, much like in an infomercial on TV, there is a bonus. In a second paper, the same authors find that the ratio of the length of the index and ring fingers of the right hand has no impact on risk taking. While that seems to be a rather odd measure to look at, there is a good reason to do so. But what annoys me that this is the exact same experiment as in the previous paper, they just use a different characteristic of the participants.
This is a bad case of turning a research project into many thin salami slices. The authors did not even bother rewriting much of the paper, with many part being cut-and-pasted from one to the other. Sadly, this second paper is already scheduled to appear in Experimental Economics. What are we to expect next? A paper about hair color? Astrological sign?
Friday, December 9, 2011
Subscribe to:
Post Comments (Atom)
10 comments:
From Burkhard Schipper, as requested by email:
1. When different measures are taken in an experiment, you should also report on those different measures and not just on measures that come out "significant". We mention all measures in both papers, but in the menstrual cycle paper we use the digit ratio as a control only in some tests. Both papers reference each other properly.
2. We decided to write a separate paper on the digit ratio BECAUSE
didn't have significant finding. There is a large literature with positive findings on the digit ratio. Usually researchers don't even bother to write insignificant findings and a publication bias can evolve easily. If we had reported the insignificant finding on the digit ratio somewhere in the menstrual cycle paper only, everybody could easily ignore it.
3. There are good reasons to look at the digit ratio and at the menstrual cycle. The digit ratio is measure of PRENATAL exposure to testosterone and estrogen. In contrast, the menstrual cycle is correlated with endogenous circulating levels of progesterone, estradiol, LH, and FSH.
4. Naturally, the next study is to take direct measures of circulating
levels of hormones. This is done in
[pdf] . We should get a comprehensive picture of endocrinological factors correlated with competitive bidding, and not just fish for some cute little results.
Since the papers use the same design and have a similar motivation, the description of the design etc. should be the same. After all we don't write novels but scientific reports.
On another point, the papers are not on "risk taking" in the narrow
sense. It is far from clear what is the connection to risk aversion.
In reply to Burkhard Schipper:
1. I agree negative results merit publication. I appreciate that you do not play games like Bruno Frey and the papers cite each other. But this should really have been a single paper.
2. One paper would still have been sufficient, especially given the amount of duplication between the two.
3. I am not doubting that.
4. If this were done from the same experiment as the previous two papers, I would have said again that this should all belong in the same paper.
I think that the issue of replication is an under-discussed one in experimental economics. Personally, I would like to see grad students running replication experiments and having an outlet to publicise them.
But there are many barriers preventing this, both cultural and physical, so I do not hold much hope that it will become the norm anytime soon.
To Economic Logician:
1. Why should have it been a single paper? You don't tell your reasons.
Again we had good reasons to separate:
a) We did not want the null result being ignored.
b) The studies have slightly different motivation. The menstruation paper is a replication study of Chen, Katuscak, and Ozdenoren (2009).
c) You can not present a detailed analysis of both measures within 35 pages.
In an ideal world of science, where people don't turn a blind eye on results that they don't like, where editors of journals allow you to present detailed analysis, I would have preferred to present both into one paper.
2. The duplication is not in results. It is in the exposition of the design and to some extent of the related literature. There is nothing wrong with that.
4. It is more complicated. The 2009 version of the menstrual cycle paper and digit ratio paper had roughly only 200 subjects. In 2010, I run an additional experiment with again roughly 200 subjects in which I took saliva and did a risk task. With respect to the menstrual cycle data and the digit ratio data, we now report everything in the 2011 versions of the menstrual cycle paper and digit ratio paper, respectively. There is no way to explain the salivary hormone methodology and analysis within the same paper. The new salivary hormone paper takes already almost 60 pages in its own.
I guess the implications of what you are saying is that we should focus only on little cute studies that can be reported within 35 pages, or be silent on many details of the experiments. I disagree with any of that for the following reasons. This type of research is extremely speculative. Essentially, it is biological data mining with all its drawbacks. The more measures you take, the higher is the chance that some measure comes up "significant". To really evaluate any findings or nonfindings, you need to know the details of the experiment, the quality of the measures, and the robustness checks performed. I believe that in our papers we are extremely careful in dealing with that and reporting it to the public.
I have never claimed a paper should be contained within 35 pages. In fact, page limits are rather obsolete in the age of online availability of research. And experimental economics papers should be longer as the detailed instructions given to participants and screen shoots need to be part of the paper's appendix.
What I object to is the duplication of papers with slightly different questions that could have been dealt in a single paper, thus avoiding all this copy-and-pasting. You are reporting about one single experiment in several papers.
What you call "all this copy-and-pasting" is more or less providing a detailed exposition of the experimental design in both papers. There is nothing wrong with it.
I don't think there is anything wrong in analyzing a rich data set in different ways. If a data set is rich, it may require several reports to do this properly. What is important is that this is made very clear in all reports. And we are very clear about it.
I explained above in detail why we could not have dealt with all measures in a single paper, decided to split the reporting but made sure that readers know about it. Partly our reasons are based on publishing constraints and our insistence on careful reporting as nobody likes to take papers that are longer than 35 pages. But page constraints may have good reasons too. Even you did not seem to have looked at our papers in detail which did not prevent you from making strong claims in the public behind your convenient veil of anonymity.
I can not help myself to say that I am very upset about you labeling it "bad research" because it is actually extremely careful research. This kind of research is intellectually not very challenging. The only reason why it is meaningful to do for me is because I strongly believe it should be done very carefully. That's exactly what you are questioning. You are very quick in stigmatizing it as "bad research". I feel this is very unfair.
From your dismissive concluding comments "What are we to expect next? A paper about hair color? Astrological sign?" I guess your motivation for your blog entry is based on your belief that research on hormones and economic behavior is pretty meaningless anyway and you did not see a point in having many related papers on it. I accept that people have different tastes. But to me, trying to gain an understanding how endocrinological factors are correlated with economic behavior seems worthwhile. While we may not learn much from it about economics like GDP or inflation, we may learn something about human nature.
I have never criticized a lack of thoroughness on your part, and it would not be justified. What I am criticizing is the slicing of one experiment into several papers.
Why is it quasi a dogma to you that every data should be analyzed in at most one paper?
If a data set is rich, that is, there are lots of variables, and the interpretation of the results require qualifications (for instance because the results are difficult to interpret without knowledge of the data collection procedures and the analyses performed), then it will be extremely hard to satisfy your dogma. It would imply that you can write at most one paper with US census data, at most one paper with German SOEP data etc. It sounds absurd. It sounds may be less absurd when we think about lab experiments because we typically expect those data sets to be "small". But are 12'000 bids and about 100 variables really small?
EL is right and Schipper is wrong.
Obviously, Schipper simply wants to maximize the number of his publications.
It is easy to have a large publication list if substantial parts of the papers are identical using copy-and-paste.
> EL is right and Schipper is
> wrong.
We thank the competent judge for his/her wise judgement.
> Obviously, Schipper simply wants > to maximize the number of his
> publications.
Then why not write more papers on the rich data set? For instance, there is a robust "very significant" correlation between a measure of optimism and the digit ratio in our data. Given the large literature on positive findings for the digit ratio, I guess writing it up would yield some good odds for another publication. I don't write it up because I fix my measures of interest before the experiment and before knowing the data (and use those other measures just as controls in my analysis).
> It is easy to have a large
> publication list if substantial > parts of the papers are
> identical using copy-and-paste.
No, it would be easier if you don't use the same description of the experimental design in both papers but would change slightly the wording and would relegate the fact that the data come from the same experiment to at most a footnote.
Post a Comment