Keynote: Trends in Meta-Analysis

17 11 2011

Image of red starSpeaker: Tom Stanley, Hendrix College

Full title: Current & Future Trends in Meta-Analysis

Image of Tom Stanley

You can watch Prof. Stanley’s keynote now below.

Alongside this keynote lecture, we are also releasing a virtual issue of journal articles on the subject of meta-analysis. You can access the free papers on meta-analysis here. One more virtual issue will be published on Friday.

We are also very pleased to be publishing an invited commentary that responds to Prof. Stanley’s keynote, by Randall S. Rosenberger, Oregon State University, USA. Take a look at the commentary for an initial response to the keynote.

Please post your response to the keynote and commentaries in the comments box below; you can also respond directly to comments left by others by clicking on “Reply” underneath a previous comment. We very much encourage and welcome considered, substantive contributions to the discussion. Please note that all responses will be moderated.

Watch Now – click on the “play” button

On a mobile device, you may find that following this link instead works better:

Or Listen Now

If you are unable to use the video player, listen to the audio file by clicking on the audio player below. You can download the slides by following the link at the end of this post. On a mobile device, follow this link instead.

PDF iconTo read Prof. Stanley’s paper on Publication Selection Bias in
Minimum-Wage Research, click here

You can also download Prof. Stanley’s presentation slides by clicking here.



16 responses

17 11 2011
Bun Hold

I think, only at full employment minimum wage will have effect on employment.

17 11 2011
Aldona Zawojska

Prof. Stanley,
Let me quote the following from Your paper conclusions:
“In any case, with 64 studies containing approximately 1,500 estimates, we have reason to believe that if there is some adverse employment effect from minimum-wage raises, it must be of a small and policy-irrelevant magnitude”.
So, let’s assume (theoretically) that in some economies with relatively small inflation rate government increases minimum wage rate from £6 to £18 an hour (UK) or from PLN 9 to PLN 27 an hour (Poland). How would it truly result in magnitude of employment in those economies?

17 11 2011
Chris Doucouliagos

Hi Aldona. The minimum wage meta-analysis study looked only at the US. There is no reason to believe that the MRA results from the US can transfer to the UK, Poland or any other country. An entirely different meta-analysis would be needed for that.
Moreover, the findings apply only to the range of min wage levels (and increases) investigated by the 64 (US) studies. It would be rather bold to assume that tripling the minimum wage would have no effect on employment.

17 11 2011
Tom Stanley

Interesting point. As an economist, I too believe that large price increases will cause decreases in the quantity demanded—employment in this case. However, there were no large increases in minimum wages in the US over the time period covered by the existing research. Also, as David suggests, economic theory does not imply a reduction in employment if there is monopsony. In general, meta-analysis is revealing that behavior is much less sensitive to prices than we economists tend to believe.

17 11 2011

I always teach about monopsony and the role of minimum wages in correcting for it….

17 11 2011
Muhammad Anees


I do not find the link to run the video. Can I have it in email Please. It seems to be of very interest. Thanks.

17 11 2011

Muhammad, did you click ‘see more of this post’? If not, please do and then the the video player will appear on the screen. If you continue to have problems the video can be viewed at the following link:

17 11 2011
Today’s sessions – Thursday « Wiley Economics Online Conference

[…] Current & Future Trends in Meta-Analysis […]

17 11 2011
Margaret Giles

Heckman’s two stage correction for selection bias (where the selection equation estimates the probability of research being published and the substantive equation estimates the average value of the parameter of interest) would seem to be an appropriate technique to correct publication bias in meta analyses. If we did have access to unpublished studies (as they do now in health/medical studies which need to be registered prior to conduct), would we apply Heckman’s procedure? If not, why not?

17 11 2011
Muhammad Anees

The meta analysis is very interesting tool of research as it collects and analyse the published research on a topic of interest and then concludes from the findings of these research publications. I would suggest to have developed a tool that offers insights on the investigation of regional disparities in publications across the world. For example, I would be interested in identifying the differences of minimum wages and employment relationships for UK, US and Pakistan, so my meta analysis would be based on the publications from these countries and would develop comparative findings on the topic of research. I hope one could have such disparaties as it would be more beneficial to generalise the findings when taken care for the regional disparaties.

17 11 2011
Tom Stanley

Great question, Margaret. Yes, in theory, we could use a Heckman-type correction for publication selection. However, as you comment indicates, we cannot apply the conventional sample selection correction unless we have information on the unreported estimates. If we did have full information on the characteristics of the unreported estimates, their magnitude and SE, then, sure, run a two-step Heckman MRA.
I suspect that this option will never be available to economists, as it is in medical research, because we can never ensure that all results will be reported. Applied econometricians can always make a few runs on their models to explore the shape of the relationships without reporting what they learned by these explorations. And, no one else has any way of knowing whether unreported regressions were made or not. In medicine, researchers cannot get their expensive randomized trial published unless they had previously registered its existence.

The PEESE method to correct for publication bias uses the standard error squared as a proxy for the inverse Mills ratio term that would appear if we had the needed information in the second step of a Heckman-type regression, and simulations show that this works rather well. See, Stanley, T.D. and Doucouliagos, C.(H) (2011) ‘Meta-regression approximations to reduce publication selection bias’, School Working Paper, Economics Series 2011-4, Deakin University.

17 11 2011
Livingstone Senyonga

I actually believe that selection bias is real. The keynote paper gives very relevant examples of the origin of selection bias especially the urge to publish and nature. However, the source of selection bias are far more than stated even to include the origin of the publisher. Studies originating from from developing countries especially those suffering from information censorship aught to suffer much from this problem. Two notable dimensions are inclining results not to cause conflict with political and other authorities, urge to to produce results that attract interest of the economic community.
Can Hechman meta-analysis in any way capture this problem?

17 11 2011
Chris Doucouliagos

Thats a really interesting idea. One way to test this would be to try and code the country of origin of the author. The easiest way would be to look at the institutional affiliation. However, that might be inadequate. Going through author’s CVs would be time consuming but could give the type of information needed for this sort of analysis.

On the other hand, even in a totalitarian state, some research and some research questions might not be distorted. Research that comes closer to questioning the regime would obviously be vulnerable.

17 11 2011
The Wrap – Day 2 « Wiley Economics Online Conference

[…] Keynote – Stanley, Meta-Analysis […]

18 11 2011
Keynotes – audio « Wiley Economics Online Conference

[…] Keynote: Trends in Meta-Analysis, Tom Stanley […]

18 11 2011
The Wrap – Day 3 – thank you and goodbye! « Wiley Economics Online Conference

[…] If you had trouble viewing any of the videos, we’ve since added audio recordings (alongside the slides), which should be compatible with any computer. This is relevant to Prof. Hendry’s keynote, linked to below, and to Prof. Stanley’s keynote. […]

%d bloggers like this: