BrianMartin
Go to
BrianMartin's publications on education
BrianMartin's publications on science, technology andsociety
Abstract
Researchers often complain aboutresearch grant schemes, but usually within a narrow frame ofreference. Looking more broadly, problems with grant schemes can beclassified as bias, waste, discouragement and orientation tointerests. There are various ways to allocate research funds,including administrative decision, peer review, performance-basedallocation, equality and community-based bids. Each has differentsorts of problems and serves different interests. By looking atdiverse systems for allocating research funds, some of theassumptions underlying usual discussions become more apparent. Recentchanges in Australian government policy on higher education researchare examined using the framework provided here.
Introduction
Any research grants scheme islikely to generate a great deal of informal complaint. Notsurprisingly, dissatisfaction is most common among unsuccessfulapplicants, with complaints about bias and wasted effort.Administrators worry about making the system work efficiently.Reformers seek methods of making better decisions, for example bychanging selection criteria and peer review systems, and ensuringaccountability.
However, most of the discussionsabout grant schemes deal only with minor changes within the samebasic structure. Yet the structure of a scheme is often the primarydeterminant of the sorts of problems that it generates. This articleaims to provide a broader perspective on this issue. I start byoutlining several key types of problems with grant schemes and thenlay out several types of methods for decision making. With thisfoundation, it is straightforward to note the sorts of problems mostcommonly or likely to be associated with decision-making systems.Finally, recent changes in Australian government policy on highereducation research are assessed using the broad framework offeredhere. (Some of the sources cited below deal with peer review forjournals, but their assessments are likely to apply to peer reviewfor grants as well.)
Problems
What constitutes a problem with agrant scheme of course depends on the observer. Hence the focus needsto be on perceived problems, acknowledging that widespread perceptionof a problem may be a problem in itself, whatever the 'facts'. Here, several different types of problem areoutlined.
1. Bias
Among applicants, especiallyunsuccessful ones, allegations of bias are commonplace. They includethe following.
- Success-breeds-success bias: successful applicants are likely to become entrenched, using their grants to produce the outputs necessary to attract further funds, while others never have the chance to get started.
- Insider bias: decisions are made by cliques of insiders, who think highly of and award most grants to themselves and a small group of favourites.
- Dominant group bias: there is discrimination against groups such as women, ethnic minorities and lower-status institutions (Peters and Ceci 1982; Wenneras and Wold 1997).
- Conventional approach bias: grants are much more likely to support tried-and-true approaches, while challenging, innovative or unorthodox proposals are seldom funded (Armstrong 1996, 1997; Epstein 1990; Horrobin 1990, 1996).
- Personal bias: administrators or referees obstruct researchers or projects that they do not like (Horrobin 1974).
To these possibilities can beadded 'random bias' due to incompetence of administratorsor referees.
2.Waste
Any grant scheme requiresadministrative overheads to assess applications. In addition,applicants may spend a large amount of time preparing applications.Even in an efficiently run operation, the cost associated withrunning the scheme and preparing applications can be a significantproportion of the money awarded in grants, especially if the grantsare small and the success rate is low. In addition, not all grantschemes are run efficiently, aggravating the problem ofwaste.
3.Discouragement
Most of the attention in grantschemes is on those who are successful, but there can be significanteffects on those who are unsuccessful. They can be disheartened byrejections or become resentful. This provides fertile soil forbeliefs that decisions are biased, which help to alleviate the stigmaof failure. Any competitive system creates this problem, whether itis grading in schools or promotion systems at work. There isconsiderable evidence that competition reduces intrinsic motivation(Kohn 1993). Even for more successful applicants, the goal of winninga grant may become more significant than doing theresearch.
4. Orientation tointerests
The possibility of getting a grantprovides an incentive to do research that pleases those dispensingthe money. Grants provided by a corporation or government departmentfor research in particular fields - telecommunications or pesticides- obviously orient researchers to particular problems. This is notnecessarily a problem in itself, but can be seen to be one in thewider context of social priorities. In short, research is oriented tothose who have money to dispense. Social problems which no one has avested interest in solving receive few resources (Arditti et al.1980; Dickson 1984; Martin 1979).
Many government grant schemes areoriented not to problems but to acquisition of knowledge. Grants goto applicants who best make the case that they are pushing back thefrontiers in astronomy or brain structure. Even in these cases, itcan be argued that there is an indirect orientation to outsideinterests. For example, problems in numerical analysis oroceanography may be influenced by military priorities; problems inorganic chemistry or electrical engineering may be influenced bycorporate priorities. This influence can occur through paradigms,potential applications of pure research, or job prospects.
Finally, researchers have a vestedinterest in their own careers, including positions and status. In asmuch as top researchers are influential in making decisions aboutgrants, it is likely that the system of research - based on full-timeprofessional specialists - will be perpetuated. This may be at theexpense of other priorities, such as pressing social problems. Thefamiliar example of researchers recommending further research issymptomatic of the problem that the grant system is oriented to theinterests of researchers at the expense of others.
Methods
Researchers are so familiar withpeer review - not least through refereed journals - that it may seemthat there is no sensible alternative. But actually there are variouspossibilities. The following are five possibilities, chosen becausethey have the potential to deal effectively with one or more of theproblems noted above. They are presented as ideal types. In practice,actual allocation of research funds typically combines elements ofseveral of these methods.
Method A: administrativedecision
In this ideal system, alldecisions about research priorities and funding are made by topadministrators, who may or may not be researchers themselves. Inmaking their decisions, the administrators take advice from a rangeof groups: political and economic leaders, researchers, lobbyists andso forth. The approach is typical of research in large bureaucracies,especially government (in particular the military) and corporations.The justification for this approach is service to national, public orshareholder interest, which are often taken as synonymous. Theresearch system emphasises cooperation and teamwork, the exemplarbeing the Manhattan project for producing the first nuclearweapons.
The patterns of research in theworld today reflect high-level administrative decisions. This appliesnot just to most government and corporate research but also to muchuniversity research, since administrators make most of the decisionsto decide the distribution of funds between different fields, provideresearch infrastructure, and to set up specific research centres andprogrammes. For example, the framework for deciding the allocation ofresearch funding between telecommunications engineering andmusicology within a university is normally decided by managers,rather than by peer comparisons, equal distribution or public opinionpoll. There is, of course, no objective method to decide such anallocation.
Researchers and other pressuregroups do have some influence on such administrative decisions. Forexample, researchers and consumer groups have some influence on theresearch agendas of electricity authorities and electronics firms.Although administrative decision is widely used, it has receivedrelatively little critical attention compared to peer review; thereare only a few researchers who enthusiastically advocate it(McCutchen 1977).
Method B: peerreview
In this model, decisions aboutresearch priorities and funding are made by peers, namelyprofessional researchers knowledgeable in relevant areas, in acompetitive merit-based system. The justification for peer review isthat peers are best able to judge the merit of research and thathigh-quality research is best able to advance knowledge and serve thepublic interest.
Peer review can be implemented invarious ways. Applications can be sent to anonymous referees who ratethem. Alternatively, applications might all be judged by a smallpanel without any outside comment. As long the panel members areknowledgeable in the field, this is still judgement by peers, thoughmore open to accusations of bias. Some key elements in peer reviewsystems are:
- independence or otherwise of referees;
- anonymity of referees versus open peer review;
- criteria for decision making;
- method of decision making.
There is a large literature onpeer review, and much discussion (Armstrong 1997; Campanario 1995;Chubin and Hackett 1990; Daniel 1993; Peters and Ceci 1982; Wessely1998). However, for the purposes here the various options in peerreview systems are secondary to the main point that ranking ofcompeting applications is primarily based on judgements bypeers.
After administrative decision,peer review plays a significant role in setting research prioritiesand making funding decisions. This operates in grants schemes and indecisions at the research project level within universities and, to alesser extent, in some government and corporate research units. Asnoted above, administrative decisions play a central role in settingthe framework for many peer-review decisions.
Method C: performance-basedfunding
This method funds researchersaccording to the outputs that they have produced, such aspublications and patents (Forsdyke 1993; Roy 1984). By defining thedesired sorts of performance and defining the rewards,performance-based funding can be implemented without formal grantapplications at all. For example, each paper in a specified journal(perhaps weighted by the journal's impact factor) could resultin a defined payment to the authors. The research quantum allocatedto Australian universities relies on a formula including weights forpublications, research degree completions and funds received. This issimilar to Roy's (1984) proposal for allocation of funds togroups of researchers based on group productivity. Obviously, peerjudgements influence funding via the outputs, but performance-basedfunding differs from typical peer review systems in being basedstrictly on outputs rather than grant applications. One justificationfor performance-based allocation is to avoid direct biases based onreputation, gender or research findings.
Offering prizes for discoveriescan be considered to be another type of performance-based funding.Nobel prize winners receive not only money but fame which often canbe translated into further resources.
Method D:equality
In this approach, every researchergets either an equal amount of funding or an equal chance at funding.Some minimum requirements can be put on who is eligible, such as allstaff at a university or in a research group. Among those who areeligible, available funding can be divided up in any of severalways.
- Each year, available funds are equally divided between researchers.
- Researchers take turns receiving significant grants.
- Grants are awarded using a lottery.
A considerable proportion ofresearch funds are currently allocated using this method, at leastnominally. If university staff are expected to devote one third oftheir time to research, then one third of the budget for salariescould be said to be allocated using the method of rough equality(though since salaries are different, the nominal allocation is notequal). For some humanities research, for example, time and access togood libraries are the most crucial elements, and additional researchfunding is less significant. On the other hand, in many technicalfields, salaries are only the beginning of what is required toundertake research.
Method E: community-basedallocation
In this model, research prioritiesand funding are decided by a range of community groups, such asgroups of workers, parents or neighbours. The key feature is thatusers at the grassroots would make the decisions, rather thanadministrators or researchers. Currently this approach is used foronly a limited amount of research, and therefore it is worthwhilespelling out briefly how it might work.
One procedure is to have a panelselected randomly from volunteers from user groups. The panel wouldhear submissions from researchers and other interested groups andthen make decisions about research priorities and funding.
Another possible and ratherdifferent procedure is for user groups to prepare submissions forresearch to be carried out by particular researchers. Theresearchers, who would be expected to do a certain amount of researchselected from the user-group applications, would choose whichprojects they preferred to undertake. This is analogous to the wayuniversity departments are expected to provide a certain amount ofteaching selected from areas that students want to study. Thejustification for community-based allocation is to serve humanneeds.
So-called "science shops" -perhaps better described as knowledge shops - are the closest thingto community-based allocation. They take questions from communitygroups, trade unions, small businesses and other organisationswithout significant resources for research, help to turn the thesequestions into researchable topics, and seek to find universitystudents or staff to carry out relevant projects (Farkas 1999; Zaaland Leydesdorff 1987).
Assessment
Table 1. Each method of allocationis matched against each major type of problem.
| Problem Method | Bias | Waste | Discouragement | Orientation to interests |
| Administrative decision | Potential bias in favour of insiders, dominant groups | Preparation of proposals; decision system overheads | Poorly supported researchers may become demoralised | Incentive to serve interests of administrators |
| Peer review | Potential bias in favour of insiders, dominant groups, successful researchers | Preparation of applications; grant scheme overheads | Unsuccessful applicants may become demoralised | Incentive to serve interests of granting body or peers |
| Performance-based funding | Potential bias in favour of successful and/or superficial researchers | Grant scheme overheads (especially collection of performance data) | Low output performers (including those with high quality) may become demoralised | Incentive to serve interests associated with output measures |
| Equal allocation | Bias against those excluded from the equal allocation | Money spent on those who are unproductive | High performers may resent allocations to those who are unproductive | Incentive for researchers to serve their own interests |
| Community-based allocation | Potential bias in favour of insiders, preferred groups | Effort spent by community groups in finding researchers | Researchers who are not sought after may feel unworthy | Incentive to serve community interests |
Although no allocation system is free of bias, waste, discouragement or orientation to interest groups, the expression of these problems can be quite different with different systems. For example, community-based allocation provides incentives to do research serving quite different interests than administrative decision. The choice of an allocation system both reflects and shapes an ongoing connection between researchers andinterest groups.
Most of the complaints commonly voiced about grant schemes - such as bias in favour of insiders or against innovators - concern more-or-less inevitable features of competitive systems of allocation. The usual focus on the problems with peer review and biases in grant schemes draws attention away from both the realities of how research priorities are set and the possibilities for greater community participation in setting researchagendas.
Broad research priorities are set primarily by administrative decision. Concentrating on problems with peer review systems diverts attention from this reality. Indeed, peer review does not even provide the means for making many of the central decisions affecting research, such as decisions to set up institutions or departments or to provide greater infrastructure or funding for certain faculties. This is because peer review mainly concerns judgements within disciplines (peer groups). The key research decisions, by contrast, concern judgements betweendisciplines.
Concentrating on problems with grants schemes as they currently exist also diverts attention from the possibility of greater participation by community groups and individuals in setting priorities for research. There are many ways in which such participation could be increased, such as by community representatives on institutional boards or advisory panels for research groups, as well as being given a say over potential research projects (Bammer et al. 1986). But most such proposals are well and truly off the agenda. The main contenders for influence are interest groups (especially government and large corporations) and researchers(especially elite researchers).
Essentially, the perennial complaints about grants schemes reflect the dependent but privileged position of most researchers. They benefit from the allocation of substantial social resources to their salaries and research projects. They are reluctant to question the dominant institutions that control their funding, or the framework in which it occurs. Few of them believe that groups in the community - at least those with little money - should have any direct say in research priorities. That would be threatening to their prerogatives and status, built up throughpeer systems.
The Australian researchsystem
Recently, the Australian government has made major policy changes affecting funding for higher education research (DETYA 1999). Although the changes are significant so far as Australian research policy goes, they are relatively minorwithin the wider context outlined in this article.
Administrative decision remains the dominant method for setting research priorities and allocating funding for much Australian research, including most corporate research both in-house and sponsored as well as much in-house government research. Within the university sector, administrative decision is commonly used to deciding on major infrastructure projects and the relative staffing and funding of different departments. Often these decisions are mediated through formulas, themselves agreed to through administrative processes. None of this is greatly affected by the government's policychanges.
Academic staff are expected to do research and typically are allocated a nominal proportion of their work week to do so. This proportion can vary enormously, from full-time for research-only positions to virtually zero for staff overloaded with teaching or administration. Despite the large inequalities in available research time, nevertheless it is more equally distributed than research grants. It is certainly the element of research support that most closely fits the principle of equal allocation. In addition, full-time research students, who carry out a substantial proportion of research done in universities, typically spend most of their time engaged in research, even more closely following the principle of equal allocation. The government's policy changes do not explicitly address time available for research, though there may be changes in time allocation as an indirecteffect.
Community-based allocation has never been a basis for research funding in Australia. Australian science shop initiatives have received little institutional support (Bammer et al. 1992). There would be a few cases where community groups, without research funds to disperse, approach academics or students in order to get relevant projects carried out. The government's research policy is entirely geared towards research oriented to groups that can pay, especially large corporations and government itself. The white paper makes no mention of communityinput into research priorities.
Peer review is the method used by the Australian Research Council (ARC) and the National Health and Medical Research Council (NHMRC) to decide allocation of research grants. However, peer review operates only within an overall framework - such as the relative allocations to engineering and humanities - that is decided administratively. The white paper signals a change to peer review procedures and their administration. Two ARC innovations will be the appointment of programme managers to oversee the peer review system and the use of external reviewers to rank grant applications, superseding the former process of relying entirely on different independent external assessors for each application. These changes may well have significant effects at the level of individuals and groups applying for grants, but at a more general level they can be seen to be a minor modification within thepeer review method of allocation.
Performance-based funding is used in Australia for determining 'institutional grants', the term for a component of research funding for universities that is allocated on a competitive basis. (This role for performance-based funding is unusual in an international context.) The white paper reports a change in the method of calculating this funding: it is now to be 60% based on success in obtaining research income, 30% on research student numbers and 10% on research output measured through publications. This is a change in percentages from previously, especially in putting more emphasis on research student numbers. However, performance-based funding is not used at the level of grant applications, so the effect of the policy changes on individual researchers, which will be mediated through administrative decision-making systems within universities, probably will belimited.
Thus, the government's initiatives on research funding, while significant in relation to previous patterns, are relatively minor within the wider context of possible methods of allocating funds. Administrative decision remains a dominant force, augmented by peer review and institution-level performance-based allocation. Equality as a method of allocation appears to have a gradually declining role, while community-basedallocation remains totally marginal.
As for the problems associated with grant schemes, little is likely to change. Problems of bias, waste and discouragement will remain. More significantly, pressures to orient research to dominant interests, especially corporations and government, are increasing. Grant schemes requiring alliances between university researchers and 'outside partners' - typically corporations or governments - have been increasingly important in recent years. This increases the role of administrative decision, exercised by the outside partner, and puts pressure on academics to link their research to groups that have money and resources. Outsidegroups without money are left out of the picture.
What does this picture imply in terms of recommendations for change? The answer depends crucially on one's goals. If the goal is to serve the interests of large corporations and governments - which are commonly legitimated by equating them with the 'national interest' - then further expansion of grants requiring alliances with 'industry' would be called for. If the goal is to promote innovation (both technological and social) then, arguably, tied money and peer review serve as deterrents and a much better strategy is to provide ample guaranteed funding for a substantial period (Horrobin 1996), along the lines of the equality model. If the goal is to help solve problems raised by those without money and power - such as poor people, communities under environmental assault and people with disabilities - then a move toward community-based allocation is theway to go.
How to move in any given direction is a big topic, beyond the scope of this paper. The key point here is that the design of grant systems involves a set of social choices that have wide-reaching effects, yet most discussion about grants takes place within a narrow set of assumptions, without mention of dramatically different allocation principles and associatedconsequences.
Acknowledgments
I thank two anonymous referees andAapo Skorulis for valuable comments.
References
Arditti, Rita, Brennan, Pat and Cavrak, Steve, eds. 1980, Science and liberation, Boston,South End Press.
Armstrong, J Scott 1996, 'The ombudsman: management folklore and management science - on portfolio planning, escalation bias, and such', Interfaces, 26(4),pp 25-55.
Armstrong, J Scott 1997, 'Peer review for journals: evidence on quality control, fairness, and innovation,' Science and Engineering Ethics,3(1), pp 63-84.
Bammer, Gabriele, Emery, Merrelyn, Gowing, Linda and Rainforth, Jennifer, 1992, 'Right idea, wrong time: the Wisenet Science Shop 1988-1990,' Prometheus, 10(December), pp 300-310.
Bammer, Gabriele, Green, Ken and Martin, Brian 1986, 'Who gets kicks out of science policy?', Search, 17 (January-February), pp41-46.
Campanario, Juan Miguel 1995, 'On influential books and journal articles initially rejected because of negative referees' evaluations', ScienceCommunication, 16(3), pp 304-325.
Chubin, Daryl E and Hackett, Edward J 1990, Peerless science: peer review and U.S. sciencepolicy, Albany, State University of New York Press.
Daniel, H-D 1993, Guardians of science: fairness and reliability of peer review, Weinheim,Germany, VCH.
DETYA (Department of Education, Training and Youth Affairs) 1999, Knowledge and innovation: a policy statement on research and research training, Canberra,Commonwealth of Australia (http://www.detya.gov.au/highered/whitepaper/).
Dickson, David 1984, The newpolitics of science, New York, Pantheon.
Epstein, William 1990, 'Confirmational response bias among social work journals', Science, Technology, & Human Values, 15(1), pp9-38.
Farkas, Nicole 1999, 'Dutch science shops: matching community needs with universityR&D', Science Studies, 12(2), pp 33-47.
Forsdyke, D R 1993, 'Bicameral grant review: how a systems analyst with AIDS would reform research funding', Accountability in Research, 2,pp 237-241.
Horrobin, D F 1974, 'Refereesand research administrators: barriers to scientific advance?', British Medical Journal, (27 April), pp 216-218.
Horrobin, David F 1990, 'The philosophical basis of peer review and the suppression of innovation', Journal of the American Medical Association,263(10), pp 1438-1441.
Horrobin, David F 1996, 'Peer review of grant applications: a harbinger for mediocrity in clinical research?', Lancet, 347 (9 November), pp1293-1295.
Kohn, Alfie 1993, Punished by rewards: the trouble with gold stars, incentive plans, A's,praise, and other bribes, Boston, Houghton Mifflin.
Martin, Brian 1979, Thebias of science, Canberra, Society for Social Responsibility inScience.
McCutchen, Charles W 1977, 'Bring back patrons', The Sciences, (November), pp19-21.
Peters, D P & Ceci, S J 1982,'Peer-review practices of psychological journals: the fate of published articles, submitted again', Behavioral and BrainSciences, 5, pp 187-195.
Roy, Rustum 1984, 'Alternatives to review by peers: a contribution to the theory of scientific choice', Minerva, 22 (Autumn-Winter), pp316-328.
Wenneras, Christine and Wold,Agnes 1997, 'Nepotism and sexism in peer-review', Nature, 387 (22 May), pp 341-343.
Wesseley, Simon 1997, 'Peer review of grant applications: what do we know?', Lancet,352 (25 July), pp 301-305.
Zaal, Rolf and Leydesdorff, Loet, 1987, 'Amsterdam Science Shop and its influence on university research: the effects of ten years of dealing with non-academic questions,' Science and Public Policy, 14 (December), pp310-316.
