Three Tales On Research Citation, Funding and Freedom

It is an ever growing development in the world of academic research, that researchers are being measured by various criteria of performance output. Two of these are citation – how much fellow researchers refer to your work, how much you have - as the saying goes - "academic impact", and ability to attract external funding. A third, that's been coming strong recently is the "societal relevance", which has prompted governments and purpose specific public as well as private financial backers of research to allocate money more and more within restricted programmatic schemes aimed to support research in (allegedly) pragmatically important areas. There is a lot of discussion of these developments among academics, and in the arts and humanities there is often a complaint that the systems installed are rigged at the outset to strongly bias in favour of biomedical and technological science, to some extent a bit of selected natural science. However, I'm not going to go down that route here. Neither will I question to what extent any of these parameters are indicators of research quality, mostly because I don't believe there is any such thing as a ready-made clear concept of research quality to use for such an assessment. My rather profane view is that different parties may be interested in backing, having or doing research for a number of very variable reasons, and any assessment of the outcome would be best assessed in light of whatever such reason is thought to motivate the endeavour.

Instead, I would like to tell three brief tales from my own research career, complicating things for anyone who think that the three parameters mentioned have some sort of joint relevance for how to assess research. This is not to say that there aren't other parameters one might want to use or that these three are "quality indicators", then, just that I will concentrate on these three, currently in strong vogue. My point with the tales is to show how easily scoring on the three parameters may come apart. More specifically, I will use them to illustrate how citation may come apart from both external funding, and dependence on directive research funding schemes.

Tale 1: The Ethics of Gene Technology and Sports
This is a field that I was more or less lured into by my friends and colleagues at the time, Torbjörn Tännsjö and Claudio Tamburrini. Claudio was pursuing an interest in the ethics and philosophy of sports, in particular the ethics of doping regulation, and him and Torbjörn has secured a book contract with Taylor & Francis to produce an edited collection on that topic, underscoring the impact on elite sports of technological development. I, at the time, had just started my tenure as senior lecturer in practical philosophy at the University of Gothenburg, had no research funding to speak about and was busy as hell with teaching many new courses, as well as working on various research proposals to secure some funding. Then Torbjörn and Claudio asks me if I want to contribute a chapter on the possible application of gene technology for enhancing sports performance, using the background of my former research on biomedical applications of gene technology undertaken in my post doctoral work. Hungry for merits as I was, and also a bit intrigued by the topic,  I accepted, and then spent quite a bit of my private time researching and writing up the piece the following months – I had no special funding of any sort, and the little faculty funding within my appointment for research was already more than fully claimed by other work. The book, Values in Sport: Elitism, Nationalism, Gender Equality and the Scientific Manufacture of Winners came out in 2000, and can be downloaded in its entirety for free from this source, including my contribution: "Selected champions: making winners in the age of genetic technology"

Pretty quickly, my chapter attracted attention among other scholars in philosophy, ethics and sports studies, I was invited to conferences and workshops, and a few years later I was again invited to contribute to a follow up volume edited by the same people, eventually published in 2005, and available for free download from here, and featuring my chapter "Ethical aspects of controlling genetic doping":

 
Once again, I wrote this chapter without any purpose specific funding, although by this time, I had secured a few projects on partly related areas and also been allocated a slightly better slice of faculty funding. However, writing this contribution was a direct result of the former chapter and would have been impossible to undertake withoput the existence of that former investment of work.

Later, both chapters have been reprinted in state-of-the-art edited collection volumes for use in academic teaching, here and here, and the first one is now my second most cited piece of writing ever, using the Google Scholar Citation tool, having collected 44 citations so far, and continuing to cited up during this year. The follow up chapter is not as well cited, being less radical in its main thesis, but also that one continues to attract interest.

Tale 2: The Morality of Scientific Openness
This story takes us even further back in time, to the period of my post doc days, which I spent in the now defunct Centre for Research Ethics, at the time run by the Royal Society of Arts and Science in Gothenburg, funded by a research project on the ethics of prenatal and other genetic reproductive testing. During that time I and the year following receiving my PhD, I had been sketching a text book on the ethics of technology that eventually never saw the light of day. One chapter in that book had been planned to be about the ethics of openness, secrecy and sharing as well as withholding information in science and research. I happened to mention this to the director of the CRE and my senior colleague in the project, Stellan Welin, who asked me if I had thought about this as a possible venture for a more advanced text, mentioning the angle of patent regulation and a context of scientific history I had been unaware of. Excited, I made some preliminary probing and, based on that, we submitted an abstract to the Congress of Logic, Methodology and Philosophy of Science, which was accepted and eventually presented in the Florence edition of this major event in the philosophy of science in 1995. Having received encouraging response there, back home we decided to draft an article, at the same time discovering the rather new journal Science and Engineering Ethics, which seemed perfect for the piece we planned. Thus submitting there, the resulting article "The Morality of Scientific Openness" (free online postprint), was published the following year in the second volume of this journal.

This entire work was the result of spontaneous impulse, lust and the time used for it was not funded in any way, especially not by any purpose specific allocation of money, besides a little bit of extra support from the Wennergren Foundation to cover some of the expenses to attend the CLMPS congress. Less sensationalist and more dryly scholarly in its tone, the piece hasn't rocketed to the top as the first chapter of the previous tale, but it is among my better cited works, and also this one continues to attract interest all through up during this year.

Tale 3: The Philosophy of (Free) Improvisation
This story takes us even further back, to time of my PhD candidate days. having always had a side-life as an improvising musician, I was asked by my two good friends in that area of activity, Mats Gustafsson and Raymond Strid, to assist in the production of a special section on free improvised music of the Swedish periodical of contemporary (non-popular) music, Nutida Musik, among other things an essay of what is special about free improvisation. Doing this in parallel to finalising my doctoral thesis on the morality of abortion, in turn undertaken within a weak funding scheme, I used whatever sources I could find at the time, and wrote what turned out to be a seminal academic analytic piece on the concept of free improvisation, called (in Swedish), "Vad är fri improvisation?", published in 1992. A few years later, I was asked by Peter Stubley, who was then setting up the web resource "European Free Improvisation pages", whom I had some contact with through an emailing list for free improvisers in the 1990's, asked me if I could translate this text to English. I did so, and Peter published it on the EFI site, where it remains accessible for free. Another stretch of time later, this text was translated into Polish, and published in the Glissando magazine in 2005. All of this work was done completely outside any of the funding available to me during this time. As this work was from my standpoint not in any way central to my academic endeavours, but rather a spin-off of my musical interests, I paid scant attention to what then happened.

But, in fact, if the Swedish, English and Polish versions of this short essay are combined (1, 2, 3), it is in fact one of my decently cited works (total of 15 citations), having attracted scholarly attention from philosophy, and the general studies of music and arts, up to and including 2012, so far.

*

So, to round off, all of these three tales document how (relatively) well-cited (in my world) works of research appears without any sort of significant connection to pre-existing funding, even less funding for the specific purpose of the work in question. Rather, the three examples illustrate how cited research may appear despite the lack of funding, as well as despite of the presence of funding claiming a research time for more purpose specific undertakings. As I recall, the same pattern holds (without any claim to comparability in other respects) also to the story told by James Watson in The Double Helix about how the discovery of the DNA molecule came about.

Why Aren't Acknowledgements Acknowledged in Academic Citation Indeces and Metrics?

Note: a crucial amendment to the original post has been made in the last paragraph. The tip from Daniela Cutas making this possible is hereby acknowledged!

Today, as many times before I received a so-called "alarm" from Google Scholar in my email inbox. This notified me of a new research publication in the GS index related to myself, but this time it was none of my own work that had appeared online and neither one in which some of my work was cited, which are otherwise the typical contents of these "alarms". It was, instead, a new article by Neil Levy in Biology & Philosophy, called "Addiction as a disorder of belief", where I'm mentioned in the acknowledgements section, due to Neil having presented a draft of this paper at our research seminar last term and, apparently, some comment of mine had been of help.

This made me think of how important an institution acknowledgements are in the academic world, especially in those parts where extensive multi-authorship is not the norm, as in philosophy, ethics and, in fact, most of the humanities and social sciences. It is in the acknowledgements that you recognize contributions of others to your work other than having actually written it together and these can be everything from subtle to such broad or basic contributions that they cannot be captured by any specific article or book citation. Often this happens with people you interact with in the course of doing your research as they are also in a work in progress stage, which in the mentioned field is often equivalent to working on the manuscript. Or it is the contributions of people who have not published anything on the topic, but nevertheless provide useful suggestions. Thus, mentioning in acknowledgements is quite a non-trivial thing; it is, in fact, an integral part of exhibiting the collective nature of research also in these parts of academia. No wonder, then, that philosophers (and I'm sure others, but philosophers are whom I know best) pay a rather great deal of attention to who is being mentioned in acknowledgements and how, as this reveals a great deal about what interactions lie behind a piece of work and also the nature of the research environment of the author. In books, the foreword/preface (where acknowledgements usually surface) is thus a central section for geting a grip on what is many times a highly complex body of information, argument and advancement of intellectual thought (regardless if you end up agreeing or not). In the natural and technological sciences, as well as biomedicine, achnowledgement do not, as far as I have come to understand through my rather abundant interactions with people from such fields over the years, occupy this type of important position. Rather, they are an aside reserved for funding agencies, and people who do not (or have not been deemed to) deserve to be included as co-authors, part of the consortium, or whatever immense collective body is the author in the particular case. Basically, if one researcher has interacted with some other researcher in the production of a piece of research, this will show in the form of co-authorship, not acknowledgement. In the humanities and social sciences, the rule is rather the opposite, which makes acknowledgements immensely valuable, as (due to a much smaller degree of fragmentation of publication) it often may take quite a while for a working in progress to end up in the form of an actually published work – in the case of books (still the finest of merit in these fields), not seldom several years. True, some mentions in acknowledgements reflect very minor contributions, sometimes just material ones, like having invited someone to present at a seminar or provided a visiting fellowship for a brief while, but that goes for co-authorship in those other academic fields as well - where a place in the list of author has, as we all know, become a currency to trade for all sort of things material related to conducting research. You want a peak at my data?/collaborate with my post doc/profit from my comments/use my tool, I get authorship – that sort of thing.

That could have been the end of it, if it wasn't for the increasing importance for funding and career of how you score on various citation metrics. This tool for measuring how influential your work comes from the natural, technological and biomedical sciences and thus concentrate on authorship and citation of published articles. Due to its increased use also for ranking humanities and social science academics, it is now increasingly including also books on academic publishers and conference proceedings (which in some fields are just as long and peer reviewed as journal articles). However, it is still citation of a published work that counts, which means that mention in acknowledgement will not improve your citation ranking. This means, that a substantial part of how influence on each other's work and collective interaction in research is conveyed for humanities and social science researchers, is simply ignored by these systems – now used for allocating our money and assessing our merit. Simply put: the tool is currently significantly biased and rigged to the disbenefit of humanities and social science researchers for no good reason.

At the same time, it seems to me, that with today's technology in publication and research accomplishment indexing, it should actually be no problem at all to have mention in acknowledgement be reflected in citation indeces and metrics. [Added in retrospect] And indeed it isn't, as was demonstrated already in 2004, in this articel. This goes for all of the larger indexes, including Google Scholar Citations, Thompson Reuter's Web of Knowledge and Ellsevier's Scopus. Alternatively, universities, governments and funding bodies paying attention to existing metrics, should add in their assessment of humanities and social sciences a further citation dimension of this type. Otherwise, the resulting allocation and assessments will be substantially misguided and missing a crucial part of what drives research and innovation outside of the already highly privileged fields of natural, technological and biomedical science.

What's This Thing Called Online Automated Bibliometrics and Citation, Anyway? - Scam, Fraud or just Plain Hustling?

It's been a while since the last substantial post, blaming deadlines, deadlines, deadlines for that. Having submitted a major delivery this Friday, here's a sort of inhale before I jump right into the next leg of the fall semester triathlon – a major research bid that I'll be heading.

So, this week, the distinguished science journal Nature's online news section published an entertaining piece on what the outcome may be when all researchers, regardless of field, are ranked according to citation – how much their work is referred to by other researchers – using open online automated resources, such as Google Scholar, or its special citation section. Using a service called Scholarometer, Nature had this guy, slightly surprisingly to many, coming out on top (wonder who he is? - click the pic!):

Strange, isn't it? Not when considering that they have been using the so-called h-index, a mathematical construct devised to reflect the citation weight (rather than rate) of a scholar (that is, this is h-index as used in Google Scholar, in the more professionally advanced and commercial Web of Knowldge, it is something else, but the purpose is the same), thereby reflecting the value of having more articles with more citations rather than just many concentrated to one publication. They then perform what is referred to as normalisation for different scholars in relation to the size of their respective fields. So, what makes Marx come out on top is that he is a more well-cited historian than what, e.g., Albert Einstein is a well-cited physicist, considering that physics is a very much larger discipline than history is. Now, of course, none of this says anything about quality or influence on the progress of research (no more than what the Billboard chart says regarding music) – it merely measures popularity as an object of citation among fellow scholars. In fact, the notion that citation proves anything over and above that others have taken some sort of interest in one's work is highly contestable – said without denying the no doubt important use that citation and citation tracking has in science and research.

But here's the funny thing. Having been pointed to the Scholarometer toy, I of course couldn't resist checking out my own pet fields! So here's what came out when looking at the h-index ranking in bioethics – the field where much of my most weighty specialisation is located (click the image to view a scaled up version):

I could recognize some names, such as Simo Vehmas who happens to be a good friend, but several others were completely unfamiliar to me.  Now, bioethics broadly conceived is a large field so it need not be surprising that one doesn't know the name of completely decent fellows within it, but the fact that I could not place any of the top four names made me wonder. But then it struck me: wait a second, I do know one of those names, the top one at that, but certainly not in the role of a bioethicist, but as a world-renowned researcher in reproductive genetic medicine and leader of the team that performed the first successful preimplantation genetic diagnosis in the early 1990's. I happened to know this, since I published a book on the ethics in the aftermath of this technological advance in 1999 (available for online reading and download through that link). So Alan H Handyside is a prime medical researcher, which of course is what ups his h-index to such heights, as may be confirmed by inspecting a Google Scholar search on his name - what makes him top name in bioethics is, seeemingly, merely that someone tagged his name with that disciplinary affiliation. So what about A Pandiella? Same story it appears, this is a cell-biologist with a no doubt impressive citation count and, I'm certain, many important results up the sleeve. Moving on to R Frydman it's almost the same story, as the bulk of the publications are here in reproductive biomedicine, but it's more complicated as it appears that there is also another R Frydman, who is publishing in the field of health policy/economics, but these persons are treated as one! Next one, J Kimmelman is likely to be a similar story, since there is one with a good number of publications clearly in bioethics [retrospective note added after publication of this post: this person, Jonathan Kimmelman has added a comment below and clarified his affiliation, which is indeed in bioethics] and another publishing in very specialised biomedical science that has attracted vast numbers of citations (I checked some of the respective author affiliations in this case and they don't seem to match either). Last, before we get to my friend Simo, we have F Olivennes, who again seems to be a purely biomedical researcher in the field of reproductive medicine and embryology, who for some reason has been tagged as belonging to bioethics.

These, then are the top researchers of my field according to Scholarometer - no wonder I never heard of them in that role. And, in fact, it seems that the problem appears already at the Google Scholar source, for checking the top name of the straight citation ranking for bioethics, we meet this guy – yup, yet another biomedical researcher classified as a bioethicist. Number two is this guy, whoever he is, same story all over again, and then come some names I'm familiar with and respect in the way one would expect of people ranked to be at the top of one's field. Just to twist the knife some extra turns, I also did a quick check for medical ethics; same story, this is the top guy, apparently, and this is no. three I hear (number two in this ranking actually is a well-known bioethicist who happens to also be a medical researcher, so that kind of animal does exist).

So, what we may conclude is that for these fields, attempts at measuring citation has been severely corrupted by failures of disciplinary/field classification that swamp rankings with citation counts of no relevance for the field at all. I haven't looked through the entire publication lists of the people mentioned, but many of them appear to have basically no output belonging to ethics of any sort. They might, of course, have tagged along on a few ethics papers led by others as clinical/scientific experts (which is fine), but this does not make them highly cited bioethicists, it makes them medical researchers whose medical citation counts look impressive in the context of a field-normalisation to bioethics rather than medicine. In addition, we have seen an obvious identity problem, where the automated online citation counters are unable to distinguish people with similar surname plus initial – makes for quite a lot of error, I would say.

But what is the root of the classification errors with regard to field-normalised/specific citation measures? There are several (possibly overlapping) possibilities. One is, of course, that authors misclassify themselves, as may happen in Google Scholar Citation, where you as author decide what fields to belong to. For example, I could myself have made a strategic choice to pass myself off as belonging to the philosophy of medicine field, which would not exactly be lie albeit bending it a bit, and with my current total citation of 408 ended up in a handsome 6th place, rather than the less impressive placings I enjoy as bio- or medical ethicist or just ethicist. But not all authors are in this system, as you have to actively join it and manage it a bit for it to work (thus your responsibility for how you classify yourself), so the problem might also come from the classification done by the Google Scholar staff; I wouldn't be surprised if several of the strange things described earlier are due to Google's experts confusing "bioethics" with "biometrics" or "biotechnical", for example. The qualification of this staff for doing what they are doing is completely blacked out to me, as I suspect it is to most other scholars, and still many us – like the team behind Scholarometer – take it rather serious. Now, with regard to Scholarometer, there may certainly be error sources located there as well, since one may require of an academically construed automation tool that it is checked for serious error of the sort I have been displaying – which has apparently not occurred to or engaged the team at Indiana University Bloomington responsible for the product.

But wait a second! Wouldn't that mean to, sort of, making the automated citation counter, sort of, not automated? Yes indeed, that is what it means! And hence the title of this little peak into the fascinating games sometimes played in the world of academia to no apparent use for anyone. Alas, though, through the way in which governments and other funders of research are increasingly using bibliometrics and citation as quality indicators to determine the allocation of funds, preferably in an as automated way as possible (partly because of the hype represented by Scholarometer and the article in Nature), thus falling prey to the sort of weirdness here described, this sad example of pretending to have a technology that works when one hasn't, is actually putting fellow scholars and researchers at risk of losing funds and other resources, miss jobs and promotions, et cetera, for no good reason at all.

My plea to Nature and other journals, Scholarometer and Google Scholar is simply this: stop pretending that there's something there that is actually not in evidence. Those who provide these services: make them work as they should or shut them down. Scholarly media: ignore them until they have something to show for real and not merely for fancy.

See you soon!