{"id":2191,"date":"2013-04-22T07:08:01","date_gmt":"2013-04-22T07:08:01","guid":{"rendered":"http:\/\/occamstypewriter.org\/scurry\/?p=2191"},"modified":"2013-04-25T13:04:51","modified_gmt":"2013-04-25T13:04:51","slug":"a-vision-for-a-better-future-using-new-tools-of-openness-and-transparency-to-improve-the-scientific-process","status":"publish","type":"post","link":"https:\/\/occamstypewriter.org\/scurry\/2013\/04\/22\/a-vision-for-a-better-future-using-new-tools-of-openness-and-transparency-to-improve-the-scientific-process\/","title":{"rendered":"A vision for a better future – using new tools of openness and transparency to improve the scientific process"},"content":{"rendered":"

This is a guest post by Pete Binfield and Jason Hoyt, co-founders of the open access journal PeerJ<\/span><\/a>. I don’t make a habit of running posts from private companies here at Reciprocal Space but have been impressed by the innovative model of open access publishing that PeerJ represents and was glad to be able to provide them with a forum to expound on their publishing philosophy. No payment was made or requested. The views presented here below are entirely their own.\u00a0<\/em><\/span><\/p>\n

The academic community tends to view peer reviewed journal articles as the most important thing\u00a0to considered\u00a0when evaluating a contribution, or an individual. But is this actually the best we can manage? Or can we apply modern tools and more enlightened thinking to come up with new and improved ways to measure a contribution to the scientific enterprise?<\/p>\n

Journal articles are regarded as the \u2018minutes of science\u2019 –\u00a0a supposedly perfect (and permanent) formulation of \u2018the final answer\u2019. And once an article is published, it is typically measured by a simple count of the number of citations it receives (or, disturbingly, to measure its worth based on a count of the citations that other articles in the same journal happened to receive).<\/p>\n

And yet, the journal article is only a single point in time in the lifecycle of a piece of work. When we only judge an article, we ignore the individuals behind it.\u00a0And when\u00a0we judge individuals based only on the article they have written,\u00a0then\u00a0we do not take account of the processes (both good and bad) that have led up to that point, or beyond. And when we judge those articles based only on scholarly citations,\u00a0then\u00a0we do not take account of the myriad of other ways that the publication\u00a0has\u00a0contributed to the scientific enterprise.\u00a0In all of these ways, we believe that the process can be improved thanks to\u00a0the development of new tools and new ways of thinking.<\/p>\n

The typical lifecycle of any new finding is to research it; to discuss and develop it; to formally publish it; to influence others by the act of publication; to accrue recognition for having done the work; and to learn from the experience when starting the next research project. Let\u2019s deconstruct each step.<\/p>\n

The research:\u00a0<\/span>Any research article starts with original research, and typically this research is kept private – confidential to the researcher and their lab. And yet, there is an increasing belief that if you shine a light upon a process, the process itself is improved. As a result, there has been a slow, but growing movement towards\u00a0<\/a>\u2018open notebook science\u2019<\/a><\/span>, which is an attempt to persuade researchers that by openly sharing their original research, in real time, they can improve their work.\u00a0Although this degree of transparency may be too much for most researchers, it is a movement which is slowly gaining traction,\u00a0and it\u00a0is clearly an attempt to document the origin and development of a scientific thought at the earliest possible stage.<\/p>\n

Discussion and Development:<\/span>\u00a0Once the research is conducted, the next logical stage in the process of documenting a piece of work is to develop a draft, often in the form of a preprint (or, in some fields, an abstract or a poster at a conference) and to\u00a0develop\u00a0that draft in light of feedback. A preprint is simply an early version of something\u00a0which\u00a0will later evolve into a formal publication and is typically\u00a0un-peer reviewed. Many fields have retained a preprint culture (for example\u00a0the physics, mathematics, social sciences and economics fields\u00a0all have active preprint servers),\u00a0however the biological and medical fields are notable for their absence.\u00a0But\u00a0if a preprint culture were to flourish in the biological and medical sciences, it could\u00a0potentially\u00a0be the missing link in the story that this post explores.<\/p>\n

In a recent\u00a0<\/a>Scientific American guest blog post<\/a><\/span>, we\u00a0explored\u00a0some of the reasons for the lack of a preprint culture in the biological and medical sciences.\u00a0In that post\u00a0we explained that we at PeerJ have put our money where our mouth is, by launching a new preprint server for the biological and medical sciences (\u201c<\/a><\/span>PeerJ PrePrints<\/a><\/span>\u201d<\/a><\/span>) and why we think this can now be successful. If we can\u2019t persuade the biological and medical fields to adopt a culture of open notebook science, then perhaps we can at least persuade them to adopt a preprint culture. Certainly these disciplines have been at the forefront of the Open Access movement, and so it is not too great a conceptual leap to go from openly sharing a final published article, to openly sharing the earlier drafts that led up to it.<\/p>\n

The Formal Publication – Not the End of the Story:<\/span>\u00a0We mentioned that the journal article of today is a fossilized manifestation of the \u2018minutes of science\u2019. If we accept that we can show the evolution of the finding, through discovery (in an open notebook) and early formulations (in a preprint), then why should we accept that the evolution of this work stops at the moment it is published in a\u00a0journal. Surely a journal article can be in error and require revision, or new results might strengthen or weaken the case? Instead of publishing an entirely new article (simply to put another \u2018counted\u2019 publication into your resume) why not revise and extend the existing publication? Again, this might be asking for too much from today\u2019s scholarly society, but once again we can see that the ability to publish drafts (aka preprints) and to measure their unique contributions, could be applied to this\u00a0use case\u00a0as well.<\/p>\n

Measuring the Contribution:\u00a0<\/span>Many people are now building what they hope will be better tools to evaluate academic contributions. Most visibly, this manifests itself in the \u2018altmetrics<\/a><\/span>\u2019 movement (also known as\u00a0<\/a>\u2018article level metrics\u2019<\/a><\/span>\u00a0when only concerning itself with an article). Altmetrics attempt to measure every way a piece of work might influence the wider world – be that scholarly citations; mentions in Wikipedia; media coverage; tweets; online usage; blog posts; or changing government policy. This approach has great promise as a way past the impasse caused by simply counting one\u00a0metric\u00a0(the scholarly citation) and is being increasingly used by publishers such as\u00a0<\/a>PLOS<\/a><\/span>,\u00a0<\/a>BMC<\/a><\/span>,\u00a0<\/a>PeerJ<\/a><\/span>,\u00a0<\/a>Frontiers<\/a><\/span>\u00a0and\u00a0<\/a>eLife<\/a><\/span>\u00a0to provide richer metrics on their published articles.\u00a0But the potential is far greater than this limited scope might show – the beauty of the approach is that\u00a0it can be applied to any digital object –\u00a0it can be applied\u00a0to open notebooks, to data sets, to software code, and to preprints too (all of which typically receive zero scholarly citations).<\/p>\n

Gaining Feedback, Discussing, Debating and Learning:\u00a0<\/span>It is important to talk about the \u2018discussion\u2019 that goes on around a piece of work, or the questions and answers that are posed to experts in the field. A finding could be the most impressive one in history, but if an author refuses to respond to criticisms, or does not engage with their\u00a0follow\u00a0academics in developing the work further, then an opportunity has been lost.<\/p>\n

One solution to this problem is to open up the peer review itself process to the broader\u00a0public\u00a0in a practice known as \u2018open peer review\u2019. With open peer review, we can encourage (or require) reviewers to provide their names, and publish the reviews alongside the published article. This opens up a previously secretive process to maximum\u00a0transparency, it means that reviewer contributions are not lost to posterity, and it means that reviewers can get credit for having performed their review. At the same time, authors can show the evolution of their work and how it has been improved by appropriate peer review. At\u00a0<\/a>PeerJ<\/a><\/span>\u00a0we operate a form of\u00a0<\/a>Open Peer Review<\/a><\/span>\u00a0(along with several other publishers such as\u00a0<\/a>BMC<\/a><\/span>,\u00a0<\/a>eLife<\/a><\/span>,\u00a0<\/a>the BMJ<\/a><\/span>\u00a0etc) and the reception has been\u00a0<\/a>overwhelmingly positive<\/a><\/span>.<\/p>\n

Another way to track the discussion, of course, is with altmetrics. Even if the publisher doesn\u2019t provide their own ability to comment on an article, people will comment in spaces of their choosing – on twitter, facebook, or in a blog post. With altmetrics, those conversations too can be collated and evaluated.<\/p>\n

Where Does This Leave Us:\u00a0<\/span>Imagine for a moment, the tenure or promotion committee of the future. When evaluating a candidate,\u00a0then\u00a0instead of scanning down a list of where (or alongside whom) they have published, the committee\u00a0insteads scans down\u00a0a list of their actual contributions and specific findings (things which today are only available in formal publications). They can see which findings were well received by the community; which generated the most debate; which ones were most rigorously defended by the author. They can drill backwards in time to see where the idea originally came from; they can see whether or not an author is openly sharing their work; putting it into the world for early feedback; trying to better their work through well informed revisions. They can look at a suite of metrics to see if the work was picked up and reused by others; did it inform governmental policy; was it read and commented upon by top academics; did it have a true impact? And from start to finish they can see and track the entire lifecycle of each contribution – from original idea, through early drafts; formal publications; and subsequent revisions and iterations.<\/p>\n

This future scenario may actually be the nightmare scenario for the people who serve on those committees, and certainly it is a lot more complex than\u00a0the situation as practiced today. But if it can be done correctly, then surely it would provide a more holistic and nuanced evaluation of the contribution of an individual, or of their work. Evaluation \u2018in the round\u2019 not \u2018at a point\u2019.<\/p>\n

Researchers would no longer be thought of as simply \u2018authors\u2019 (which is only the end point in their process of research) – instead they can demonstrate their entire contribution and position within their community. They can show how their data sets or software\u00a0commits\u00a0contributed, for example. They can show how they are a good\u00a0actor\u00a0within the community – commenting on, and helping to improve the work of others. They can finally get credit for work that previously went unnoticed in today\u2019s environment which only cares about their published output.<\/p>\n

At\u00a0<\/a>PeerJ<\/a><\/span>, we consciously avoid a measurement of significance, impact, novelty, or degree of advance\u00a0before\u00a0<\/span>publication. Instead we are building tools which will allow the community to determine these things for themselves by looking at the work \u2018in the wild\u2019 (and increasingly \u2018in the raw\u2019). With our new preprint server (PeerJ PrePrints<\/a><\/span>)<\/span>\u00a0people are able to \u2018show their workings\u2019 and place an early draft in front of their community for broader feedback (using feedback functionality which has just been launched); the peer review process of the\u00a0PeerJ\u00a0<\/span>journal then asks reviewers to only comment on the validity, or soundness, of the science presented and at the same time promotes transparency by encouraging reviewers to name themselves; authors are encouraged to publish their peer review and revision history alongside their article; and finally the published article (as well as any preprint versions) is made available with a suite of article level metrics including usage data, referral stats, social media mentions, scholarly citations and so on.\u00a0And throughout every interaction with the PeerJ system, people can accrue credit for their activities as authors, reviewers, editors, or commenters using a new system of\u00a0\u2018user contribution\u2019 points<\/a><\/span>. As can be seen from the arguments above, we believe that by encouraging this level of transparency, and by exposing the lifecycle of a scholarly manuscript, the work of academics can be more fairly evaluated and more effectively built upon.<\/p>\n

Of course, we are fully aware that change like this will not happen overnight, but we do believe that\u00a0<\/a>change of this nature<\/a><\/span>\u00a0is now underway. We see the growing popularity of preprint servers like the\u00a0<\/a>arXiv<\/a><\/span>,\u00a0<\/a>SSRN<\/a><\/span>, or\u00a0<\/a>RePEc<\/a><\/span>\u00a0(and our own\u00a0<\/a>PeerJ PrePrints<\/a><\/span>); we see the development of new sharing tools such as\u00a0<\/a>Mendeley<\/a><\/span>\u00a0and\u00a0<\/a>FigShare<\/a><\/span>; we see a string of experiments in the peer review space such as\u00a0<\/a>PeerJ<\/a><\/span>,\u00a0<\/a>Rubriq<\/a><\/span>, \u00a0F1000 Research<\/a><\/span>\u00a0and\u00a0<\/a>eLife<\/a><\/span>; we see successful companies being launched in the altmetrics space such as\u00a0<\/a>Impact Story<\/a><\/span>,\u00a0<\/a>Altmetric<\/a><\/span>, and\u00a0<\/a>Plum Analytics<\/a><\/span>; and we see the now unstoppable rise of the open access movement which will place all content online in a maximally sharable format. It feels to us like all the pieces are now in place. It simply needs the will of the academic community to embrace the change, and to move\u00a0themselves\u00a0forwards.<\/p>\n","protected":false},"excerpt":{"rendered":"

This is a guest post by Pete Binfield and Jason Hoyt, co-founders of the open access journal PeerJ. I don’t make a habit of running posts from private companies here at Reciprocal Space but have been impressed by the innovative … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[152],"tags":[145,244,121],"class_list":["post-2191","post","type-post","status-publish","format-standard","hentry","category-open-access-2","tag-open-access","tag-peerj","tag-publishing"],"_links":{"self":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2191","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/comments?post=2191"}],"version-history":[{"count":0,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/posts\/2191\/revisions"}],"wp:attachment":[{"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/media?parent=2191"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/categories?post=2191"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/occamstypewriter.org\/scurry\/wp-json\/wp\/v2\/tags?post=2191"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}