NEJM Open Access Round Table

As hours of lecture pass by, I, like most of my fellow students nowadays, will flip through my smart phone to check out facebook, check my email, check facebook again, check what the weather is like today and tomorrow and hope for a nice weekend, and then zone back in to realize I lost 5 minutes of lecture on a critique of a journal article.  As more medical schools emphasize the teaching of evidence based medicine and work into curriculum the appraisal of research articles, the more peculiar I found on my smart phone a discussion we've been having on this blog for a year and a half.

I have an app (free) for the New England Journal of Medicine which provides weekly editions of its articles (again free of charge) and I became engrossed and totally failed to pay attention to the appraisal of a paper discussing dexamethasone to treat bronchiolitis.  (I read the article, from NEJM ironically enough, and it's a solid one, but I digress.)  I failed to pay attention because my app showed me this week's (February 28, 2013 Vol 368, No. 9, pages 785-793) NEJM topic of focus--open science!

I don't want this to become an advertisement for this free app or OSK endorsing NEJM over other publications (they do provide electronic articles free of charge to readers though from my experience).  But there was some fantastic commentary both for and against open access. 

Check out the link.  And I'll try to pay more attention in class.

OSK solves the journal problem

I've enjoyed writing little pieces for OSK. Unfortunately, as I leave Kentucky, my writing days for Kentucky's open science source of record must end. I want to leave this last piece documenting thoughts my colleagues and I had when starting this endeavor a year or two ago. We tried to bring it to fruition, but we simply did not have the time or resources to do so. We hope one of you will.

This is not a comprehensive characterization, but it's the beginning of a comprehensive solution. Big ups to my partners on this project, Adam Robison, Keevin Bybee, and Patrick Bybee. Thanks to many others who've contributed time and mind.

---------------------------------------------------------------------------

The academic publishing enterprise is archaic at best, unethical at worst. Read commentary here.

Below is our solution to this problem. It would be a huge undertaking to create it and we don’t have the requisite skills/time to do so. We just want to float it on the web so someone else might pick it up and start the process of making it a reality.

Crowd-sourced ratings systems, on a large enough scale, can be very effective. We envision a publishing system where scientists upload novel research to a cloud-based platform in somewhat traditional manuscript form, including introduction, methods, results, conclusions, limitations, etc. Peers will evaluate these manuscripts by the same criteria that are used today. Diverging from today, manuscripts will be linked to underlying data supporting said manuscripts. The value of the manuscript will be decided by the peer community, rather than by 3-5 anonymous “experts” requested by the manuscript’s author or selected by the journal’s editor. Non-anonymous researchers will evaluate the manuscript for quality/importance, leaving a “score” and supporting discussion for posterity (comments that will also be given a score by reviewers). Manuscript, based on an algorithm, will be given a comprehensive score that will be dynamic (more about algorithm below). Good manuscripts will make the “front page”, their visibility will increase as they’re targeted to “high value” lists, and poor quality manuscripts will fall to the bottom. However, all manuscripts will remain visible/searchable, allowing an idea ahead of its time to receive proper recognition with age. Authors will be listed as now, and each author will be linked to a historical research profile. Each manuscript/comment score is factored into a historical user score, which will serve to weight the importance of comments left on other manuscripts/comments.

In the end, manuscript submission is completely transparent, has supporting data, includes relevant critical commentary traditionally seen only by authors, eliminates the "third reviewer" phenomenon, rewards authors with novel ideas without regard to institution/financial backing/geography, and overcomes problems documented at the polymath project. It would have completely open access. It would eliminate user frees, freeing information to all. It would have other perks, but you get the idea.

We envision this system laid out in a three-tier model with a “dashboard” type user interface streamlining ease of use. The “bottom” level of this system would include all data sets. Datasets would include relevant epidemiological characteristics. Datasets would be linked to citations of data within articles that’d identify which pieces of data, equations, and interpretations were used therein. Checking for accurate interpretation of data by peers would be simple and limits could be built into citation software prohibiting inaccurate data utilization. All new manuscripts would be tied to data in a dataset, though the dataset used might not be new. An independent user might identify something novel in a dataset not identified by the original dataset poster, and he/she could create a separate manuscript from it, giving full credit to the original submitter with data-links. Original posters would be rewarded each time a dataset was used successfully via historical user scores mentioned throughout this article (encouraging submission of data for community processing). This system would reward transparency and thoroughness on original submission.

The “middle” level of the system would include manuscripts and their associated comments. We envision this level being a split screen, allowing users to read the new manuscript, while older manuscripts cited in the new manuscript would come directly into view on the second screen (this could be customizable to allow several manuscripts and data, multiple manuscripts, and outside sources to be viewed simultaneously for cross-referencing). Users could read new manuscripts, give them a rating, and leave comments with citations in the form of discussion here. Comments could address issues with new manuscripts, or the data underlying new manuscripts, directly. Consequently, rather than discrete manuscripts, new information would be disseminated as a novel dialogue, allowing important commentary on new ideas to be included in the historical record (ensuring accountability for new information, providing context, and facilitating quicker synthesis of new ideas with older ideas). As above, good manuscripts/comments would float to the top via a strong score, rewarding the submitter and the scientific community.

The “top” level of this system would be an information synthesis similar to Wikipedia. This level would allow users to cite and synthesize information from manuscripts and comments (as well as, initially, from outside sources) in discrete articles “about” scientific topics. This level eliminates the need for independent external publishing sources, as all discrete information at this level could be linked directly to relevant manuscripts/comments discussing it, making “truth” easily linked to primary sources.

Underlying this process is an algorithm accounting for the spectrum of user contributions. A user’s individual score would account for all of his historical contributions to the project, including scores for manuscripts provided, when/where/how often each manuscript is cited by other users, constructive comments left, and datasets submitted. Credentials (appointments, higher education, etc), similarly, could be accounted for in the individual user score. Value associated with comments he/she leaves on other manuscripts would take historical user score into account, weighting his/her comments fairly for historical contributions. Similarly, new manuscripts submitted by a user would receive higher/lower baseline ratings based on historical contribution. We envision this score being included on academic curricula vitae, replacing absurdly long lists of publications/abstracts that give no indication of the quality of individual manuscripts.

The beauty of this system is that it allows complete transparency while offering each user a rating assigned by his peers. It would include translation software, eliminating linguistic barriers. It would allow new literature to be targeted directly to user preferences, allowing users to customize lists of manuscripts they would like to read and review by key words, discipline, user score underlying a new manuscript, “hot’ manuscripts receiving a lot of attention from readers/commenters, country/state/neighborhood of origin, etc. It would facilitate streamlined and effective literature searches, allowing users to search for key words across the entire body of scientific knowledge, including comments and datasets. It would force improvement of data quality by making data subject to inspection along with commentary on that data. It would facilitate high quality meta-analyses by allowing users to compile original data for processing, rather than processed data. It would eliminate siloed departmental research by opening commentary to all disciplines and encouraging interdisciplinary collaborations.

This system could be supported financially by grants, government funding or other charitable contributions. Alternatively, we envision this system having the potential to support itself (and be quite lucrative) through advertising revenue. Because a user’s entire historical research archive would be posted, his/her search and review history would be available, his biographical sketch posted, his discipline noted, advertisers would have a wealth of information for targeting. For example, a device manufacturer selling pipette tips would know that user X recently posted a dataset in which 30,000 pipette tips were used. That company would also know that user X searched for manuscripts from users completing similar research. Consequently, it would be clear that user X purchased many, many pipette tips and advertising could reflect this.

As I said, this is not a complete characterization of this project. Leave your thoughts and questions in the comments!

Open altruism..?

A quick post to link to a website called Admitting Failure.

This is a great site started by the guys at Engineers without Borders that allows folks working in various altruistic fields (for lack of a more proper term) to post their stories of perceived personal failure. The premise is that, in the decentralized world of international NGOs, an excellent way to do better work would be to learn from the mistakes of others.

Worth a look.

Take a stand against the academic publishing industry.. Do it now

If you're reading this, you might have some interest in signing this list. If you're going to sign it, do it now.

Many of you reading this (my peers) are very, very early in your academic careers. I understand the fear of signing a pledge of this sort without the blanket of anonymity. However, in my opinion, the momentum is shifting away from the archaic medium that is the academic publishing industry -- who wants to be caught falling behind a tide like this? Make a pledge to yourself to be a reformer. Promise yourself you'll overcome whatever challenges you face by doing the right thing. This pledge should indicate that you're among like-minded people who will follow you into this fight.

You'll find my name on it. I hope to find all of your names there.

RIP Aaron Swartz

If you're reading this blog, you're likely aware of Reddit and the passing of one of Reddit's founders, Aaron Schwartz (link to Ars Technica here). I can't say I knew anything about Schwartz prior to his death, but one remarkable achievement of his life is relevant to the Open Science Kentucky blog.

In 2011, Aaron Schwartz was charged with "illegally" downloading millions of academic papers "owned" by JSTOR, a publishing middle-man that owns its users' scholarly content once they submit it for "publishing", with the intention of sharing them with the world. Schwartz faced decades of imprisonment for this crime. If convicted, his sentence, no doubt, would've served as a stern warning to those who'd challenge lucrative publishing conglomerates and their undue influence on our civilized society.

We've written about the absurdity of the academic publishing industry here, here, here, here, here, and probably elsewhere too. I could link to any number of thousands of conflicting articles in competing journals cited by proprietary news organizations owned by those profiting from sensationalism collectively indicating how huge a problem the academic publishing industry actually is. Not that I'm alone in my disdain; there's nearly ubiquitous acrimony among those in the research community toward the highly-flawed publishing industry (here's a tongue-in-cheek portrayal of this) and its lucrative stranglehold on the public record of knowledge.

So, Aaron Schwartz committed a crime and would've been punished as dictated by law. The law upvotes and downvotes, it does not decide right and wrong in the larger sense. Deciding legal versus illegal, right versus wrong, is up to us. We've decided, as a civilized society, that information (often created with public tax dollars) can be owned by individuals and corporations. We justify it with any number of platitudes and anecdotes about innovation and genius and the university and impact factor and curricula vitae. We hand our lifes' works to publishing houses and trust that they'll use them for the benefit of the communities we cherish. Again and again, we learn that our system is flawed. Yet, we continue on this course, punishing those who challenge it with academic isolation and prison.

Aaron Schwartz believed that information belongs to everyone. We agree.

FOAM, not what you think

Free Open Access Medical Education, who doesn't already love the concept, regardless of the number of hundreds of thousands of dollars "invested" into one's education?
http://lifeinthefastlane.com/foam/, #FOAMed (twitter, i guess)
As I've been only reading the emergency medicine blogs, i first stumbled across it there, but it seems to be a rapidly expanding movement promoted by the online leaders in the field (Scott Weingart, of EMCrit; Michelle Lin, of academiclifeinem).
I feel this is extremely pertinent to the Open Science Movement and wanted to keep everyone abreast of this new phenomenon!

One Scientist's Solution...

...to the question of where do we get funding if not from institutions and grants, which provide such research money based on merit defined as publications, presentations, etc, is crowd-funding the fees.

A really neat article and video about Ethan Perlstein's approach to the open science and open source model and his research on evolutionary pharmacology.  This is a fantastic idea and could be used as a model for other researchers out there.  This at the very least could be used as a means for researchers to  engage open science and sharing data and cultivating a public collective conscience about scientific research and its funding.

Greg Henry's "Renewing Research"

Some great commentary on working toward a smarter research culture: "Renewing Research," by Greg Henry.

Generally, he's reminding you to do the right thing because it's the right thing to do, whether you're doing research or running the free world. Huge appreciation of his liberal arts-style editorializing in the medical literature.

Here's a link.

For those involved with emergency medicine, there are a lot of strong, provocative articles linked from the parent website.

Spin in medical literature, a cohort study

Hey, all,

A cool little study published in PLoS Medicine linked here.

Essentially, it's a "retrospective cohort study" looking at factors influencing relative hyperbole, or "spin", that appears in the news media following the publication of new medical science (specifically, randomized controlled trials). The authors conclude that, most commonly, spin creeps into the news when study authors include it in press releases. Ultimately, it suggests both that press releases need to be done responsibly and that the news media rarely reads or digests our newly published scientific literature!

I'll admit that, when overloaded with things to read and process with limited time, I occasionally take the easy route, skip past my assessment of medical literature, and begin with the conclusions. I think we all do.

There's clearly a problem if we're expecting outsider-types to digest our literary canon to separate our trashes from our treasures. They don't have the background in epidemiology/biostatistics/whatever scientific field we're publishing from to do so. Ultimately, even if they did, it isn't in the press' interest to restrict hyperbole.

So it's agreed: responsible, accurate press releases from now on. Good.

Open Science Competition

This is a pretty awesome:  Genspace and Assay Depot are holding a competition in the tri-state area of NY, NJ, and Connecticut looking for the best amateur innovators and researchers with cash prizes for receiving top marks.  This article does an excellent job of describing all the basics behind it. 

What would be great is the ability to have a much wider dispersion of these sorts of competitions.  As much as anyone may like to tinker or experiment, having the incentive of cash to really throw some results to the greater public is a perfect theater for open science to grow.  These competitions also would provide an excellent means of getting these amateur scientists to meet and possibly collaborate, putting a face to to the facebook of open science, perhaps.

Open Science Startups and Supercomputers

Seems like everyone is figuring out the shtick of open science can be a highly desired enterprise:  Mendeley, a London-based startup, acts as a Wikipedia of open science and has really exploded in terms of its monthly traffic.  Check out Mendeley's site here to learn more.

Not only that, more and more investments into collaborative science are being made with hefty investments.  The National Energy Research Scientific Computer Center (NERSC) signed a $40 million supercomputer agreement with Cray Cascade System with a specific aim of advancing open science efforts. 

So here we have examples of how there are very lucrative means of engaging open science as well as sites available to really get involved, not to mention innumerable other groups like Mendeley.  Seems like some things are going in a very positive direction.

Open-Access UK

In an unprecedented move toward open-access, the British government unveils plans to make all tax-payer funded research available free to readers by 2014. Controversial? Yes. Necessary? Absolutely.

Read the complete article here.

Can we expect a similar transition in the US?

Call your congressman!

Sequestered Data Hinders Scientific Process

Apologies for the lengthy sabbatical, open science news has been abounds.  Recently large companies with monstrous data banks of (occasionally personal) information have been working with a select number of researchers to publish articles, but refuse to share such banks with other researchers for any verification or scrutiny.

"It’s antithetical to the basic norms of science to make claims that cannot be validated because the necessary data are proprietary." - Michael Eisen

Check out the article here.

 

Harvard slams the publishing industry racket

For those who haven't participated in scientific publishing directly, I'll describe it briefly:

A scientist, potentially funded by your tax dollars, has research he wants to publish. He writes a manuscript using data he's accumulated using this funding. That manuscript is submitted to a scientific journal (like Nature or JAMA) and undergoes a peer-review process of some sort (which may or may not be free for the author). Let's assume that manuscript is accepted for publication by the journal. The journal "publishes" the manuscript, at which point it is included in print and online versions. The manuscript can be accessed through a direct subscription to the journal or to a publishing house like Elsevier. These subscriptions are very expensive.

So the scientist pays to provide content and the reader pays to view it. What exactly does the publisher do?

This isn't meant to be exhaustive (I wrote about it a bit more here), but one definitely feels a little icky thinking about where all that money goes.

In today's Guardian, Harvard University notes that it can no longer keep up with the costs of maintaining a comprehensive database of journals. In real terms, this means Harvard students/faculty/physicians don't have access to all new information. If Harvard, with its multi-billion dollar endowment, can't keep up with rising costs, there's no doubt this system is in trouble.

We at OSK are glad to see Harvard is thinking progressively. We're disappointed it's only doing so for financial reasons. Such is life, I suppose.

Data dump

At OSK, we love data.

Big ups to the Obama administration for a $200 million data dump.

Research Works Act defeated

Huge success toward preserving the open science movement's momentum.

The Research Works Act would've set the open science movement back years by restricting congress from requiring that research created at the tax payer's expense be made publicly available. More clearly: the Research Works Act would've prohibited congress from requiring tax payer-funded science be made available for free to the tax payer.

I couldn't find a good summary in our "paper of record," so please check out the Slate.com article here.

Also worth noting is the (originally) unconditional support for this bill by Elsevier, a publishing conglomerate. I'm not sure their support of it can be construed as anything but self-preservation, greed, taking a cut of the pie, a desperate act by an unnecessary middle-man.

I ask again: why do we need publishing houses? Why do we even need journals?

We do not. We need an effective, rigorous peer-review process that'll provide transparent, high-quality scientific publishing. This absolutely does not need to be accomplished via publishing houses and journals. We have other options.

Love to DataLove

Huge ups to Datalove.

Check them out. Get involved.

Boycott publishing?

A great little piece from our northern neighbor on the absurd financial entanglements of the publishing industry. The backlash against academic status quo is clearly growing.

Read it here.

We still need a better system with which to publish.

The Cautionary Tale of Anil Potti

Anil Potti is a former Duke University researcher who was one of the catalysts for the promise of chemotherapy genomics, i.e. decoding a patient's cancer's genome and picking out which chemotherapy would work best at killing it. It's an amazingly fascinating concept and it's still actively pursued to see if there is something to salvage. The reason there is something needing to be salvaged, however, is because Dr. Potti fabricated data. All of it. This was going to be the 'next big thing' in medicine, right up there with regenerative medicine, as the panacea to one's (cancerous) ills. Thankfully other diligent researchers closely examined the data and found incongruities and finally brought the sham of research down. If there was a more open source method to data review, perhaps these inconsistencies would have surfaced sooner. Because of the current system's behind-closed-doors policy, real people were subjected to faulty research premises. While it has yet to be shown if the patients did not receive true standard-of-care, it begs the question. Open Science: it can save lives.

You can learn more about Dr. Anil Potti here, here, and here.

Also, you can look up Dr. Potti and find several websites full of stock photo images of doctors and smiling young people as he details all of his grand exploits and gleans over his recent misconduct. He also falsely claimed he was a Rhodes Scholar. So yeah, he's a real piece of work.

Federal Research Public Access Act

Here is an interesting article about the Federal Research Public Access Act. It provides links to a .pdf file that's 7 pages of the actual bill, which opens up federally-funded research to public access six months after publication (i.e. a really thoughtful name to the bill). What is great about this is that it follows on the heels of the Research Works Act, which would have effectively destroyed such a possibility and nullified the NIH open access policy. The latter bill was backed by Elsevier Publishing and stinks to high heaven; already numerous scientists have signed a boycott of Elsevier Publishing. Read on and contact your state representatives to support FRPAA and squash RWA.