The Issues

The real problem of irreproducible data


You have to know what is real before you know what is relevant. In science we trust that published means “real”. We may debate the impacts of different conditions and ooze skepticism over significance measurements and choose, sometimes less than tactfully, to object to an author’s conclusions, but overall we trust the data are real for the experimental conditions and that repetition of the experiment under the same conditions would yield the same outcome.

However, in a Catch 22 the data on reproducibility disagree.

In a collection of articles in the open source Nature special issue on data reproducibility, the most optimistic studies show 25 percent reproducibility of academic research. One article from Glenn Begley and Lee Ellis led to the development of Begley’s Six Rule for Reproducibility, which include blinded studies, full disclosure of all results, and reagent validation. Based on these rules, the optimistic 25 percent of reproducible studies are probably coming from far less than 25 percent of total academic labs. I know very few researchers that can honestly read all six rules without a little shame. 

On July 9th, Oxbridge Biotech Roundtable (OBR) hosted a panel discussion on this issue of data reproducibility. The panel included Dr Liz Silva, the MIND Program Manager at UCSF and the former Senior Editor at PLoS ONE; Dr Tim Gardner, founder of Riffyn; Dr William Gunn, Head of Academic Outreach at Mendeley; and Dr Corey Goodman, a partner with venBio.

Goodman opened the conversation with the first question he asks of founders pitching their science to him: “who has reproduced this?” The rest of the discussion focused on the problems, like an increasing rate of article retraction, and potential causes, such as intense pressure to publish, funding shortages, and increased oversight. Somewhat frustratingly, very few solutions were offered.  Gardner made a compelling argument for an incentivized approach to solving the problem. He suggested that two factors hold back scientists from solving the reproducibility issue: our culture and our lack of tools. We operate in “a culture of noise” in biology and accept that irreproducibility is unavoidable. However, if we employed adequate documentation tools similar to other industries it could greatly diminish the issue. Through incentives, like lower costs and timesavings by eliminating unnecessary, faulty repetitions, we can develop and adopt new tools and change the culture. (Gardner discusses this more in the Podcast referenced later.) When asked about automation as another tool in this solution, the panel felt it might be in the future, but currently there are other more pressing issues to address. (This could also be a reflection of the fear of the outdated bench scientist, but that is a topic for another day.)

Another potential, but more dogmatic approach, is a mandated NIH requirement for reproducibility standards. However, this reeks of additional strain on an already stretched system and more delays in publication, which will only compound the stresses that contribute to irreproducibility in the first place.

More adaptable and agile solutions are being developed through the Reproducibility Initiative. This partnership between Science Exchange, Mendeley, PLoS, and Figshare was briefly touched upon, but has the potential to revolutionize the way research is conducted as we incorporate tools, such as those being developed by Riffyn, and commercial replication services become cheaper and more convenient.

Overall this is a discussion that needs to continue and it requires total engagement from the entire scientific community. There is no clear solution but isn’t this the kind of problem we live for?

(To solicit more thoughts on solutions and the topic in general from two of the panelists, Silva and Gardner, I was asked to conduct an impromptu interview for a Podcast. By impromptu, I mean this interview was conducted on about five minutes notice.  I even forget Liz’s last name!  Anyway, it is clear I am not ready for the evening news.  As soon as the Podcast is up I will add the link here, so stay tuned!)


The opaque problem of medical pricing transparency

It’s easy to see the benefits of our free-market, transparent economy:  it keeps our iPhones affordable and WebMD always at our fingertips to tell us that our headache could be from stress, or a brain tumor. As you race for an MRI, you probably won’t consider the cost; and it doesn’t matter because that information is careful concealed until you get a bill for more than an iPhone…or four. Medical pricing in the US is an opaque and secretive system that leads to Americans spending far more on health care with fewer returns on investment than other Western nations. San Francisco-based Stroll Health is developing a disruptive app to make finding the price of procedures as easy as, and far more accurate than, self-diagnosis.


               Per capita spending according to OECD Health Statistics 2013 

According to the Centers for Disease Control and Prevention, in 2010 the US spent $8,402 per capita on healthcare, which totals 17.9% of GDP. Compared to other developed nations, we spend almost twice as much per capita with no improvement in outcomes. A Washington Post blog on March 26, 2013 by Ezra Klein highlights national differences through 21 graphs of outrageous pricing and spending on medical procedures in the US.


              From 21 graphs that show America’s health-care prices are ludicrous” 


              Life expectancy in the US (blue) compared to the rest of the world according to OECD Health Statistics 2013

American health care is a sluggish and entangled behemoth. We are currently facing the daunting task of not only identifying the causes of the dysfunction, but also implementing effective solutions that directly address the underlying issues. A November 2013 article in The Journal of the American Medical Association (JAMA), found that price increases account for 91% of the increased cost of health care, which far outpaces the costs associated with an aging population. The root cause of this dysfunction is a fragmented healthcare system in which parties act as lone wolves in segmented areas of the system. The lack of coordinated efforts to manage costs or fight diseases, coupled with lagging information technologies, minimizes the role of patient as consumer and further exacerbates the dysfunction. Author David Matheson stated for Russ Britt’s Market Watch article, “The extreme fragmentation makes it hard for anyone to be held accountable for health care.”

In the same issue of JAMA, Dr Uwe Reinhardt expanded on the culpability of fragmentation for sky-high health care costs. Fragmentation results from a private health insurance system that does not allow individual insurers to influence market pricing as in all other capitalist markets. With prices secretly negotiated in the private sector, health care providers do not have to provide the costs of their services to patients and doctors.

Several states have enacted policies in response to the federal government’s resistance to address the issue of health care pricing. While these policies clearly identify a pressing issue, most are too weak and have proven ineffective. Case in point, New Hampshire is one of two states, along with Massachusetts, cited in the Health Care Incentives Improvement Institute’s (HCI3) Report Card on State Price Transparency Laws as having adequate price transparency laws. However, if you visit New Hampshire’s HealthCost website, you find, “a failure in the data reporting process has made updating price estimates at the moment impossible.” The delay could last months.


               New Hampshire’s HealthCost website has yet to provide costs

What is the solution to reining in runaway health care costs in America? Federal intervention represents a coordinated effort, but it risks bureaucratic creep. The University of California, San Francisco held the first “The Future of Healthcare Transparency in California: A Multi-stakeholder Summit” last October to discuss the issue with key opinion leaders in the field. The summit emphasized encouraging voluntary action on the part of health care providers to provide pricing data and increase access to claims data. This appeal, predicated on acting together for the greater good, naively neglects the motivations of individual players within the health care complex.

Alternatively, disruptive information technology in the form of publicly accessible websites and mobile apps provide a more flexible and pragmatic solution. Nimble, innovative startups can increase price transparency by putting the consumer back into the market. To asses the growing number of available tools, HCI3 published their list of comprehensive specifications for price transparency tools based on the capabilities and drawbacks of existing applications.

Dr Reinhardt included Healthcare Blue Book and Castlight Health in his article on disruptive technologies. While Healthcare Blue book is an excellent resource, it provides “fair prices” for services based on your zip code rather than the actual prices of specific providers.  Castlight Health is more specific, but it is only available through employers and for their health care plan. Additionally, neither of these tools are available as a mobile app. A quick search of the App Store for “health care prices/health care costs” and related permutations returns less than seven potential iPhone apps including FAIR Health’s Healthcare Cost Estimator and the Health Cost Estimator, which are both based on “fair pricing”. Welter by DynaHIT LLC is the only one that approaches price transparency. In its beta version, it provides the cash price for basic primary care appointments in Chicago. However, in reality their “data was generated through a cost survey and should be considered cost estimates”.

To effect change someone needs to fill this need and fill it fast.  Enter Stroll Health, an energetic  group ready to solve this gigantic problem. The San Francisco-based startup founded in 2013 by Jordan Epstein, Matt Maurer, and Drew Moxon, has created an iPad app that begins to put real prices for health services in the hands of doctors. They aim to enrich the doctor-patient relationship by allowing doctors to instantly identify the best service for the patient based on actual cost, quality, and location. “In our experience, we’ve seen that tools marketed directly to the patient are always underutilized,” said Epstein, “we believe that bringing the cost transparency discussion to the physician’s office is a big win for doctors and patients.” Doctors are being confronted more frequently with concerns about personal out-of-pocket costs for the services they order and are frustrated by their inability to answer. Unfortunately, costs outside of their offices are as much a mystery to doctors as patients.


              Stroll cofounders, Maurer and Epstein, with OBR volunteer, Laura Sasportas, at OneStart Bootcamp

In tandem with these disruptive technologies, high-deductible plans are growing. These plans make patients responsible for a higher percentage of their health care costs. By increasing the patient’s financial incentive to seek out lower priced and higher quality services the market pressures can correct the lopsided system. Employers are indirectly driving this as they opt to reduce their health care burden by moving to these high-deductable consumer-directed health plans (CDHPs). This 2007 debate provides a good primer on CDHPs. If the consumer is expected to contribute more, they should have access to pricing. Dr Nathan Handley, physician at UCSF, “While it certainly makes sense that the patients who are most interested in discussing cost would be the ones who clamor most for Stroll, so far in my experience nearly every non-HMO patient I’ve talked to wishes they were informed of these choices with their physicians and had the ability to be engaged meaningfully with their physician—they just don’t know this opportunity is available.”

This recent trend further shows the relevance of Steven Brill’s article for TIME magazine, Bitter Pill: Why Medical Bills are Killing Us, which conveys the impact of opaque pricing on the individual. Everyone gets sick and worrying about the potential and unpredictable financial burden of an illness is not conducive to getting well fast.

Stroll recently won the first Prebacked hackathon and competed as finalists in the OneStart Americas competition. The data in the beta version is drawn from samples from insurance companies and Medicare. Currently they are seeking additional partnerships with insurance companies and physicians willing to conduct pilot tests.   

(Thank you to Charlie Carbery and Jordan Epstein for their comments and insights.)

CRO Event: What contract research means to science and biotech

Images by Li Tai Fang

Contract research organizations (CROs) are becoming an integral part of the biotech research and development landscape, especially as big pharma continues to restructure its model for early stage discovery (January 16thpost), and everyone is taking this exploding market seriously.  However, the more we hear “CRO” the more confused some of us are as to what constitutes a CRO and how CROs fit into the research-sphere.  On Monday, January 27th, the Oxbridge Biotech Roundtable (OBR) Bay Area Chapter hosted “CROs: Partners and Disruptors in Drug Discovery” at the University of California, San Francisco to clarify some of this confusion.

An audience of nearly 200 registered attendees gathered to actively interact with a panel of CRO experts including Ken Meek, director of sales and marketing at Aragen Bioscience; Joe Francisco, Senior Client Services Scientist at Charles River Laboratories; Elizabeth Iorns, cofounder and CEO of Science Exchange; Jesse McGreivy, Chief Medical Officer at Pharmacyclics; and Kitty Yale, Senior Director at Gilead Sciences, Inc. The panel began by summarizing CROs as any entity willing to perform research for a second party for pay. The research includes everything from basic discovery to clinical trials. The panelists emphasized the importance of CROs to the biotech industry at all levels, but particularly startups and early stage ventures. David Rabuka, founder of Redwood Bioscience, did not attend the event but commented prior to his talk at the BioScience Forum event later in the week that CROs are essential to minimizing research costs and keeping startups viable. For the remainder of the organized discussion, panel members fielded questions from attendees including what are the drivers for the growth of CROs, of which increased regulatory challenges topped the list, and how is the perception of PhD scientists accepting jobs with CROs improving as CROs become more common.

As outsourcing increases, established CROs are growing to meet rising demand. Charles River, founded in 1947 on the shores of the Charles River in Boston as a provider of lab animals, has grown to locations in 14 countries, 25 locations in the US alone, offering services from toxicology to pre-clinical services. In addition, larger companies looking to expand their service portfolio are rapidly acquiring smaller providers. Aragen’s 50 or so employees are about to reach a larger market as Aragen’s acquisition by GVK BIO, an India-based CRO, was announced on January 29th. One attendee, perhaps with entrepreneurship on his mind, inquired what is missing in the enormous, but potentially crowded, CRO space? All the panelists agreed that providers of extremely specialized services are difficult to find. This is where the highly specialized research of academic labs becomes a commodity and Science Exchange is working to fill this need. Science Exchange puts consumers in touch with registered contract research providers. According to CEO Dr Iorns, several of the registered providers are academic labs looking to raise money through offering their niche services. Academic labs are realizing this is a perfect platform for fundraising to compensate for the deficits created by reductions in traditional funding. Furthermore, acting as a CRO avoids the complications surrounding crowdsourcing the biological sciences.

After the panel discussion, Sajith Wickramasekara, founder of Benchling, commented, “The increasing use of CROs by researchers highlights a greater trend in life science: an industry that’s embracing collaboration and specialization.” With greater collaboration comes the need for more formal data organization and sharing between team members. Benchling, which provides a cloud-based forum for analyzing and sharing DNA sequences and molecular biology tools is one of many startups capitalizing on these needs.

During the post-panel networking session, attendees, including a diverse mix of graduate students, postdoctoral fellows, and industry scientists, enjoyed food and drinks while making new connections. Benjamin Gaub, a PhD student at the University of California, Berkeley working to restore vision in blind animals through retinal reanimation as well as developing echolocation devices for the blind, appreciated the time as an opportunity to, “expand my network and get some exposure to industry.” It may be fair to assert that by brining people together for these events, OBR is providing a CRO-like service for networking. 

For more information on upcoming OBR events visit our site here

The extinction of big pharma discovery research

Historically, a PhD biologist leaving academia for industry was expected to send their resume to the large pharmaceutical companies and wait for the storied labs of Merck or Pfizer to decide they needed an expert in their niche field. Today this option is going the way of the dinosaurs as big pharma opts to outsource their discovery research to smaller, external labs and to reallocate their research dollars to innovation hubs. These centers, strategically positioned in technology hotbeds like Silicon Valley, Boston, and Shanghai, seek to identify research, both early- and late-stage, for licensing or acquisition. Most recently Merck announced their intention to followed Pfizer, Johnson & Johnson, and GlaxoSmithKline down this path as covered in my January 10th Oxbridge Biotech Roundtable Newsflash here.

Instead of lamenting the loss of the behemoth pharmaceutical research machine, we should rejoice in the shifting focus to smaller startups that offer PhD scientists more research freedom and allow young scientists to take an active role in the future of the organization; making them more than a pipetting, replaceable cog tethered to a lab bench. However, this paradigm shift to entrepreneurship does come with the disadvantages of lower salaries and less job security, but the opportunities to grow laterally and to find research that engenders passion by choosing from the diverse startup landscape compensates for these losses. Of course the issue of low funding for discovery stage biotech entrepreneurs remains, but perhaps continued change at the top of the foodchain will lead to increased support for early-stage ideas. It is an exciting time in biotech and an exciting time to be a scientist. 

Bacteria: from gut to lab to disease


B. fragilis, National Review of Medicine

Bacteria are not forgotten like the fungi but they are somewhat neglected when it comes to prioritizing research dollars and drug development efforts. It is perplexing considering that bacteria are often in the mass media with tales of flesh eating infections and antibiotic resistant killer infections, but very little is invested in basic bacteriology research and antibiotic development. As this image from the group Extending the Cure shows there are almost no new antibiotics coming to market despite the Infectious Diseases Society of America’s 10x’20 initiative (10 new antibiotics by 2020). 


Fortunately the UK is taking action to rectify this situation and has announced the creation of the MRC Centre for Molecular Bacteriology and Infection, which I discuss in the current OBR News of the Week. This follows up the Newsflash from last week when the UK announced the formation of the Manchester Fungal Infection Group (MFIG). These two new institutes put the UK at the forefront of action in the infectious disease world. What’s especially cool about the MRC Centre for Molecular Bacteriology is that its focus is on the basic science of bacteria to identify new antibiotic targets and mechanisms instead of only high throughput drug screening. The Centre is investing in cutting-edge technology that leads to some really amazing eye candy that is also useful such as the 3D imaging of bacterial colonization over the entire course of infection. 

Research similar to an August 18th online letter in Nature from Caltech on the identification of a set of genes necessary and sufficient for specificity and stability of some gut bacteria would be the stuff of the MRC Centre. I review the findings of this paper briefly in the OBR News of the Week, but basic research of this type that provides a better understanding of the organism itself will give us loads of targets for drugs, and allow us to create drugs that we actually understand their mechanisms. If we could manipulate bacterial stability in the gut we could battle conditions from obesity to autism. This probably deserves a few dollar bills.