Ed Yong demands higher accountability in science journalism and has made me think of how in the last two days I’ve run across two examples of shoddy reporting. These two articles I think encompass a large part of the problem, the first from the NYT, represents the common failure of science reporters to be critical of correlative results. While lacking egregious factual errors, in accepting the authors’ conclusions without vetting the results of the actual paper, the journalist has created a misleading article. The second, from Forbes, represents the worst kind of corporate news hackery, and shows the pathetic gullibility of reporters regurgitating the fanciful nonsense of drug companies without any apparent attempt to vet or fact-check their story. With a google search the facts are smashed.
The first article Digital records may not cut costs, I think is typical of most science reporting. That is, it’s not grossly incompetent but it overstates the case of the article involved and fails to amplify the shortcomings of the research.
The NYT article is describing this article from Health Affairs, which caught my eye before the NYT article was even published because I believe electronic medical records (EMRs) will prevent redundancies and lower costs. So, am I wrong? Will EMRs save us money or possibly increase redundancy as the HA article suggests?
I haven’t given up hope. This article is a correlative study based on survey data, and proves precisely nothing.
We analyzed data from the 2008 National Ambulatory Medical Care Survey, a survey of 28,741 patient visits to a nationally representative sample of the offices of 1,187 nonfederal physicians.19 The survey excludes hospital outpatient departments and offices of radiologists, anesthesiologists, and pathologists.
…
The survey collects information about the practice setting, including detailed information about computerization, as well as about the characteristics of the patients seen and the tests ordered at each surveyed visit.
…
To assess whether computerized access to imaging results reduced the ordering of imaging, we separately analyzed predictors of whether a patient received a computed tomography scan; magnetic resonance imaging; any advanced imaging procedure (computed tomography scan, magnetic resonance imaging, or positron emission tomography scan); or any image (an advanced image, X-ray, bone density measurement, ultrasound, or other image).
We examined two indicators of physicians’ access to imaging results. The first was whether the practice had what the survey called “a computerized system for viewing imaging results”–that is, a system that presents a text report of a physician’s interpretation of the imaging study, an actual visual electronic radiologic image, or both. The second was, for those practices with such a system, whether “electronic images [were] returned”–that is, whether in addition to or in place of a text report, the actual visual images were returned electronically.
In those few cases where physicians indicated that they had such a system but its capability was “turned off,” we considered that they did not have access to imaging results.
So rather than testing two cohorts of physicians that are equivalent except for access to patient EMR, or even better, introducing EMRs to physicians offices with or without access to outside providers tests and studying ordering habits, they are just mining survey data. Worse, the comparison is between physicians with potentially a huge variety of systems, not necessarily compatible with other systems, not necessarily connected to every place a patient may get imaging or lab results. After all, there is a huge heterogeneity of EMRs, they are not compatible with one another. There is no reason to think if some imaging is available, all imaging will be available to the physician.
One of the most frustrating things as a doctor is when you’re evaluating a new patient transferred from somewhere else, going through their chart, and trying to figure out everything the previous hospital did. Inevitably the discharge summary is buried somewhere in a pile of thousands of nursing notes, computer-generated order histories, and outdated labs. Then after you’ve finally found the summary and are working out the real history of what’s happened to patient you run across the imaging. Invariably, the imaging was from a few hours ago and showed something so terrible they had to be transferred to your quaternary care center immediately – before the final read from their radiologist could be included in the chart. You pop the disc in the computer, and…nothing happens. Cursing you take the film downstairs to the radiologist and she tells you, “oh, that format is incompatible with our system.” Frustrated, you call the hospital that sent the patient, and you find out their medical records department will open Monday at 9am (it’s Saturday at 2AM) and you are left to make a surgical decision. You now have a sick patient, an incomplete record, and no imaging and no chance to get the information that you need that may result in a surgical decision for the next day and a half. All you have is the unconfirmed word that the patient has a big abdominal problem, and you have to make a big decision on the notoriously inaccurate report of the transferring physician (guess what, smaller hospitals dump inconvenient patients on higher-level centers all the time – sad but true).
What do you do? Well, you re-image the patient. This means additional delay, additional cost, additional radiation to the patient. Sound horrible? It is. And this example is just one end of the spectrum from the outpatient physician who rather than wait for records in transfer just runs a new CBC and chem7, to the other extreme which is that in transfer most hospitals do not accept any of the tests from the outside hospital and re-run everything, including imaging.
Now our hospital has an EMR, and the ability to access some other EMRs, but it in no way guarantees compatibility or access to records from hospitals outside our network.
This is why this study isn’t helpful. The potential of the EMR to reduce redundancy isn’t in the mere presence of a hospital or office having some kind of electronic records system. It will come when there is a top-down regulatory framework imposed which will require record compatibility between systems. The problem is, again, the market. Every electronic medical record software company insists on proprietary record formats. They are not readily transferable as that might prevent their disgusting attempts to garner market share at the cost of patients’ health and efficient use of medical resources. If I were dictator, I’d just force every hospital to use the CPRS system from the VA, which while clunky, is comprehensive, fully functional, secure, and compatible across every VA health system. Instead we’ve bought the privatization cool-aide and rely on one of a dozen different companies, all vying against each other and ensuring none of their software talks to anyone else. This is the worst possible situation (as are many in US health care). A compromise would be to allow any company to generate software but have an agreed upon format that is universal, like the formatting code for dvds or cds. You can have a Sony player or a Pioneer, but they all can read the same damn disc.
So, did the NYT article emphasize any of these flaws within the article? Yes, with a throwaway paragraph from a dissenter at the bottom:
Dr. David J. Brailer, who was the national coordinator for health information technology in the administration of George W. Bush, said he was unconvinced by the study’s conclusions because they were based on a correlation in the data and were not the result of a controlled test.
The study did not explore why physicians in computerized offices ordered more tests. Dr. McCormick speculated that digital technology might simply make ordering tests easier.
Now, Brailer brings up a good point. This study proves exactly nothing. Instead it shows a correlation based on what I believe are highly flawed assumptions, to disparage EMRs’ potential. But does that stop the NYT from making the first paragraph basically swallow this flawed message? Nope:
Computerized patient records are unlikely to cut health care costs and may actually encourage doctors to order expensive tests more often, a study published on Monday concludes.
This study shows no such thing. I can understand saying “there is a correlation between doctors with electronic records and more test ordering”, but nothing about increasing costs as it says nothing about EMR compatibility, function, or whether the doctors are even able to access other providers records. Also, centers with more technology are likely to result in more use of technology, including advanced imaging and the correlation might just be due to patients present at more technologically advanced centers. I somehow doubt your country doctor is going to start ordering all sorts of new tests because someone hands him a laptop. And the authors themselves note that their results do not exclude self-referral, the practice you may remember I criticized yesterday creates incentives for physicians to over test.
This is an example of a reporting that I think is just mediocre. It reports results without much emphasis on the weaknesses of the correlative nature of the study and overstates the significance of the results. Yes there is a throwaway paragraph from a dissenter for balance, but if anything a paper that shows a correlative effect, contrary to numerous other analyses showing the opposite effect, needs a higher level of scrutiny and skepticism.
Now, the Forbes article, “The Truly Staggering Cost Of Inventing New Drugs”, on the other hand, is a truly egregious example of a reporter falling hook-line-and-sinker for some drug company propaganda. Mathew Herper writes:
During the Super Bowl, a representative of the pharmaceutical company Eli Lilly posted the on the company’s corporate blog that the average cost of bringing a new drug to market is $1.3 billion, a price that would buy 371 Super Bowl ads, 16 million official NFL footballs, two pro football stadiums, pay of almost all NFL football players, and every seat in every NFL stadium for six weeks in a row. This is, of course, ludicrous.
The average drug developed by a major pharmaceutical company costs at least $4 billion, and it can be as much as $11 billion.
1.3 billion was obviously bullshit, and just based on a guess I would say it overshoots by 10-fold, but that’s a pretty typical corporate overstatement of their trials and tribulations. Herper wants to increase the BS to a 100-fold exaggeration. He does this by the highly dubious process of dividing their entire R&D budget by the number of “new” drugs the companies produce in a given year.
Rebecca Warburton and Donald Light at Plos expose the flaws in this argument:
Firstly, the estimates in Forbes accept company R&D figures uncritically and ignore evidence that what companies count as “R&D” may be broader than the costs of bench, lab, and trial research that make up R&D. Drug companies work hard to hide their real costs from any outside scrutiny. And they never link their alleged costs to how quickly they earn them back at high prices.
Secondly, the estimates in Forbes divide total reported costs by the number of “new drugs.” Given the small number used in the Forbes estimates, for example only 5 in 14 years for AstraZeneca, “new drugs” must mean NMEs or new active ingredients. The big companies turn out many more newly patented variations on existing drugs that involve less risk, time and cost. In other words, the Forbes estimates divide total R&D for research on all products by the handful of NMEs. However, the me-too variations are the main products of R&D, and they account for about 60 percent of the United States’ drug budget.
Sure, if you shrink the denominator to 1/5th of what it should be, your numbers are going to jump up. For every “new” drug, 8 or 9 “me too” drugs are developed. I checked the data on approvals here and just fact checking the drug maker at the top of the list, Astra Zeneca, quoted as producing 5 drugs from 1997-2011 in the Forbes article, looks like it produced closer to 15 over that time period. Glaxo, quoted as making 10 drugs, made closer to 40. This “me-too” effect has been known since Marcia Angell exposed the myth of drug company R&D almost a decade ago. Further, development of new compounds and investigation of novel pharmaceuticals is often in NIH-sponsored labs who then sell the rights to a drug company to develop. Drug companies get a more reliable return on medications that are copy-cats than on taking risks on potential dead-end new development research.
Oh, and the drug companies are screwing with the numerator too:
Finally, half of the industry’s average cost of R&D is not real R&D costs at all, but an estimate of profits foregone – a highly inflated estimate of what companies would have made had they put their money in an index fund and not developed new drugs in the first place! Given the staggering cost estimates in Forbes, you might think that drug companies should do just that and become investment banks.
So what is the real cost? Warburton and Light explain:
Our own estimate of pharmaceutical R&D is often misquoted as an average of $43 million per new drug, which commentators reject as being absurd. However, we make clear this estimate for the year 2000 does not include the cost of discovery (because it varies greatly and no one has accurate figures), nor the “cost of capital” (for reasons explained in our article which can be read here). Our estimate is the net cost to major companies after taxpayers cover about 50% of their R&D expenses. We use the median cost because the average cost gets inflated by a few costly R&D projects.
In sum, we estimate that the median, net, corporate cost to develop a new drug, based on the confidential cost data that companies reported to their policy research center at Tufts University, is $56 million in 2011, plus the unknown company costs of discovery and the artificial estimate of profits foregone, if you think it should be added. We also show that R&D costs for in-house new active ingredients are much higher, and costs for me-too variations are much lower than this single figure.
Our estimate is almost double the only solid corporate report of R&D costs, which can be found in audited tax returns from the late 1990s. Here companies reported average costs for clinical trials per drug of only $22.4 million. Not $224 million but $22.4 million
…
if the Forbes figures and business arguments are correct, then nearly all of the global pharmaceutical companies listed in their article would have gone bankrupt between 1997 and 2011.
The NYT article merits a sigh, the Forbes article merits a trip to the whipping shed. In a single google search you find he’s not just missing the point, his reporting fails at basic journalism.
McCormick, D., Bor, D., Woolhandler, S., & Himmelstein, D. (2012). Giving Office-Based Physicians Electronic Access To Patients’ Prior Imaging And Lab Results Did Not Deter Ordering Of Tests Health Affairs, 31 (3), 488-496 DOI: 10.1377/hlthaff.2011.0876
Leave a Reply