Riveting read on the decline and fall of science:
Something has gone very wrong in the basic research complex that we do have in the universities and in the federal funding system. We know something has gone very wrong because of the replication crisis. John Ioannidis at Stanford wrote this classic paper where he said “50% of published research is non-reproducible” which means you basically can’t reproduce the same results meaning you can’t do anything with the research, it’s basically fake. Interestingly he was studying biomedical research, which is an area of research you think would be very focused on getting things right. It subsequently turned out he might have been underestimating the problem. Fields like biomedicine might be as fake as 70%.
Anyway, science right now is in an existential crisis. This is a real, real issue, and there’s now a generation of scientists who specialize in pointing this out and analyzing it. Andrew Gelman and others. I’ll give you an example: I had a conversation with the long-time head of one of the big federal funding agencies for healthcare research who is also a very accomplished entrepreneur, and I said, “do you really think it’s true that 50-70% of biomedical research is fake?” This is a guy who has spent his life in this world. And he said “oh no, that’s not true at all. It’s 90%.” [Richard laughs]. I was like “holy shit,” I was flabbergasted that it could be 90%.
He’s like “well look, 90% of everything is shit”, which is literally this thing called Sturgeon’s law which says that 90% of everything is bad. 90% of every novel written is bad, 90% of music, 90% of art… 90% of everything is bad. So his analysis was, anything you get in the field of medical experimentation, biomedical development, and this is going to be true of any field, there’s like five labs total in the world that are really good at what they’re doing...
But once you get out of those top five labs, it’s pretty much make-work, incremental, marginal improvements at best, and a complete waste of time otherwise. And I said “good God, why does the other 90% continue to get funded if you know this?” And he said, “well, there are all these universities and professors who have tenure, there are all these journals, there are all these systems and people have been promised lifetime employment.”
**
There is a real argument that there are just a certain number of super-elite people. There are a certain number of people who are going to be really good scientists and it’s just not going to be that many. It’s some magical combination of intelligence, honesty, industriousness, integrity, the ability to recruit and build a team. In some ways being a top researcher is like being an entrepreneur, you have to actually pull all these different kinds of dilemmas together.
And then of course, the implication of that from a societal standpoint is that we’ve really got to know who those people are and we’ve really got to give them room to run. We’ve really got to make sure they have room to run and are not driven out.
**
So we’ve put these games in place because we thought we were systematizing ways of achieving excellence. If you go back and you read the people in the 1920s, people like Vannevar Bush, FDR’s science advisor, who basically codified modern university research in the 40s. He thought he was creating an optimized system for maximum output of high-quality research. And it’s an incentives problem. To your point, and everyone in business knows this, every time you define a metric immediately people will start gaming the metric. So if you tell me as a researcher in a university I need to start publishing journal articles, immediately I’m going to start publishing journal articles. Now, will anybody read them? Will anybody cite them? Will anybody care? Do they actually move the field ahead? Maybe, maybe not. So I think it’s possible we as a society spent 50 years assembling these systems and these games in the first half the 20th century, and then we spent the last half of the 20th century gaming the games.
No comments:
Post a Comment