Thursday, May 07, 2020

The very model of a modern scientific man

Your humble Devil was thoroughly amused by Neil Ferguson's fall from grace, and is very pleased to have found the time to outline Ferguson's manifest and repeated failings—for posterity, you understand.

And it seems that the media scrutiny of Ferguson's sordid personal life has put the wind up some of the government's scientific advisers...
One scientific adviser to the government said Ferguson’s resignation had created “an awful lot of concern” and that the mood in the community was “very depressed”. The events revealed how university academics who lent their advice to government were having to cope with an increasingly difficult situation, the adviser added.

“He’s an academic researcher. He doesn’t make decisions. He’s not paid for any of this. We are being drawn into a political situation which is very unpleasant,” they said.
Oh dear, what a pity—how sad. Hey, science bods—you know you might avoid this kind of media scrutiny? Yes, that's right: don't take political appointee jobs and refuse the fat taxpayer-funded salaries. You don't want to be caught up in politics? Then don't play at politics.

Are these people simple, or what?

In the meantime, a programmer has finally got around to looking at the Imperial College modelling code—and her assessment is not pretty. [Emphasis mine—DK]
I wrote software for 30 years. I worked at Google between 2006 and 2014, where I was a senior software engineer working on Maps, Gmail and account security. I spent the last five years at a US/UK firm where I designed the company’s database product, amongst other jobs and projects. I was also an independent consultant for a couple of years.
So, a reasonably credible source then. I wonder what she found? Let's cite some choice extracts from the assessment, shall we?
Clearly, Imperial are too embarrassed by the state of it ever to release it [the original model code] of their own free will, which is unacceptable given that it was paid for by the taxpayer and belongs to them.

Due to bugs, the code can produce very different results given identical inputs. They routinely act as if this is unimportant.

[...]

Investigation reveals the truth: the code produces critically different results, even for identical starting seeds and parameters.

I’ll illustrate with a few bugs. In issue 116 a UK “red team” at Edinburgh University reports that they tried to use a mode that stores data tables in a more efficient format for faster loading, and discovered – to their surprise – that the resulting predictions varied by around 80,000 deaths after 80 days...

[...]

Because their code is so deeply riddled with similar bugs and they struggled so much to fix them that they got into the habit of simply averaging the results of multiple runs to cover it up… and eventually this behaviour became normalised within the team.

[...]

Although the academic on those threads isn’t Neil Ferguson, he is well aware that the code is filled with bugs that create random results.

[...]

Imperial are trying to have their cake and eat it. Reports of random results are dismissed with responses like “that’s not a problem, just run it a lot of times and take the average”, but at the same time, they’re fixing such bugs when they find them. They know their code can’t withstand scrutiny, so they hid it until professionals had a chance to fix it, but the damage from over a decade of amateur hobby programming is so extensive that even Microsoft were unable to make it run right.

[...]

The Imperial code doesn’t seem to have working regression tests. They tried, but the extent of the random behaviour in their code left them defeated.

[...]

Much of the code consists of formulas for which no purpose is given. John Carmack (a legendary video-game programmer) surmised that some of the code might have been automatically translated from FORTRAN some years ago.

[...]

This code appears to be trying to calculate R0 for “places”. Hotels are excluded during this pass, without explanation.

[...]

R0 is both an input to and an output of these models, and is routinely adjusted for different environments and situations. Models that consume their own outputs as inputs is problem well known to the private sector – it can lead to rapid divergence and incorrect prediction.

[...]

Despite being aware of the severe problems in their code that they “haven’t had time” to fix, the Imperial team continue to add new features; for instance, the model attempts to simulate the impact of digital contact tracing apps.

Adding new features to a codebase with this many quality problems will just compound them and make them worse.
Yikes. And the conclusion...?
All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.
Well, I think that this fucking debacle goes some way to explaining why Ferguson's models have been such a complete and utter failure from the get-go.

And let us remind ourselves that our government were stupid enough to believe this fucking team of charlatans, and that they are busy cratering the economy on the strength of a computer "model" (I use the term advisedly) that produces complete garbage.
On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.
Indeed they do.

It's odd though. There's something at the back of my head, something niggling at me—a real sense of familiarity about this situation...

Where ales have we encountered a commentary on a computer model that has huge political and economic consequences but which, having been written by a bunch of amateur fuck-wits, provides absolute fucking garbage...?

Oh yes—it's Harry again.

Do you remember, in November 2009, that there was a leak from the University of East Anglia's Climate Research Unit (CRU)? Most of the media spent their time exposing the dirty tricks revealed in the emails between the "scientists" who are the main proponents of the Catastrophic Anthropogenic Climate Change (CACC) theory—including using collusion and blackmail to prevent dissident papers appearing in "reputable" journals.

But what was less widely reported was that, along with the emails, the computer "models" (again, advisedly) were released—alongside a very long commentary by an unfortunate programmer who was tasked with making sense of them.

The programmer was called Ian Harris, and his HARRY_READ_ME.txt file was, for those of us who like to delve into these things, an absolute treasure trove—revealing the incompetence of these so-called "scientists", and the utter invalidity of their much-vaunted "climate models".

And here we are again: with a government fucking our economy and freedoms, all on the basis of useless, garbage-spouting models.

Dear Boris (and every other government): for fuck's sake, stop giving any credence at all to these models. Models are not evidence and they are not science: even the most well-coded model would be nothing more than a theory—and, as we have seen with both COVID-19 and CACC, the people building said programmes are nowhere near being competent.

These so-called scientists are not: they are hobbyist coders (and bad ones at that). And where they attempt to sell their models as reliable, these people are frauds—and they should be prosecuted. If not, then a class-action lawsuit might find a large number of backers—especially if a case carries the prospect of personally bankrupting Neil Ferguson. Certainly, I would happily donate.

UPDATE: Tim Almond explains why "it's stochastic" is no excuse at all, and the Streetwise Professor is as incensed as your humble Devil...

8 comments:

Tom said...

It’s policy-based evidence-making and the customers don’t care if the code’s defective as long as the “science” justifies their seizing more of our money and taking more control over our lives.

Bloke in North Dorset said...

The events revealed how university academics who lent their advice to government were having to cope with an increasingly difficult situation, the adviser added.

“He’s an academic researcher. He doesn’t make decisions. He’s not paid for any of this. We are being drawn into a political situation which is very unpleasant,” they said.


I don't remember any concern from the Guardian about Roger Scruton, who also wasn't paid. Wasn't Toby Young unpaid as well?

Bloke in North Dorset said...

PS

A very close cousin died at the start of the lockdown (cancer). His funeral was attended by his wife, daughter, son-in-law and their newborn baby. The rest of us watched via a poor mobile phone zoom connection. I still haven't found the right word to describe the added emptiness and sadness that caused. His wife didn't have the comfort of the presence of his extended family and friends to ease her grieve and I think she'll be scarred by that for the rest of her life.

We're not alone. The lockdown may have been needed and I am conscious of hindsight bias so I won't hold that against anyone, making those decisions in a fast moving situation is difficult and their are no right answers, but I for one will never forgive the hypocrisy.

Anonymous said...

I agree that epidemic modelling, like climate modelling and economic modelling, should be treated with extreme scepticism until models start producing accurate predictions. However, this is not a fair and charitable appraisal of the code. It's not obvious at all that non-deterministic output from multithreaded code would be worthless. Obviously it's desirable for regression tests, but may be hard to achieve, due to the need for synchronisation between threads.

It's not surprising that the results for numerical calculations would change depending on the processor architecture due to the amount of precision used and how rounding is done. This is not a flaw in the code. The article blithely says there were "no tests", but this is clearly not true: only that the tests do not work on all architectures or when run in multi-threaded mode.

As an aside, I was annoyed at how the author says that "senior, experienced engineers" would be needed to get multithreaded code to work, makes sweeping comments about what is considered "poor practice", malign "amateur hobby programmers", and talks up "working software engineers" and "professionals". Yet, they rail against "rampant credentialism" in academia.

microdave said...

From The Sunday Express on April 26th:

"SCIENTISTS working to steer Britain out of the coronavirus crisis fear they could be blamed for the Government's failings once the pandemic is over"

Professor Hugh Pennington said "following the science" was meaningless for politicians because researchers have already reached different conclusions about Covid-19. He said: "The politicians who are making these measures on lockdown do not have a scientific background which is very unfortunate because they cannot interrogate the science. But in a way they are shielding themselves with science because they cannot get blamed if things go wrong."

This was echoed by Professor Robert Dingwall, a Government adviser on managing the pandemic.

He said: "The public should be sceptical of any claim by ministers to have 'followed the science'". "Science is flimsy, riddled with doubt, contradictions and uncertainty and always changing."

Ed P said...

I'll happily help fund a class-action.

It's not just the economic damage wrought by politicians believing these incompetent people with their wonky software, there's also the not insignificant matter of how many their "advice" has killed. So a corporate manslaughter charge might succeed.

It'll take seeing Ferguson's head on a spike on London Bridge, possibly Twitty's too, before I'm happy again.

Anomalous Cowshed said...

"Science is flimsy, riddled with doubt, contradictions and uncertainty and always changing."

But only some types of "science", obvs.

Anomalous Cowshed said...

Whilst I'm here;

"John Carmack (a legendary video-game programmer) surmised that some of the code might have been automatically translated from FORTRAN some years ago."

To my mind, this is possibly the worst thing about this, as the implication is that someone didn't understand the code in FORTRAN, so what are the chances they understood the resulting C? So for "some years", there's a bloody good chance that no-one fully understood what it was doing.

NHS Fail Wail

I think that we can all agree that the UK's response to coronavirus has been somewhat lacking. In fact, many people asserted that our de...