It Does Not Work Well

Hannah Sharland writes:

A report on the South London scheme that put job advisers into NHS hospitals is stacked with statistical biases. That being the case, it means that the positive findings from it are largely bogus. Of course, this happens to be the very same programme Department for Work and Pensions (DWP) boss Liz Kendall has lauded for its employment “results” and “dramatic” mental health outcomes.

As it turns out then, the existing scheme Kendall has been parading as a pretext for rolling out job advisers for patients with “serious” mental health problems is not the resounding success the DWP secretary has quite made it out to be. Crucially, if anything, it pours cold water on the egregious notion that work is always good for people’s mental health.

Of course, it’s not really surprising – since the Canary and others have repeatedly warned DWP-NHS integration like this is a dangerous and flawed idea. However, this bunk evidence should be another nail in the coffin of Kendall’s alarming project to put job advisers into more healthcare settings.

DWP mental health ward infiltration

On Wednesday 16 October, the BBC published a puff piece on Kendall’s plans to post job advisers in hospitals.

Initially, the BBC gave over the entire article to Kendall, in what it boasted as an “exclusive interview” with the DWP boss. Presumably due to rightful backlash against the scheme and odious piece of fawning stenography, the Canary noticed it has since updated this. Now, the article carries quotes from disability charities challenging it.

This included Scope’s executive director of strategy James Taylor, who underscored that:

We need to see evidence that work coaches being sent to visit seriously ill people works, and doesn’t cause distress.

Well, the Canary went in search of this so-called “evidence” for the DWP mental health plans. And let’s just say to start that it’s not exactly solid proof for Kendall’s claim to the BBC that:

the results of getting people into work have been dramatic, and the evidence clearly shows that it is better for their mental health. 

Kendall and the DWP are basing this on a programme that South London and Maudsley NHS Foundation Trust have had in place.

So, we dug out an evaluation of the Work Well scheme there. Researchers from mental health organisation the McPin Foundation carried out the independent analysis of the scheme. It’s from 2020, was updated in 2022, and reviewed three years of the programme from 2017 onwards. Obviously, it’s likely this is – at least in part – the ‘results’ Kendall was crooning over. However, they’re hardly anything to write home about. In particular, it found that out of 551 participants:

28% (152) found employment by the time the project ended.
25% (140) were in paid employment when it finished.
10% (57) had stayed in this job for at least six months.

Far from the “dramatic” results Kendall claimed then.

Sly statistical selection bias

By contrast, in terms of mental health outcomes, ostensibly, the programme did help. That is, until you scratch a little under the surface. This was where some sly statistical gymnastics came in to tell the story the programme clearly wanted to project.

The report measured participants’ anxiety and depression over three separate periods in time. Notably, it found from the start of employment they both decreased at each time point. The evaluation isn’t completely clear here. It says it assessed people at three months on and then further at six, nine, and twelve months. Of course, it gives only two extra time points after the baseline, yet it doesn’t clarify which of these points in time they actually were.

Furthermore, there was a ‘but’, and a significant one at that. Specifically, the evaluation applied this to only the 57 participants who were still in work at six months or more. In other words, it didn’t assess the mental health of nearly 500 participants on the scheme. These were people it said had “disengaged” or left the project altogether.

What we don’t know then, is why those people dropped out or drew back their engagement with it. And of course, it’s very plausible that some of these people did so because the scheme – and work itself – was making their mental health worse.

Basing results solely on the people who maintained employment beyond six months in itself is clear selection bias.

DWP mental health plans: ignoring people who dropped out

The report goes to little effort to find out why a) some people didn’t stay in employment, and b) the impact the programme had on their mental health. It simply noted that some had resigned, others had their contracts come to an end, a few were dismissed, and one hadn’t passed a training test.

Nor did it really assess the mental health of those that participated in the scheme, but didn’t start a job, or dropped out altogether. It’s blatant selection bias to only look at the people staying in their jobs for six months or more – not least because it’s people with less severe mental health problems who are more likely to be able to work anyway.

On top of this, the report conducted in-depth interviews with some participants. From these emerged a swathe of glowing praise for the programme.

However, confirmation bias struck here again. For starters, out of a pool of 551 participants, it did these with only eleven of those. Once again, these were evidently among those who’d stayed engaged with the programme. At the time of the interviews, five of these were volunteering, three working, and one was studying part-time.

Here too then, it ignored the fact that people who’d maintained contact with the scheme were more likely to have had positive experiences with it. It didn’t appear to interview people who’d dropped out of it, or attempt to find their reasons for doing so. The report also solicited the views of eight Work Well staff, and six stakeholders from organisations that referred or took people on from the project. Unsurprisingly, they too largely hailed the programme.

Individual Placement and Support (IPS): more dodgy ‘evidence’

Besides all these issues, the background to the report made the bold claim that:

Individual Placement and Support (IPS) models for people with mental health problems are well evidenced

Despite the brazen assertion, it doesn’t actually reference any of the said evidence for this. Much further down the report it states:

Individual Placement and Support (IPS) models in the UK have achieved modest outcomes for people with severe mental health problems (Howard et al, 2010), where 13% of IPS clients retained competitive employment after one-year follow-up. In another UK study, 35% of people who saw an employment specialist within a mental health team after one year follow up (Marwaha et al, 2014).

So, that would be, barely over a tenth in one instance, and little over a third in another one in employment a year after participating in the scheme. Of course, the year on review doesn’t exactly show long-term retention. Aside from that, these studies are now over a decade old. None of this exactly screams “well evidenced”. 

Yet funnily enough, this is exactly the guff claim its own creators have made about it too. The Canary’s Steve Topple has pointed out the problems with this before, highlighting that:

The creators of IPS claim it’s evidence-based. However, one study found IPS only got people into work for a limited time. After six years, there was not a significant gap in employment outcomes between IPS and people who had standard psychiatric treatment.

Additionally, he underscored another more recent study that found:

Moreover, the success of IPS in the UK is questionable, too. Between January 2016 and March 2019:

31% of IPS participants started a job.

22% kept that job for at least six weeks.

12% kept that job for at least six months.

Hardly a resounding success. 

Most significantly though, these employment statistics tell us absolutely nothing about the mental health impacts of IPS on participants. Why? Because the studies haven’t measured it. This is, of course once again, precisely the problem.

Care not job advisers in hospitals

Ultimately then, the report doesn’t actually give us an accurate picture of participant’s experiences of the programme. So, Kendall’s claim that “the results have been dramatic” is quite a stretch at best. At worst, it’s wilful manipulation of reality.

This is because, while this evaluation might be flawed, as we previously pointed out, other DWP reports HAVE shown the devastating impacts of linking work as a health outcome. And the picture they’ve painted has been one of harm – of actively making people sicker – and putting their lives at risk.

The simple fact is, people accessing mental health services in hospitals categorically do not need DWP job advisers sticking their noses into their care.

The government is pushing these DWP mental health plans, knowing full well it’s evidence for doing so is self-fulfilling bullshit. It’s hard to see it as anything less than the eugenicist urges of a DWP hell-bent on coercing mental health patients into work.

However, people living with serious mental health problems do not need career advice. They need compassion and care. But that’s the two things that Kendall, Starmer and company hardly have a single fibre of between the lot of them.