There have been few more toxic failures of politics in recent years than the bitter, emotive and often bad faith exchanges that have come to characterise public discussion of trans rights.

Lives have been consumed, careers and reputations trashed, and good people exhausted by experiencing politics at its worst – which is to say politics used less as a democratic means for resolving difficult conflicts, and more as a weapon. Terrified of saying the wrong thing, well-intentioned public figures have fallen silent. Progressives in particular have all too often come across as nervous and confused, thrown by the concept of an equal rights issue in which both sides can justifiably claim to be vulnerable and fearful of violence.

It’s been painfully obvious for a while now that the Labour party has been struggling with its position, particularly since the furore over whether Isla Bryson – a Scottish transgender woman who raped two women before she transitioned – should serve her sentence in a men’s or women’s jail, a question that sent the firmly pro-trans rights Scottish National party into a tailspin. This week Anneliese Dodds, the shadow equalities minister charged with the thankless task of finessing all this, formally confirmed the result: a U-turn on Keir Starmer’s promise to introduce self-identification for trans people, only two years after he apparently committed to it. What’s left is a nuanced, practical, middle-of-the-road Labour position with something to infuriate activists at both poles of the argument. For millions in between, however, the prevailing emotion may be one of cautious relief.

Like it or loathe it, at least everyone now knows what an incoming government would do. Labour frontbenchers now have something concrete to say, instead of unhappily ducking “gotcha” questions about whether a woman can have a penis. And Starmer has finally jumped off a splintering fence, while characteristically managing to land not quite on either side of it.

This isn’t what trans rights activists wanted or what they were promised, and many on the Corbynite left will take it as fresh dispiriting evidence of the party deserting them. Starmer won’t, after all, rip up the medicalised process for trans people who want a certificate legally recognising their identity: he’ll merely reform it, so a formal diagnosis of gender dysphoria from one doctor (rather than an expert panel as at present) will do. In other words, nobody would be able to declare themselves a man or a woman simply by signing a legally binding document – a position that puts UK Labour at odds with the Scottish Labour party, which does support a demedicalised process.

But gender critical feminists don’t get all they wanted, either. Labour will sensibly keep the Equality Act provisions that allow single-sex spaces like domestic violence refuges or rape counselling services to exclude the opposite sex, where that’s a proportionate means to legitimate ends – essentially, when you couldn’t reasonably provide the service otherwise. (Though Labour’s quid pro quo should be cast-iron commitments to properly funded bespoke services for trans people excluded from any mainstream service).

Practically speaking, as Dodds says, that means there will always be places “where it is reasonable for biological women only to have access”. But still, some gender critical campaigners worry that the law isn’t clear enough, with service providers afraid of being called transphobic if they try to use it: there will be pressure for Labour to be more specific about who can compete in women’s sports, say, or how schools should handle trans-identifying pupils, or NHS treatment for trans teenagers. Perhaps Labour is quietly hoping such awkward questions will be resolved by someone else before the next election.

Nor is Dodds offering what many women on the left who have been vilified and ostracised for supporting what now turns out to be the officially sanctioned position might have wanted to hear, which is an overt statement that Labour initially got it wrong – the implied message of any U-turn.

These, then, are the people this announcement won’t please. Beyond them, however, is a large, exhausted swathe of public opinion looking for that elusive middle ground: a place where both trans people and vulnerable women and girls are protected, clashing rights are thoughtfully balanced, and there is no angry vacuum left for the far right or other bad actors to exploit.

Labour’s new position marks the beginning, not the end, of a long road: there is much detail still to be thrashed out, and some immensely difficult choices to be made by any future government anxious to take public opinion with them on trans rights. But to accuse a sensitive and conscientious politician like Dodds of surrendering to a “transphobic framing of the debate” – code for accepting that a minority of trans women may potentially pose a threat in female-only spaces, or that some women might have genuine reasons for not wanting to share intimate space – is not only unfair, but fails to recognise complex realities.

Of course it’s transphobic to make sweeping assumptions about trans people, and of course cases like that of Bryson or Karen White (the trans female rapist convicted of sexually assaulting fellow inmates in a women’s prison) are rare. But just because something is rare, that doesn’t mean policy can ignore it; just because a question is ideologically inconvenient, that doesn’t mean it can’t be asked. The ultimate reward for going through this painful process is that in a decade’s time the nation hopefully looks back – as we did with gay marriage, and countless other LGBT reforms – and wonders why extending trans rights ever seemed so controversial.

Gaby Hinsliff is a Guardian columnist

Do you have an opinion on the issues raised in this article? If you would like to submit a response of up to 300 words by email to be considered for publication in our letters section, please click here.