Interestingly enough, the 'silent Tories' were a feature of every election in recent history, with the notable exception of 2017.
I don't think people are scared to say they're voting for them any more. It's entirely acceptable now to be an openly racist bigot, where previously you'd have kept quiet about it.