If we took that approach slavery would have never in a million years have been banned. Interracial marriage, gay marriage, desegregation in America, decolonisation, banning the death penalty, all these things the majority of people were or are against.
Representative democracies exists to make decisions on behalf of the people so that society actually functions.