Of course it would not go well. Does not change the fact that they deserve (though probably would not get, at least in meaningful quantity, since they are the majority) for demanding you conform to their sensitivities when it was them who decided to come to your place. Your point is a practical one, an argument of "advisable/inadvisable". "Right/wrong" on the other hand is a domain of moral philosophy. Something may be the right thing to do but still be a bad idea, like assassinating a dictator, or a good idea despite being wrong, like the tax avoidance schemes of pretty much every major company.
Granted, the particular situation of deliverypeople is a bit more complex in that they are there because you are requesting a service, so it is not that they are unsolicited visitors who deserve no courtesy. Does not change the fact that pandering to the easily-offended on principle is a slippery slope that leads to less, rather than more free expression, but one could make a case for trying to arrange a no-contact delivery or at least giving advance warning if you just cannot be arsed to dress.