The simple way around it is when challanged as to why you are not wearing a mask, you declare that you are medically exempt. They are not allowed to query your medical issue. And you do not need a lepers bell around your neck.
I played this card at a British Sugar site in the first lockdown. The lad ran off to squeel to his manager and he came back later with tail between legs mumbling that all was fine and he was not allowed to ask what was wrong with me! It all seemed a bit daft at the time as we were working outdoors within our small team of installers and were segregated from any British Sugar worker.