Yeah, there are horror stories about them turning up at yards and taking action for a flat tyre that was a slow puncture that had only just occurred/developed that night etc.
There’s a lot to be said for electric security gates and swipe cards, on yards, as at least it puts pay to any surprise visits. Just for the aggro, not because you particularly have anything to hide.
Your yard/premises then yes but if, for example, you were delivering or working somewhere they could only be on there with the landowner’s permission. Not sure what the situation would be with a rented parking space on someone else’s larger property, who wasn’t an operator I suppose they might forego their rights to deny access by renting out for the purpose of an operating centre.