Building sites are probably some of the worst places to start - I know, I started there too doing precast concrete deliveries. Everyone’s an expert, they could all drive the truck (yeah right) and they are all in a rush to get to the pub basically.
Guy in office is supposed to be there as buffer between you and clients, but sounds like he just not suited or is an arse.
Think you made the right decision and kudos to the guy for giving you a shot, but would apply for more on the road driving jobs to get your confidence.
No one cares if you squash a few BMWs. 