You’re told to swap and wait with a broken down vehicle obviously as part of a reasonable planning decision and instruction by the management.Don’t see the problem regardless of whether you’re agency or not.
As to whether agency v permanent matters any more I’ve recently walked away from two non agency self employed car delivery/collection job offers.Which also required the ‘driver’ to work as an unpaid car sales executive personally and financially responsible for successful retail customer handovers.Or personally and financially responsible for end of lease condition assessment and subsequent negotiation with lease customer.Obviously acting as an unqualified under paid intermediary,between lease company and customer in that regard,all for the massive sum of £28.That’s after being expected to make my own way between ‘delivery’/‘collection’ jobs fair enough,oh wait either walking or using public transport.Whichever option is deemed most ‘cost effective’ by the firm and no payment will be made assuming the latter.
Then to add insult to injury replied to an advert for an employed permanent garage service delivery/collection ‘driver’ at around minimum wage.To be told that,in addition to the yard marshalling duties,there’s probably no room for negotiation for a job share arrangement with another semi retired person,regarding the 8 am - 6 pm Monday - Friday + Saturday mornings hours.Oh wait and my back issues might be a problem regarding some of the ‘other’ possible unstated/unknown ‘duties’ as required. 
I’d guess the lesson is don’t burn your bridges and pick your arguments with employers wisely. 