Think agency job covering weekends or whenever you are free from your day job might be the way to go. Try to marry your holiday up with one of the really busy periods for HGV drivers. Offer your services for that period and suddenly your new pass wont be as relevant. But I have seen people test the water of their new career purely as weekend agency drivers.
I have to be honest and say I have zero experience of HGV agencies but tons of dealings with agencies in my old career as an electrician. The great thing is siezing control away from the agency. As soon as they “need” you then you take control by telling them quite clearly when yuo are available and what rate you wont work under…That certainly worked for me when i used to do agency work as a spark. Maybe it’s different in driving, but the scenario is the same, they need you as much as you need them.