Its a delicate situation you find yourself in, you need to be very tactful because other than being on the books you don’t have any sort of actual status over this guy by the sounds of things.
I would be asking “why do I need to sharpen up” and that then opens up a conversation about driving styles… But if you don’t feel comfortable riding with him, tell the gaffer that, and if he/she has got anything about them, they will assess him and tell him they’ve had a complaint from a member of the public.