I drive a Daf cf with the mirror guards for a builders merchant.
Exact same problem.
You can tighten the nut at the top of the arm and that helps a bit.
Thankfully it’s only the drivers side that seems to be affected especially in a head wind.
Duncan