Question For Agency Drivers

I’d not think just this scheme on agency will get the tax man after you , I’m agency paye & get letters on a regular basis telling me I owe x,y or z