It’s unlikely that removing the driver card will ever be an issue unless it raises suspicion that you’re trying to cover up an offence, however it is technically an offence to needlessly withdraw your driver card without permission during a shift where you keep the same vehicle.
Clearly if it’s not practical to get back to the vehicle to change modes ejecting the card is the only practical thing to do.