I don’t think what has happened is right, but bear in mind it is a £19 card and a new one normally comes quite quickly in the post if you were ever to decide to leave. The employment law specialist may tell you it is wrong but doing something about it may be another matter entirely.
It isn’t worth even losing half a day of pay over, so bite the bullet I would say. I would return the card myself and provide an online driving summary to prove it has been done. You have agreed not to do any other driving, so the only inconvenience is ordering a new card and the time it takes to arrive if you ever leave.