The fact that you want to withdraw the card on other work suggests that you will do some work after ejecting the card, if that’s the case then legally you’re required to do manual entries for any work done after you ejected the card in the last shift worked and any work done before inserting the card in the current shift, on newer tachographs that pretty much means you’ll have to manually record something for the period between the two shifts.
If you’re not doing any significant work after the card is ejected in the last shift or before inserting the card in the current shift there’s no legal reason to do manual entries regardless of what mode you leave the tachograph on, but if that’s the case and it bothers you why not put it on rest before ejecting your driver card.
If you’re not doing any work between ejecting the card and reinserting it and you manually recording the rest period is for some reason upsetting the companies system then don’t do a manual entry for the rest period.
By the way, for the working time regulations holidays count as 8 hours for individual days but 48 hours for whole weeks Monday to Sunday.