I’ve just had the exact same issue. I asked how much noticed I needed to give and he said 4 week. I left after three. I have to pay back £125 in owed holidays which I’m not too bothered about because the new job is better pay and it was worth my while leaving early. At the end of the day, it’s every man for himself, obviously your previous boss won’t see it like that, mine hasn’t. But you’ll be forgotten about the week after you leave. Do what’s right for you and your family if you have one.