It’s a bit more than that. If there wasn’t a shortage of manual workers - we’d not have immigrants unchecked coming into this country. If we didn’t have immigration - we’d have a labour shortage, since the top jobs have had it cushy over the years, leaving manual worker’s wages behind. Now you’ve got 3 bedroomed semis out of reach of anyone earning less than Cameron in far too many areas, not all in the South of England even.
There’s empty properties in some areas - but no jobs there.
Work puts up house prices, house prices draw in more immigrants, immigrants keep the wages down, low wages keep working people in the same place for longer. The same people in the same areas for long enough eventually bring prosperity - but not for around another generation as of yet.
Ask yourself this question: "What were you doing for work, rest, & play 10, 20, 30, and 40 years ago respectively…
10 years ago I was working 5-6-5-6, avoiding voting for Blair, and earning pretty well.
20 years ago I was happy to have a job, as there was a recession on. I was in the right place for that time.
30 years ago I was at college doing Chemistry, Physics, and Computing - but totally skint.
40 years ago I got badly sunburned in the summer, was practically eating ladybirds every step I took that same summer, and was in the final class of Junior school with a great maths teacher who got 27 of us out of a class of 36 through the 11-plus… Happy days. 
The relevance is that you think times are hard when you’re living through them, and then look back on the good old days when things get even worse in the future… What makes us feel “Worse off”? - Relative poverty. If the people around you get to do, buy, and go places you still can’t afford - then that pretty much guarantees that you’re not going to be happy with your lot…
Life is 90% luck and 10% working towards something. Of the luck, most of it is just good or bad timing that decides what kind of luck you get. Property buyers in the mid-80’s did well. Buyers in 1990 got killed. Buyers in 1995 did well… and so on… cycles.
This round of austerity has lasted a bit too long though, because this time around it’s touched far too many industries, and far too few of the business services sector… 