3 MILLION EU WORKERS IN THE UK - WHERE IS THE FAIRNESS?

If 2 million went back home that would equalise the number of EU workers living here and UK people (who are not necessarily workers) living an ALL the countries of the EU (which in itself is unfair, as there is no balance of people between the UK and each of the other countries).  What we have is an unregulated free for all, and the UK is the big loser.  Why!!!  Because if the 2 million EU workers left, we would immediately solve the housing crisis and the NHS funding crisis. So what do we need the EU workers for, if not to make money for a few people?