You’re right. The thing is that almost all immigrants, even the low-skilled ones, represent a net positive contribution to the national income and/or fill jobs natives don’t want to do (eg seasonal work in farming). Immigration is not really a ‘problem’ of supply (out of work benefits are not generous in the UK) but of demand for labour, and the government has failed to scale up public services, despite the extra tax and NI revenue from the immigrants.
A points style immigration system post-Brexit was rejected early on, but true to type, the government remains vague about exactly how that control would happen (some spread the malicious rumour that the government doesn’t have a clue).
In practice, the free movement does actually mean lots of movement in and out. People like me, foreign residents for 25+ years, are rare – I think more than 1/4 remain for less an a year. But it is also a big benefit to employers: there is no need for work permits, visas etc – they can flexibly recruit Greek physicists, Portuguese nurses, Finnish engineers or Romanian fruit pickers without any paperwork. A system of control would make that much more difficult and costly.
The free movement that is a condition of the Single Market doesn’t really affect movement of economic refugees (I wouldn’t call any intra-EU migrants refugees in any meaningful sense). Most migration of people of working age, on the other hand, is economically inspired – the #1 reason for moving to another EU country is work-related.