I think the problem is more Airbnb/short-term rentals than rich outsiders. Those golden visa people are buying €500K+ properties that are out of the reach of average income locals. It is the Airbnbs that are taking potential rentals and more affordable homes off the market.