John Phillips, 39, built the four chalets in his garden in Gower, South Wales, in 2022, renting them out on Airbnb for up to £185 per night.
But he did not have the correct planning permission for the pods at Landimore, which overlook the Wales Coast Path and the Loughor Estuary.
Swansea City Council rejected his retrospective planning application in July last year and served him with an enforcement notice four months later.
It said he had to stop renting them out, demolish them and clear the land within six months.
John subsequently appealed the refusal decision and enforcement order, but now a government-appointed planning inspector has dismissed both appeals.
So now he finally has to take them down.
But he was given 12 months, instead of the original six, to comply with the order to demolish and restore the site.
The inspector, Richard Duggan, accepted that more time was needed to cancel bookings at the high-demand chalets.
John, who says he plans to take the case to court, called the order “madness.”
He said: “It’s really unfair – it’s one rule for us and another for everyone else.
“They (the municipality) have just given permission to a house in our village to build a ten-room Airbnb.
John, who rents out the chalets with his wife Kerrie Garrett, 34, and lives on site with daughter Darcy, four, says he doesn’t know what they will do if they have to bulldoze the holiday homes.
“If this happens, we won’t be able to pay our bills,” he said.
“It pays our living expenses – it’s our full-time job.
“We did not look at the price for its demolition. We will fight this every step of the way.
“We have hired a planning consultant and are currently working on an appeal.”
John moved to the property three years ago, but feels his family is treated differently because they are not local.
“We are from Swansea and they don’t like anyone from outside the Gower coming in,” he said.
“It seems there is a hidden revenge campaign against us.
“They’re all people you know here.”
Planning inspector Richard Duggan said the chalets’ terraces and hot tubs were visible from the coastal path.
He said: “The view from the public footpath, albeit only about 100 metres, is valuable as part of the experience of walking within the AONB and the development is seriously damaging that view.”
Swansea City Council said: “We welcome the planning inspector’s decision to support the previous enforcement action taken by the council in relation to this unauthorized development.
“Hopefully the outcome of this case will send a clear message that anyone considering development in the city should ensure they do so with the correct planning permission.
“This is particularly important in terms of development on Gower due to its protected status where additional planning policies are in place to ensure development is suitable for Gower and in the right location.”