Mark Campbell, from Jubb Consulting Engineers, said rock falls are difficult to predict: “It could happen in days or it could happen in years.”
The Wales Coast Path has reopened since the first rock fall as it was deemed to be a safe distance away from the area.
But Ian Williams, who owns the land and the nearby Oxwich Bay Hotel, said it was vital people did not venture too close to the site.
He is concerned some people might try to climb the boulders, which have been described as being the size of houses.
“It’s spectacular and people will want to come and look at it, but the message is please don’t go into it,” he said.
“We’re advised that more could move, and if they did, people wouldn’t get out in time. It’s as simple as that.”
The first rock fall happened on 6 January, followed by another about three weeks later.
Experts say the area is at particular risk because the bedrock, which supports the hard limestone cliffs above, is tilting downwards at an angle towards the beach.
Mr Campbell said “tens of thousands of tonnes” of boulders had fallen.
“There’s nothing that can be done to stabilise it, it’s such a big rock fall. It’s just a matter of keeping away.”
Steven Matthews, the coastguard’s senior operations officer in Swansea, said he was stunned by the size of the rock falls.
“The UK does suffer with a lot of coastal erosion, but this scale rock fall is the largest that I’ve seen,” he added.
“I’ve also been informed it’s one of the largest in the UK in recent years.
“If people are coming to use the beaches and the coastline this summer and this spring then it’s vital people stay away from the cliff edges.”