Of the 50 or so caves on the Gower, a number have traces of Romano-British occupation, and a few have traces of Dark Age inhabitation. Stella Elphick* notes, largely on the basis of finds at Swansea Museum, the following caves have indications of Romano-British or Dark Age use:

Culver Hole: Finds include "Romano-British coins and a brooch and brass ring dated from the Dark Ages".
Spritsail Tor Cave [Prissens Tor]: Excavations revealed Roman pottery and human remains*, possibly suggesting a mix of uses, including occupation.
Minchin/Mitchin Hole: Excavations between 1946 and 1959 found four heaths from the first five centuries CE, severn bronze brooches, plus over 750 pieces of domestic ware including a series of fine bone spoons*. The conclusion of the then Curator of Archæology at Swansea Museum, J.G.Rutter, was that the hole was used by locals in the Dark Ages at times of trouble*.
Bacon Hole: "occupied during both the Iron Age and Romano-British eras".

Of these, the nearest to Oystermouth is Bacon Hole (Info), though it's not especially close. Two other caves are closer to Oystermouth: a fissure cave on Mumble's Hill which was found to contain human remains that have not been dated* (Details), and "Bob's Cave" on Mumbles' Head (Photo). Holland (p.90)* gives the following legend from "A Book of South Wales" (1861) which probably refers to Bob's Cave:

There is a legend that where the lighthouse now stands a holy monk, or a succession of holy monks, had charge of a small cell or chapel, tributary to one of the religious houses; and the legend tells of an aged monk who after sunset was telling his beads and looking across the waters to the opposite shore, when he perceived a boat rowing inwards. He watched it with the interest a lonely man always feels in the approach of fellow men, and seeing that it made direct for the small Mumble rock, he descended to the shore to give it welcome.

The rowers drew in, and a man of grave aspect stepped on shore and gave the monk a sign, which he understood. He then caused a body to be brought up the path to a cave under the monk's oratory. The body was bravely dressed, like that of a man of high degree, and his still features were white as chiselled marble. The monk looking on him could not help saying, "So young and handsome!" He was laid in the cave, and money was deposited with the monk for masses to be said for the repose of the soul. The boat rowed away, and the holy monk was faithful to his trust, and said double the usual quantity of "massess"; but to this day it is believed that the spirit of the poor murdered man cries out from that cave for Christian burial in consecrated ground."

This is plainly very suggestive (and mysterious: what was the sign? why introduce murder?), but it is hard to say any more than that about it. In addition, local rumour has it that there used to be a cave behind All Saints Church (See Note) in which a saint lived, however, what happened to the cave is not clear and this is possibly a distortion of the Wonder description. Local legend also has All Saints as being set up by Illtud, however all current remains suggest the church is 13th C. and was built on the site of a Roman villa.