Of the 50 or so caves on the Gower, a number have traces of Romano-British occupation, and a few have traces of Dark Age inhabitation. Stella Elphick* notes, largely on the basis of finds at Swansea Museum, the following caves have indications of Romano-British or Dark Age use:

Culver Hole: Finds include "Romano-British coins and a brooch and brass ring dated from the Dark Ages".
Spritsail Tor Cave [Prissens Tor]: Excavations revealed Roman pottery and human remains*, possibly suggesting a mix of uses, including occupation.
Minchin/Mitchin Hole: Excavations between 1946 and 1959 found four heaths from the first five centuries CE, severn bronze brooches, plus over 750 pieces of domestic ware including a series of fine bone spoons*. The conclusion of the then Curator of Archæology at Swansea Museum, J.G.Rutter, was that the hole was used by locals in the Dark Ages at times of trouble*.
Bacon Hole: "occupied during both the Iron Age and Romano-British eras".

Of these, the nearest to Oystermouth is Bacon Hole (Info), though it's not especially close. Two other caves are closer to Oystermouth: a fissure cave on Mumble's Hill which was found to contain human remains that have not been dated* (Details), and "Bob's Cave" on Mumbles' Head (Photo). Chris Elphick* quotes a local legend (here) about two men sailing up to a monk living in the cave and dumping a murdered corpse on him. This is plainly very suggestive, but in the absence of a source it is hard to say any more than that about it. In addition, local rumour has it that there used to be a cave behind All Saints Church (See Note) in which a saint lived, however, what happened to the cave is not clear and this is possibly a distortion of the Wonder description. Local legend also All Saints was set up by Illtud, however all current remains suggest the church is 13th C. and was built on the site of a Roman villa.