Carlisle* in 1811 notes there used to be a chapel near the Yalden and Pemberton Quarries:

At the Quarries, near the Village of The Mumbles, several Lime Kilns, on an improved plan, have been erected by Messrs Yalden and Pemberton...In digging the stone in these Quarries, several Human bones of large size were discovered: Tradition says, that a Chapel formerly stood near this spot, but no vestiges of it are now to be seen: it is, nevertheless, not improbable but that there might have been one, as the Sea has made great encroachments on this part of Swansea Bay...

However, it isn't clear where these quarries were. The 1st Edition OS maps (1854-1901) show the closest lime kilns to the coast at Colt's Hill Quarry, but Toft* reports that these opened between 1840 and 1850, after Carlisle wrote. Toft's map of kilns operating in 1879* suggests there was a kiln in this area, and it was probably therefore at Callencroft Quarries, just west of the Colt's Hill Quarries.

More importantly, there are the remains of what appear to be lime kilns built into the carpark near Knab Rock. This location for the quarries/church matches up with a tradition noted in a leaflet from the current Oystermouth Church, All Saints (said to have been founded by Illtud, but 13th C., built on a Roman Villa)*:

Old folk in the 1800s recalled a tradition of there being a chapel on the cliff at Knab Rock and quarrymen were later to cut through its cemetery.

Knab Rock (The Knab) has now been largely built over by a small harbour/pier. It can just be seen in the outer corner of the harbour wall in this Photo. There is actually a quarry ~550m northeast along the coast from the rock (where the "T" of "The Mumbles" is on this Map). This is shown on the first OS map of the area (County Series, 1:10560, 1849-1899). However, more notably, Gabb (p.28)* notes that the rock arch at Knab Rock "was much threatened by quarrying" - the associated image showing the cliffline as coming well up to the current coast and suggesting a good 30m of quarrying has occurred at the location itself.

Knab rock would obviously make a perfect spot for our wonder, fitting in as it does with both the expected location and being close to at least one possible cave ("Bob's Cave": See Note). However, there is little evidence here excepting local belief and memory, so it is hard to say how likely this location really is.