Soon the small Welsh fortress at Tenby was captured, probably by peaceful penetration. For the next few centuries Tenby's safety and prosperity were firmly linked with that of the Earls of Pembroke. By the middle of the 12th century some form of primitive castle had been built on the Castle Hill and garrisoned by English and French troops. Built by Gerald de Windsor, or a little later by Gilbert de Clare - first Earl of Pembroke(1138-1148), it was designed to protect the Norman's eastern flank from attack by Welsh chieftans.

Along with tyhe network of castles and other defensive structures theNormans built, it was hoped that the Tenby fortifications would provide a period of stability to enable them to establish their manorial system. It would allow them to settle French, English and Flemish immigrants whom they imported in numbers

In 1151 members of the Tenby garrison wounded Cadell ap Grufydd, a Welsh nobleman, hunting in the woods near Saundersfoot. Two years later, in 1153, his brothers Maredudd and Rhys carried out a daring night time revenge assault and captured the town. Having punished the men responsible for the injuries to their brother, they handed the castle back to their cousin, William Fitzgerald, who was acting constable for the Earl of Pembroke. In 1187the town was attacked and ransacked by the marauding welsh, under Maelgwyn ap Rhys. In 1260 it suffered its last battering at Welsh hands when Llewelyn ap Gruffydd put the town to the sword in protest against the Norman occupation of Wales.