Walmer Castle and its beautiful formal gardens are a major attraction for visitors. The official residence of Lord Warden of the Cinque Ports since the 18th Century, the building is now an English Heritage property. Famous Lord Wardens have included Queen Elizabeth the Queen Mother, Sir Winston Churchill, William Pitt the Younger (whose niece Lady Hester Stanhope first created the gardens), and the Duke of Wellington (of the Battle of Waterloo fame). Wellington lived there for 23 years and the castle houses not only a collection of Wellington memorabilia but also the room in which he died.

The Castle was built in 1540 as one of three on this part of the Kent coast by orders of Henry VIII.