Sailors' Snug Harbor came into existence through the benevolence of Robert Richard Randall, who, four days before his death on June 5th, 1801, signed a remarkable will which directed that his mansion and 21-acre farm on Manhattan Island (just north of today's Washington Square in Greenwich Village) be used to create a home for "aged, decrepit and worn-out sailors." However, as the 1820s approached and land values began to soar, the legislature was asked to modify the Randall will so that Sailors' Snug Harbor could be built somewhere other than the Randall farm. The trustees of the institution sought a more rural site and in May 1831, a 130-acre farm overlooking Upper New York Bay and the Kill van Kull was purchased on Staten Island. The Manhattan land was leased and that income endows the institution to the present day.

The first three buildings were designed in the Greek Revival style by the young New York City architect, Minard Lafever. Buildings were added over the years until there were 55 major structures. When the first building opened in 1833, Sailors' Snug Harbor was home for 17 aged seamen. By the turn of the century nearly 1,000 men lived there and it had become one of the most famous benevolent institutions in the nation. The Harbor produced its own electricity and steam, grew its own food, and had its own water supply, a church, cemetery, hospital, theater, and library. Snug Harbor flourished during the heyday of New Brighton's suburban development. The old sailors liked being near New York Harbor while the trustees liked having them away from the temptations of the city.

Beginning in 1950, as part of a 'modernization and improvement plan,' two dozen buildings on the Staten Island property were bulldozed. Next on the destruction list were the Sailors' Snug Harbor dormitories which would be replaced by a 120-room modern infirmary insisted upon by the State Department of Health. At this point, the city's new Landmarks Preservation Commission stepped in. On October 14, 1965, at its first designation hearing, the Commission landmarked and saved the old dormitories.

Property for a new institution for the old sailors was found in Sea Level, North Carolina, down the road from a hospital just taken over by Duke University Medical Center. Citing the proximity of Duke's hospital to the new Harbor site, New York's surrogate court approved relocation.

Snug Harbor is now owned by the City of New York and is a cultural and educational center. The 80-acre site, in addition to numerous 19th-century landmarked buildings, contains rolling lawns, ponds and gardens. 2001 marked the 25th anniversary of the Snug Harbor Cultural Center. It features a children's museum, botanical gardens, a maritime museum, a contemporary art gallery, artists studios, an art school, and the New York Chinese Scholar's garden. In the summer months there are open air concerts on Sundays.