Hazelwood began as three 19th-century country estates on what was then a forested Clark's Point. The park takes its name from the estate of Joseph Congdon, who called his property "Hazelwood" and laid out an English garden around the Gothic Revival stone house he built in 1839–40 — a house that still stands near the center of the park today.

By the late 1890s the city was buying land for public parks as development spread south, and it acquired the Hazelwood property in 1902. To shape it, New Bedford hired landscape architect Warren Manning, who had trained under Frederick Law Olmsted — the designer behind New York's Central Park.

Three of the park's buildings were added to the National Register of Historic Places in June 2010, making Hazelwood one of the few places in the city where you can play a set of tennis surrounded by preserved 19th-century architecture.