The "Castle on the Hill" was a gift of love to a hometown. Henry Huttleston Rogers a Fairhaven native who rose to become a Standard Oil magnate — donated the school to the town in 1906, on the 50th anniversary of his own graduation from Fairhaven's original wooden high school. Ground was broken in 1905, and Rogers spared no expense: he consulted the era's top educators and hired renowned architect Charles Brigham to create what he called an "educational palace."

The result was astonishing for its time — marble floors, oak paneling, stained glass, carved woodwork, its own electric generating plant, and even the first indoor basketball court in an American high school. Its castle-like towers and ornate stonework earned it the nickname it still carries today.

It was the last public building Rogers gave to Fairhaven before his death in 1909 — and in 1981, it was added to the National Register of Historic Places, cementing its place as one of the town's crown jewels.