The last I remembered, the station operated as a scallop house in the early 1960s, and then was abandoned altogether. As kids, we use to play around the station. Unfortunately it was always securely locked, so we could not get inside. Once a nice looking building in it's time, a donation by Mrs. Russel Sage, it was torned down in the spring of 1966 to make room for an ugly drive up bank that still stands today. Something that could never happen in today's time with landmark preservation.
For the full story about the Sag Harbor Railroad branch click here to visit: Arrt's Arrchive who has the complete history of the LIRR.
