Ryder was born in New Bedford, Massachusetts. New Bedford, a bustling whaling port during the 19th century, had an intimate connection with the sea that probably supplied artistic inspiration for Ryder later in life. He was the youngest of four sons; little else is known of his childhood. He began to paint landscapes while in New Bedford.
The Ryder family moved to New York City in 1867 or 1868 to join Ryders elder brother, who had opened a successful restaurant. His brother also managed the Hotel Albert, which became a Greenwich Village landmark. Ryder took his meals at this hostelry for many years, but it was named for the original owner, Albert Rosenbaum, not the painter.