Miller was born in Baltimore, Maryland, the oldest of nine children of George W. and Harriet J. Miller. Millers father was a merchant and tavern keeper in central Baltimore, and also had a farm in Hawkins Point. Miller attended a private school in Baltimore, John D. Craigs Academy, but did not receive formal art instruction there. He may have received his first lessons in art from Thomas Sully.
In 1832, with the financial support of his family and art patrons in Baltimore, Miller traveled to Paris to study art. He was admitted as an auditor to life drawing classes at the École des Beaux-Arts, and copied paintings in the collections of the Louvre. In 1833, he traveled to Italy, visiting Bologna, Florence, and Venice before settling in Rome, where he studies at the English Life School. During his travels in Europe, he became friends with the Danish sculptor Bertel Thorwaldsen and studied with the French painter Horace Vernet.