Brett was born near Reigate on 8 December 1831, the son of an army vet. In 1851 he began lessons in art with James Duffield Harding, a landscape painter. He also studied with Richard Redgrave. In 1853 he entered the Royal Academy schools, but was more interested in the ideas of John Ruskin and William Holman Hunt, whom he met through his friend the poet Coventry Patmore. Inspired by Hunts ideal of scientific landscape painting, Brett visited Switzerland, where he worked on topographical landscapes and came under the further influence of John William Inchbold.
In 1858 Brett exhibited The Stonebreaker, the painting that made his reputation. This depicted a youth smashing stones to create a road-surface, sitting in a brightly lit and brilliantly detailed landscape. (The treatment provided a strong contrast with Henry Walliss painting of the same name, exhibited the same year.) The precision of the geological and botanical detail in Bretts version greatly impressed Ruskin, who praised the painting highly, predicting that Brett would be able to paint a masterpiece if he were to visit the Val dAosta in Italy. Partly funded by Ruskin, Brett made the trip to paint the location, exhibiting it in 1859, again to high praise from Ruskin, who bought the painting. Other critics were less effusive, one describing it as a "gravestone for post-Ruskinism".