Born into a poor family in Aberdeen in Scotland, Phillips artistic talent was recognised at an early age. Lord Panmure paid for Phillip to become the student of Thomas Musgrave Joy in London briefly in 1836. His education at the Royal Academy of Arts was paid for by Panmure.
While at the academy, Phillip became a member of The Clique, a group of aspirant artists organised by Richard Dadd. The Clique identified as followers of William Hogarth and David Wilkie. Phillips own career was to follow that of fellow-Scot Wilkie very closely, beginning with carefully detailed paintings depicting the lives of Scottish crofters. He moved on to much more broadly painted scenes of Spanish life influenced by Bartolomé Esteban Murillo and Diego Velázquez.