Small was born in Edinburgh into a working class family. His father was a bookbinder and his mother a straw hat maker. He trained at the Royal Scottish Academy schools, and by the age of 17 was a professional wood engraver. At the age of 22 he moved to London where he became a practicing artist and soon after began exhibiting his paintings and drawings at the Royal Academy and the Royal Watercolour Society. He was and elected member of the Royal Institute of Painters in Watercolours and Royal Scottish Academy. His usual subject matter for paintings was tragic or romanticised images of working class people or farming communities.
Small was also a graphic artist and book illustrator, producing images for various periodicals including the ‘Cornhill Magazine’, ‘Harpers’, ‘The Graphic’ and ‘Good Words’, and also books of poetry, children’s songs and stories. As a graphic artist, Small produced drawings with strong lines and compositional features which contrasted with a much looser and hazy style that he used for his painting.