Scarborough was born a slave in 1852 in Macon, GA. After study at Atlanta University, he earned his B.A. and M.A. from Oberlin College. His Greek textbook (1881) drew national attention. He was a member of the American Philological Association for 44 years, and presented many papers. He was the first black member of the Modern Language Association (1884) and was president of Wilberforce University (1908-1920).