Our knowledge of language only goes back to the first writing. At the moment, archaeologists think the earliest writing systems were developed more or less at the same time, about 5,000 years ago, by people in the countries that are now Iraq and Egypt. The languages of this oldest writing were Sumerian (probably) and ancient Egyptian.
But humans developed language maybe 100,000 years before anything was written down. Because many languages have changed in regular and predictable ways, some scientists try to learn about prehistoric languages by working backwards. It's more than guesswork, but without finding a caveman, their results are hard to prove.
George, Andrew. 2014. “Ask a grown-up: what was the first language?”. The Guardian. Posted: March 8, 2014. Available online: http://www.theguardian.com/lifeandstyle/2014/mar/08/ask-grownup-what-was-first-language