Khanyile was born in Soweto, and was forced to abandon his education at the age of fourteen in order to earn a living, after his mother died. His father was a miner and performed traditional a capella songs, and his brother John played in a reggae and soul covers band. Jabu followed them into music, first joining a local band called The Daffodils, and in 1974 joining John's band The Editions as a drummer, later becoming the group's vocalist. In 1977 he moved on to The Movers, and by 1984 Khanyile had joined Bayete as drummer, a band that combined Afro-jazz and reggae. Bayete split up in 1992, and Khanyile embarked on a solo career, with releases credited to Jabu Khanyile & Bayete, although none of the original members of Bayete were involved by this time. Khanyile became known internationally in 1996 after an appearance at the Royal Gala evening in honour of Nelson Mandela.
In 1996 and 2000 he won the Kora award for best Southern African artist. He performed internationally with Youssou N'Dour, Angelique Kidjo and Papa Wemba. He was known for his Pan-Africanist approach to music, attempting to unite different African styles. He generally performed in a Maasai costume carrying a trademark fly-whisk, an African symbol of royalty.
Khanyile performed at the Live 8 concert in Johannesburg in July 2005. His final public appearance was in July 2006 at the "Africa Calling" handover ceremony at the end of the World Cup in Berlin.
He died in November 2006 after a battle with diabetes and prostate cancer