He was born and grew up in Duncan Village, a township of the South African city of East London.
In the early 1960s he was a member of South Africa’s first integrated jazz band The Blue Notes, with Mongezi Feza on trumpet, Dudu Pukwana on alto saxophone, Nikele Moyake on tenor saxophone, Chris McGregor on piano, and Louis Moholo on drums. The band fled South Africa in 1964 to seek musical and political freedom.
In 1966, Dyani toured Argentina with Steve Lacy’s quartet. Lacy, Dyani, and Moholo recorded “The Forest and the Zoo”. He later moved to Denmark and Sweden, recording many albums under his own name, often on the Steeplechase label. He recorded with Dollar Brand, Don Cherry, Steve Lacy, David Murray, Joseph Jarman, Clifford Jarvis, Don Moye, Han Bennink, Brotherhood of Breath, Mal Waldron, and many others.
After his death the remaining members of The Blue Notes reunited to record a moving tribute album Blue Notes For Johnny on the label Ogun Records.