Crowds paid tribute, dancing and singing in front of Mr Mandela's former home in Soweto throughout the night.
Flags flew at half mast after President Jacob Zuma announced his death in a late night national TV address.
Mr Mandela spent 27 years in jail before becoming South Africa's first black president in 1994.
His administration replaced the racist white-minority regime that had enforced segregation of black and white people in a policy known as apartheid.
Mr Mandela went on to become one of the world's most respected statesmen.
A service of national mourning is expected to be held at a 95,000-seater stadium on the outskirts of Johannesburg on Monday. His body will then lie in state for three days in the capital, Pretoria, before being taken for a state funeral in the village of Qunu in the Eastern Cape, where he grew up.
"God was so good to us in South Africa by giving us Nelson Mandela to be our president at a crucial moment in our history," said long-time ally Archbishop Emeritus Desmond Tutu.
Mr Mandela had been suffering from a lung illness for a long time.
He had been receiving treatment at home since September, when he was discharged from hospital.
As soon as the news broke, small crowds began to gather in Soweto's Vilakazi Street, where Mr Mandela lived in the 1940s and 1950s.
They chanted apartheid-era songs, including one with the lyrics: "We have not seen Mandela in the place where he is, in the place where he is kept."
By daybreak, dozens more had gathered.
"We are celebrating his life and all that he did for us," said one of the mourners, Terry Mokoena.
Crowds also gathered outside Mr Mandela's current home, in Johannesburg's northern suburb of Houghton, where he died.
Across the world, leaders, celebrities and members of the public have been paying tribute.
Pope Francis said he had forged "a new South Africa built on the firm foundations of non-violence, reconciliation and truth.
"I pray that the late president's example will inspire generations of South Africans to put justice and the common good at the forefront of their political aspirations," he said.
Queen Elizabeth II said she was "deeply saddened" to learn of Mr Mandela's death.
"He worked tirelessly for the good of his country, and his legacy is the peaceful South Africa we see today," a statement issued by Buckingham Palace said.