Witnesses said the destruction began on Friday.
"The militants broke the doors and wooden gates to the grave and then used the cloth inside to set fire to the tomb," said Sankoum Sissoko of Timbuktu. "They started chasing people away, telling them they shouldn't worship a saint, a human being."
The World Heritage Committee on Thursday accepted the request of Mali's government to place Timbuktu on the list of endangered UNESCO sites. That request was reportedly what angered the militants, who accused UNESCO of cooperating with the government.
France, the former colonial power in Mali, also condemned the attack, calling for an end to the violence.
Mali was plunged into chaos by a military coup in March that ousted former President Amadou Toumani Toure. Since then, the Tuareg rebels and the Islamists have taken advantage of the uncertainty to attempt to seize control over the northern portion of the nation.