Historically, Ganesha is one of the latest Hindu gods that arrived Tamilnadu. In 642 AD, when the Narasimhavarman the first of Pallavas conquered Vatapi, his army took many artefacts from Vatapi including a Ganesha statue. Later Vatapi Ganapathy became a term for Lord Ganesha. This could be the first arrival of Lord Ganesha to Tamilnadu, which is in line with the literature.
In sangam literature (600BCE - 300 CE), there is no mention of lord Ganesha and "Vinayagar Agaval" was written much later in the 10th century AD (during the bhakti literature era; Avvai of the sangam age should not be confused with the later Avvai who wrote this and Athichoodi). Hence Vinayaka would have arrived somewhere between the 3rd and 10th centuries. Why did Vinayaka the first god to be worshipped in a temple arrive much later than all other Hindu gods?
Not just in Tamil, Lord Ganesha has appeared much later even in the Hindu texts. In Rig Veda, the term Ganapati is mentioned twice, when praising Brihaspati and when praising Indra. Many consider this as a direct reference to Lord Ganesh, however in both places Ganapati (lord of Ganas) is used as a praising title rather than referring to the elephant-faced god. The first mention of an elephant-faced god appeared in later Upanishads (period 100 to 300AD); It is noteworthy that this happened centuries after the widespread of Buddhism.
The legend which states the Mahabharata was dictated by Vyasa and Ganesha wrote it down, is not widely accepted as the original story as it appeared only in some later versions of Mahabharata written after the 9th Century AD.
Kangiten, is an elephant-faced god in Japanese Buddhism, which is considered to be derived from Vinayaka from Hinduism. More than 250 shrines in Japan has Kangiten statue and also in some Buddhist temples throughout south-east Asia. One might wonder, if Vinayaka appeared after the Buddhism era in India, how it could have a Buddhist version. The reason is that Buddhism in Asian countries didn't evolve from a single branching out from the Indian culture. Throughout the medieval age, many Chinese scholars visited India to update their scriptures from the Indian literature and many Indian religious scholars travelled to China. During the 7th and 8th centuries, series of Indian monks travelled to China to spread tantric Buddhism and in the early 9th century, a Japanese monk named Kukai travelled to China and obtained the teachings of tantric Buddhism. He started the school of Shingon Buddhism which is one of the few surviving tantric Buddhist lineages. And the Kangiten is a god in these schools of Buddhism which evolved after the 9th Century in Japan. Hence the Vinayaga reached China and eventually Japan around the same time he reached Tamilnadu.
It is important to establish the fact that Ganesha/Vinayaka appeared much later than Buddha, to unveil a significant connection between them, "Arasa Maram" (aka bodhi tree). Buddha attained his enlightenment under a bodhi tree and in Buddhism, it is considered as the central place for worship. Interestingly, even today "Arasa Maram" is a centre of worship in Tamilnadu but instead of Buddha, it is Ganesha who is being worshipped almost under every "Bodhi tree". Until the 10th Century, Buddhism and Jainism were the major religions in Tamilnadu, and the culture of worshipping the bodhi tree and circling it around for any problems is retained by Tamils. However, there is no relevance for Ganesha to this particular cultural practice except for the reason that Vinayaka was used to replace Buddha from the popular culture. The name Vinayaka itself is considered to be derived from the term Vihar Nayaka (Buddha's title).
Once a prominent religion, Buddhism is entirely wiped out from the Indian and Tamil society not by discarding or conversion, rather by engulfing the entire cultural practices and dissolving the remembrance by providing a parallel and eventual substitution.
Unfortunately, the story didn't end there. Since Vinayaka is the replacement of buddha, he is anti-caste in nature. Anyone can have the idol; anyone can do poojas to him; anyone can touch him because anyway, eventually he will be dissolved in a water body, the classical ritual to get rid of a sin caused by touching an untouchable.
It does not matter whether it is an idol of Buddha or Ganesha if it is a symbol of anti-caste and helps to embrace equality, it should be celebrated. But it is not just that. Ganesha as a symbol of anti-caste is being used to unite Hindus just to oppose Muslims; to show the Hindu majority by indulging in violence against Muslims during the Visarjan rallies. How can something be anti-caste and divisive at the same time?
If wisdom is to filter out bad from the good, we should embrace the anti-caste nature of Ganesha and kolukatai (Modak) too. Definitely should filter out the rallies. After all, both Buddha and Ganesha are considered lords of wisdom.