The question itself may sound very strange to Twelvers, however, one only needs to simply look critically at the bigger picture of what the Twelve claim is. Apart from the context, there are three main reasons why Ghadir Khumm is not evidence for Twelvers, and we shall expand on these reasons below.
Vagueness of the “Appointment”
It is well known that the term mawla has several meanings, and that due to this, the debate of whether referring to Ali as a mawla meant his appointment/coronation or meant something else altogether has lasted for over a thousand years.
If the Prophet (peace be upon him) had instead said, “Ali is to rule after me,” then the case would have been resolved.
Even though modern Shias argue that the term mawla is explicit evidence for the Imamate of Ali, we find that some of the top Shia scholars of the past did not hold this view. One example of this is Al-Karajiki in Kitab Al-Ibana (Mujalat Turathina – 85th Edition p. 328), who states, “The appointment through statements of the Imamah of the Commander of the Faithful (as) falls under two categories: The first is the clear appointment… and the other, is the ambiguous appointment… for example, when he (peace be upon him and his household) said, “Whosoever I am his mawla then Ali is his mawla.”
Al-Shareef Al-Murtadha in Al-Shafi fi Al-Imamah 2/67 makes a very similar statement, where he states that some statements of appointment are clear, while others are not, and he mentions two narrations as examples of statements that are unclear, which are: 1) You are to me as Harun was to Musa but there is no prophet after me, and 2) Whosoever I am his mawla then Ali is his mawla.
Similarly, there are also statements by Al-Tabrasi in his Al-Ihtijaaj p. 255 and Al-Majlisi in Al-Bihar 93/123 that Allah sent the Prophet (peace be upon him) with a vague appointment and not a clear one when he told him to tell the people, “Whosoever I am his mawla then Ali is his mawla.”
Ghadir Khumm at most is only for Ali
As those that are familiar with Shia history are aware, there were many Shia sects in the past that had different views in regards to the identity and even number of appointed Imams. Some of them, like the Waqifi Shias, limit the Imams to seven, with Musa Al-Kathim being the last Imam. The Fathiyyah add in a completely different Imam, Abdullah Al-Aftah, the son of Ja’afar Al-Sadiq. Another group, Al-Mohammadiyyah, believed in eleven Imams, with Mohammad bin Ali Al-Hadi being the last. One only needs to skim through Firaq Al-Shia by Al-Nawbakhti to get a complete view of the differences of opinions that are held by these sects in regards to the identity of the Imams.
What does all this mean? It means that all these sects used Ghadir Khumm as evidence, even though, according to Twelvers, these sect are upon falsehood. In other words, when a Twelver brings up evidence for the Imamate of Ali by using Ghadir Khumm, the Twelver has only completed one-twelfth of the job, and there are eleven Imams that require explicit appointment for the Twelver sect to be a valid one.
In other words, Ghadir is only useful when debating Sunnis, but it is not sufficient to establish the correctness of the Twelver sect as a whole.
Hadith of the Pen and Paper
This might sound like a strange point to make, however, if one were to think critically, one would realize that the “appointment of Ali” at Ghadir Khumm was not as explicit as Shias would like to think it was. Shias often suggest that the Prophet (peace be upon him) intended to write the name of Ali as the appointed successor a few days before his death, however, he never got the chance to do so.
One naturally asks, why write something down when the appointment of Ali was made in front of thousands of people a few weeks earlier. The answer, simply enough, is that the “appointment of Ali” was not explicit, which is why Ali had to be appointed in writing.
Moreover, Shias often retell the event of Ghadir Khumm to be one in which those present congratulated Ali. Not a single protest was made in regards to the alleged appointment. However, we still find the Prophet (peace be upon him) attempting to write a will for Ali’s succession.
Shias cannot have your cake and eat it.
Since the two events are historically accurate, the only possibilities are that 1) the event at Ghadir Khumm was not explicit and that the Prophet (peace be upon him) realized that he made a wrong choice of words since the term “mawla” could have multiple meanings, and thus, decided to appoint Ali clearly in writing, OR 2) that the Prophet’s (peace be upon him) tongue didn’t betray him, and that he wanted to write something else that was completely unrelated to Ali.
The latter is the correct view, since it is not acceptable to believe that the Prophet’s (peace be upon him) eloquence would betray him at such an important juncture.
With the above three points in mind, it becomes clear that the conversation does not end by citing “Whoever I am his mawla then Ali is his mawla”. These words lack the explicitness that is necessary to take Ali as an infallible Imam and guide.
One should be aware that explicitness is a condition of Imamate, for Ja’afar Al-Sadiq himself states in Al-Kafi 1/170 that an Imam is known by the “clear appointment,” not an ambiguous appointment.