In the wake of the recent visit of the Indian Prime Minister to Indonesia a tech-savvy youngster asked the popular Generative AI (Gen AI) tools—ChatGPT, Gemini, and Grok—“Who was the first Indian leader to visit Indonesia’s historic Prambanan Shiva temple?” The unanimous answer was: Narendra Modi. It was wrong.Suchitra Karthikeyan of The Hindu recently reported about this anomaly in her article Newspaper archives still trump answers generated by AI.On December 14, 1958, India’s first President, Dr. Rajendra Prasad, visited the temple during his official tour of Indonesia.(Sign up for THEdge, The Hindu’s weekly education newsletter.)This is far from an isolated glitch.A Supreme Court Bench recently set aside judgments from two adjudicating tribunals after discovering they relied on fake, AI-generated legal precedents.Join THEdge LinkedIn groupA few days back, Google’s “AI Overview” feature went viral after its bizarre advice to users to add non-toxic glue to pizza sauce to help cheese stick—a recommendation some users foolishly tried out!Now, the unsettling question in the minds of over 1.2 billion AI users all over the world—over 250 million of them from India—is: “Can we afford to trust Gen AI blindly just because it is machine-made?”The urgency of this question has increased in the last few days, when OpenAI disclosed that an autonomous AI agent “went rogue” and managed to break into another company, the Hugging Face platform, to retrieve evaluation answers, without human instruction or permission, and claimed a second victim just days later, raising the issue of security.What are AI hallucinations and why do they happen?To AI professionals, these errors are neither surprises nor traditional technical bugs. They are known as AI hallucinations, instances where a Large Language Model (LLM) confidently generates false, misleading, or entirely fabricated information, sometimes inventing non-existent products or people. Why does it happen?Gen AI models do not know the real facts, unlike the way a Google search engine looks up in a structured database to answer a query. Most of them are programmed to predict the next word based on patterns learnt from diverse data sources rather than a particular knowledge base. If the training data is limited, not updated with the latest information, or inconsistent the model fills in the gaps with something plausible but untrue in reality, as it is not based on a trusted data source.When the prompt is not clear or when the model is under pressure to answer a question for which it does not have adequate updated information to provide a correct answer it prefers to camouflage a made-up response confidently rather than honestly admitting its ignorance or inability to answer.Ask an AI tool the same question twice, and you may get two noticeably different answers. This is due to how these models work - responding to every prompt based on a probability distribution over possible next words rather than always picking the single “best” one.Language barriers, sycophancy, and working under pressureThe issue becomes even more acute when viewed through an Indian and cross-cultural lens. Due to a lack of comprehensive training data in Indian languages, global GenAI models suffer severe hallucinations when handling non-English prompts or translations. Researchers found that a model with 80% factual accuracy in English can suffer a 25% to 40% drop in veracity when queried in regional global languages.LLMs tend to flatter users. When a prompt is phrased with a clear bias or strong opinion, the model frequently agrees with the user rather than offering objective truth.A 2025 MIT study revealed a troubling pattern: when AI models hallucinate, they are actually 34% more likely to use confident phrasing such as “definitely” or “certainly” than when they are correct.When stretched beyond their thinking and reasoning limits on complex problems, some Gen AI tools exhibit erratic behaviour, generating repetitive sentences or outputs based on completely ungrounded claims.The cognitive toll on higher educationIn India, where surveys indicate that over 80% of university students and 60% of teachers use GenAI for academic work, these hallucinations pose a growing risk. Students rely on them for ideation, clarification of concepts, debugging Python scripts, and editing and summarising research. Educators use AI to draft quiz questions and prepare course materials. However, this reliance creates severe cognitive tension due to hallucinations.While GenAI promises speed and productivity, users must diligently fact-check every line to catch subtle errors. This process at times requires more mental effort than writing the assignment or paper independently.Overwhelmed by the effort required to verify AI outputs, many users stop fact-checking altogether. Offloading critical analysis, synthesis, and problem-solving to machines risks gradually eroding foundational reasoning and writing skills over time.How to test the honesty of generative AIIn human relationships, “honest” and “trustworthy” are often used interchangeably, but in the realm of Generative AI, they mean distinctly different things. Trustworthiness refers to a model’s predictability, technical competence, and structural reliability over time. Honesty means an LLM accurately generates output based on its internal knowledge without resorting to hallucinations.A model can be perfectly honest by refusing to answer difficult questions when it lacks data, yet remain untrustworthy if it frequently crashes or fails to execute basic prompts. We need systems that deliver both.Just like in humans, AI honesty is not a single measurable trait. It spans four distinct dimensions: truthfulness, non-hallucination, honesty under pressure, and low sycophancy.To evaluate these traits, international benchmarks like TruthfulQA, HonestyBench, MASK, Scale AI Honesty, BrokenMath, and HAUNT have been developed. However, no single GenAI model currently scores high across all benchmarks. To bridge this gap, institutions are adopting models like Vectara’s HHEM alongside open-source evaluation frameworks such as RAGAS and TruLens to systematically assess hallucination rates.Global initiatives to combat AI dishonestyRecognising these risks, global tech pioneers and industry consortiums are taking action.Formed by major AI players, including OpenAI, Anthropic, Microsoft, and Google, the Frontier Model Forum shares safety best practices, defines technical red-teaming protocols, and engineers advanced mechanisms to bound model confidence. In an unprecedented move, OpenAI and Anthropic ran each other’s internal safety tests on their public models and published joint results.The Coalition for Content Provenance and Authenticity (C2PA) is deploying open cryptographic standards to tag metadata in AI-generated content, allowing users to track the origin and truth profile of digital media.Researchers are currently working on training Gen AI models to issue “confessions” as a separate output in which the model explicitly admits to undesirable behavior or rule violations. While a model’s main response is evaluated on metrics such as correctness, style, and helpfulness, the confession output is judged on a single criterion: honesty. By rewarding models that faithfully report their own missteps, developers hope to encourage greater transparency in AI models.Policy frameworks for higher education institutionsLeading international universities, including Arizona State, Cornell, MIT, Oxford, and Stanford, alongside Indian institutions like IISc, the IITs, and Manipal Academy of Higher Education, have established explicit policy guidelines governing academic honesty, data security, and appropriate AI use by students and teachers.Drawing from their experiences, Indian Higher Education Institutions (HEIs) should adopt a proactive strategy. Before drafting written policies (which should align with the India AI Governance Guidelines of July 2026), HEIs must survey student and faculty AI usage patterns. Students should be trained on structured prompting frameworks such as CLEAR (Concise, Logical, Explicit, Adaptive, and Reflective). The final step, reflection, is critical as it requires students to systematically evaluate and fact-check AI outputs.Institutions must redesign assignments and shift toward more in-person, oral, or proctored evaluation components while training faculty to identify signs of AI hallucinations in submissions. While traditional software like Turnitin checks for plagiarism, source-grounded academic tools such as Consensus, Scite, and Elicit should be encouraged to ensure referenced research actually exists. HEIs should maintain an internal dossier documenting specific use cases and technical limitations of the AI tools used on campus. To enhance AI fluency, universities must invest in AI capability labs and provide subsidized access to AI tools.The way forwardGenerative AI tools offer immense potential to boost productivity and serve as valuable partners in furthering quality of education. While these tools occasionally generate errors with supreme overconfidence, yield to sycophancy, or evade complex prompts, technical advancements are rapidly enhancing their integrity. Nevertheless, academic trust must never be surrendered to an AI algorithm. It must remain firmly anchored in human curiosity, critical verification, and intellectual integrity. To adapt the popular adage for the AI age: trust, but verify. It is up to the Higher Educational Institutions to promote the productive use of AI balanced with adequate guardrails for verification.(Prof. O.R.S. Rao is the Chancellor of the ICFAI University, Sikkim. Views are personal)