Large language models like ChatGPT and Claude are common now. ChatGPT was observed being utilized by a friend’s family on New Year’s Eve. The upcoming generation will rely on large language models (LLM) for quick answers as the norm. LLMs have issues like hallucination, providing wrong answers. Understanding the output of LLMs from an information perspective is intriguing. LLMs work as probability distributions over word sequences. Entropy measures the model’s confidence in predicting the next token. Experiments with ChatGPT’s probabilities and entropies show the model’s reliability varies. The idea is to discern truth regardless of who or what speaks it.
https://nikkin.dev/blog/llm-entropy.html