Klarity is a tool for analyzing uncertainty in generative model outputs, providing deep insights into model behavior during text generation. It offers dual entropy analysis, semantic clustering, structured output, and AI-powered analysis. The tool can be easily installed from GitHub and used with Transformer models like Qwen2.5-7B. The analysis output includes structured JSON analysis for each generation, offering insights into uncertainty points, high confidence areas, risk areas, and suggestions for improvement. Klarity supports frameworks like Hugging Face Transformers and various tested models. Advanced features allow for custom analysis configuration. Contributions to enhance the tool are welcome.
https://github.com/klara-research/klarity