Daksh Gupta, co-founder of Greptile, discusses the challenges faced with their AI code review bot leaving too many comments on PRs. Attempting to reduce the noise, they experimented with prompting the LLM to generate fewer nits, but failed. The LLM’s judgment of severity was unreliable, leading to a new clustering method – filtering out comments similar to downvoted ones. Surprisingly, this approach was successful in increasing the address rate from 19% to over 55%. While not perfect, it significantly reduced noise from the bot. This ongoing problem will likely be addressed in a future update.
https://www.greptile.com/blog/make-llms-shut-up