Recent advancements in large language models (LLMs) are raising hopes for accelerating scientific discovery by autonomously generating and validating new ideas. However, evaluations have not proven that LLM systems can produce expert-level ideas or complete the research process. A study compares expert NLP researchers to an LLM ideation agent, finding that LLM-generated ideas are more novel but slightly less feasible. It highlights issues with LLM self-evaluation and lack of idea diversity. The study proposes further research into whether these differences in novelty and feasibility judgments impact the final research outcomes.
https://www.arxiv.org/abs/2409.04109