NLP research has experienced a paradigm shift with the rise of large language models (LLMs), proving the effectiveness of scale. However, the high cost and limited accessibility of these models pose a challenge for researchers without significant resources. Nevertheless, there are still important research directions that can be pursued with limited compute. These include improving efficiency throughout the LLM stack, exploring sparsity and low-rank approximations, prioritizing experimentation in small-scale settings, focusing on analysis and model understanding, and addressing data constraints in specific domains and multilinguality. Additionally, evaluating LLMs through reliable benchmarks and advancing reasoning abilities are also crucial areas of research. Despite the challenges, there are ample opportunities for innovative research beyond traditional NLP tasks.
https://nlpnewsletter.substack.com/p/nlp-research-in-the-era-of-llms