The author discusses the challenges of productionizing language models and the solutions to these challenges. They note the ambiguity of natural language, which can lead to silent failures and inconsistency in user experience. To mitigate this ambiguity, the author suggests applying as much engineering rigor as possible. They also discuss the cost and latency of language models and the tradeoffs between prompting and finetuning. The author finds the use of embeddings and vector databases promising for applications such as search and recommendation. They highlight the use of prompt tuning and finetuning with distillation as promising techniques.
https://huyenchip.com/2023/04/11/llm-engineering.html#backward_and_forward_compatibility