ChatGPT-4’s performance in Advent of Code 2023 was a subject of concern, as some believed that AI models like ChatGPT could potentially replace the need for programmers. To test this, an experiment was conducted to observe how well ChatGPT-4 could tackle the programming challenges in Advent of Code. Advent of Code is a yearly event where programming problems are released daily for the first 25 days of December. Participants are required to submit their solutions to the website. The problems start easy and progressively become more difficult, but generally, they are solvable by a CS graduate within a few hours. Last year, ChatGPT participated in Advent of Code and even reached the top of the leaderboard in some days. However, its performance was not as impressive as expected, struggling to solve problems after day 5. This year, ChatGPT-4’s performance was slightly worse than the previous version. ChatGPT Plus, which has access to problem inputs, performed better but still had limitations. It’s worth noting that ChatGPT lacks debugging skills and tends to rewrite entire functions or programs from scratch when faced with errors. It also seems to struggle with following instructions, especially when they are not straightforward. The difficulty of the
https://www.themotte.org/post/797/chatgpt-vs-advent-of-code