Nepenthes is a tarpit designed to trap web crawlers, particularly those scraping data for LLMs. The software generates pages with countless links that lead back to the tarpit, causing endless loops. It includes intentional delays to prevent server overload and offers the option to add Markov-babble for AI model training. Installing it behind nginx or Apache is recommended to hide its presence. Nepenthes can be used defensively to block crawlers or offensively to overload AI models with irrelevant data. The configuration file allows for customization, including setting delay times and training a Markov corpus. Proceed with caution as this can negatively impact search results and CPU load.
https://zadzmo.org/code/nepenthes/