The author begins by expressing excitement over the recent coding accomplishments they have made, which include reducing RAM requirements, eliminating the need for system downtime during upgrades, expanding document indexing limits, and increasing keyword limits. The author then reflects on past issues with the search engine, specifically mentioning an outage caused by scaling problems and lack of RAM. They explain that the outage occurred because the system needs to go offline during an index switch, resulting in the URL database being wiped. To address this, the author proposes merging two tables in a single SQLite database, and generating unique URL IDs. These changes significantly reduce the system’s RAM usage. The author also discusses improvements made to the reverse index construction process, eliminating the need for a lexicon and reducing RAM requirements further. Overall, the changes have been highly successful in improving system performance and addressing scalability issues.
https://www.marginalia.nu/log/87_absurd_success/