Andrew here; so last night, we noticed load on the server while just being, slowly creeping up. After some digging, and pecking with AI's assistance, we determined that it was ChatGPT indexing our site, which, clearly didn't read our initail message on our website. HA. The funny part is I had to bring in the assistance of AI to mitigate the issue. I'm impressed, carrying on.
GPTBot was knocking on /login.php — it did until it re-read robots.txt — but each knock is cheap now.
The shop used to run create-table, create-index, and about 76 SHOW COLUMNS checks on every hit. WTF - AI ASSISTANCE on that one. Pffft. That now happens once, yea buddy! A second pass costs 1 query instead of hundreds. SHOW COLUMNS has stopped growing.
GPTBot and Facebook’s crawler still show up in the access log, but since 11:46pm (9/8/2026) there have been zero new activity-log rows while login was still getting hit every second.
robots.txt no longer gives GPTBot a free pass. The User-agent: GPTBot / Allow: / block (and the same for Claude, Perplexity, etc.) is gone. Those crawlers now use the same disallow list as everyone else, including /login.php and /register.php. Product pages stay allowed.
The login ↔ register loop is broken. A next= that points at login, register, or a giant nested URL is thrown away. The “create account” link on login is now just /register.php, not a longer copy of the current query string.
On the box: query rate dropped from about 450/sec to ~40/sec, and load average went from ~1.1 to 0.62.
If there's any questions, please let us know via the contact form. Thanks!
GPTBot was knocking on /login.php — it did until it re-read robots.txt — but each knock is cheap now.
The shop used to run create-table, create-index, and about 76 SHOW COLUMNS checks on every hit. WTF - AI ASSISTANCE on that one. Pffft. That now happens once, yea buddy! A second pass costs 1 query instead of hundreds. SHOW COLUMNS has stopped growing.
GPTBot and Facebook’s crawler still show up in the access log, but since 11:46pm (9/8/2026) there have been zero new activity-log rows while login was still getting hit every second.
robots.txt no longer gives GPTBot a free pass. The User-agent: GPTBot / Allow: / block (and the same for Claude, Perplexity, etc.) is gone. Those crawlers now use the same disallow list as everyone else, including /login.php and /register.php. Product pages stay allowed.
The login ↔ register loop is broken. A next= that points at login, register, or a giant nested URL is thrown away. The “create account” link on login is now just /register.php, not a longer copy of the current query string.
On the box: query rate dropped from about 450/sec to ~40/sec, and load average went from ~1.1 to 0.62.
If there's any questions, please let us know via the contact form. Thanks!