snipette/weflock
snipette/weflock
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Add robots.txt, disallowing /article/ and /api/

Browse source 
That's because /article/ and /api/ are dummy paths that can be
used for an AI tarpit like Nepenthes!
This commit is contained in:
Badri Sunderarajan 2025-03-23 01:13:20 +05:30
parent 5e23bb65e6
commit 17e6e9e0d1
Signed by: badri
GPG key ID: 9F594532AD60DE03
1 changed files with 9 additions and 0 deletions

9
robots.txt Normal file
View file

@ -0,0 +1,9 @@
User-agent: *
Sitemap: https://snipettemag.com/sitemap.xml
Disallow: /ghost/
Disallow: /email/
Disallow: /members/api/comments/counts/
Disallow: /r/
Disallow: /webmentions/receive/
Disallow: /article/
Disallow: /api/