Jayden/AFFiNE-Mirror
1
0
Fork 0
mirror of https://github.com/toeverything/AFFiNE.git synced 2026-02-12 04:18:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: update crawler rules (#6369)

Browse Source
This commit is contained in:
darkskygit
2024-03-29 01:33:12 +00:00
parent d14552b5af
commit 0ce6401a6f
1 changed files with 25 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
packages/frontend/core/public/robots.txt Normal file
View File

@@ -0,0 +1,25 @@
# If you would like to crawl AFFiNE public workspace contact us via developer@toeverything.info
User-agent: *
Allow: /
Disallow: /oauth/*
Disallow: /workspace/*
Disallow: /public-workspace/*
Disallow: /auth/*
Disallow: /signin/*
Disallow: /_next/*
Disallow: /invite/*
Disallow: /*/invite/*
Disallow: /api/*
User-agent: PetalBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Barkrowler
Disallow: /