With Crawler Instructions, you improve the Web Crawler’s search results by converting users’ natural language questions into targeted search terms. These search terms better match the structure of your website.
Note: Crawler Instructions work exclusively based on your website's URLs – not on the actual content of the pages. The search terms are compared only to the retrieved URLs.
What do Crawler Instructions do?
When a user asks your AI Agent a question, the Web Crawler uses Crawler Instructions to select the right URLs. This process works as follows:
-
The original question is translated into relevant search terms.
-
These search terms are then matched against the retrieved URLs.
-
URLs that match one or more search terms are added to the AI Agent's knowledge.
This approach helps your AI Agent provide more relevant and specific answers, especially if your website contains a large number of pages.
How to use Crawler Instructions
Writing effective Crawler Instructions is about choosing search terms that match the structure of your website. Here are some guidelines and examples.
Example
User question:
"I'm looking for a good, small Bluetooth speaker. What do you recommend?"
Optimized search terms:
-
Bluetooth speaker
-
Bluetooth speakers
-
Mini speaker
Tips for Effective Crawler Instructions
-
Use keywords that match terms in your website’s URLs.
-
Include both singular and plural forms.
-
Keep search terms short and specific.
Avoid rigid instructions like “Always return a URL” or “If someone asks for three links, always give three.” These don’t align with how the Web Crawler works.
By using Crawler Instructions wisely, you ensure your AI Agent performs better, with more relevant answers and more effective use of your website’s knowledge.