On this page
Website Crawler#
The website crawler reads pages from your site and extracts their content directly into a knowledge base. Point it at a URL, choose which pages to crawl, and the crawler does the rest — navigating pages, pulling out structured data, and writing it to your knowledge base so your chatbot can answer questions from it.
The crawler handles both server-rendered and JavaScript-rendered sites. It also respects your site's robots.txt, so pages marked as disallowed are skipped automatically.
What you can import#
The crawler works with all four knowledge base types. Pick the type that matches the content on your site before you start — the crawler uses it to decide what to look for on each page.
| Knowledge Base Type | What Gets Extracted |
|---|---|
| Product Catalogue | Product records — name, price, category, description, and more. |
| FAQ | Question-and-answer pairs, with optional category. |
| Document | Page title and body content, stored as searchable document chunks. |
| Location | Location records — name, address, coordinates, and related details. |
Starting a crawl#
All crawls start from the Console, inside the knowledge base you want to populate.
- Open the knowledge base in the Console and click Crawl Website.
- Enter the URL of the site or section you want to crawl and choose a crawl mode (see below).
- The Console fetches a list of discoverable pages from your site and shows them to you.
- Select the pages you want to include. An estimate for the crawl is shown at this point.
- Click Start Crawl. The job runs in the background — you can leave the page and come back to check progress.
When the crawl finishes, newly extracted items appear in your knowledge base as inactive. Review them and publish the ones you want the chatbot to use. Items that already existed in the knowledge base are updated in place.
Crawl modes#
The crawl mode controls how pages are discovered before you start. All modes use the same extraction logic once the crawl is running.
| Mode | How pages are discovered | Best for |
|---|---|---|
| Full Site (default) | Reads your sitemap and page links to build a candidate list. | Most sites — broad coverage with minimal setup. |
| Sitemap Only | Reads only your sitemap.xml. | Sites with a well-maintained sitemap that you want to use as the authority on which pages to crawl. |
| Specific Section | Limits discovery and crawling to a URL path prefix you specify (for example, /products or /help). | Sites where content is organised into clear sections and you only want one of them. |
| Single Page | Starts with exactly the URL you entered. | Extracting content from one specific page, or testing extraction before a larger crawl. |
| Custom URLs | You enter the URLs manually, one at a time. | When you know exactly which pages you want and automated discovery is not reliable for your site. |
Crawl lifecycle#
A crawl job moves through these statuses. The Console shows the current status and a live count of pages visited and items extracted.
| Status | What it means |
|---|---|
| Pending | The job has been created and is waiting to start. |
| Crawling | Pages are being visited and content is being extracted. You can see per-page activity in the Console. |
| Saving | Extraction is done. Extracted items are being written to your knowledge base. |
| Completed | The crawl finished successfully. Items are now in your knowledge base. |
| Paused | You paused the crawl. Remaining pages are saved so the crawl can be resumed later. |
| Cancelled | You stopped the crawl. Any items extracted before cancellation are kept. |
| Failed | An error stopped the crawl. Items extracted before the failure are kept. |
While a crawl is running, you can pause it (saves progress so you can resume later), resume a paused crawl (picks up from where it stopped), or cancel it (stops immediately without saving remaining pages).
Re-crawling and updates#
You can run the crawler again at any time to pick up new or changed content. The crawler matches incoming items against what is already in your knowledge base and updates existing records rather than creating duplicates.
| Knowledge Base Type | How existing items are matched |
|---|---|
| Product Catalogue | By SKU first, then by product page URL, then by name (case-insensitive). A match updates the existing record. |
| FAQ | By source page URL first, then by question text (case-insensitive). A match updates the existing record. |
| Document | By source page URL first, then by title (case-insensitive). Only changed sections of the document are re-embedded on the next sync, so unchanged sections are not billed again. |
| Location | Matched and updated in the same way as other knowledge base types. |
Items that do not match anything already in the knowledge base are added as inactive. You can review and publish them from the knowledge base page in the Console.
Limits#
- Only one crawl job can run at a time per knowledge base. Starting a second one while a job is active (including a paused one that has not been resumed yet) is not allowed.
- All crawl URLs must belong to the same domain as the source URL you entered. Cross-domain links are not followed.
- The crawler respects your site's
robots.txt. Pages listed as disallowed are skipped. - You can select up to 500 URLs when starting a crawl. The crawler may also visit additional pages it discovers while crawling, up to the page limit you configure.
- A knowledge base holds at most 10,000 items. Crawled items that would push the total over the limit are skipped.
After a crawl completes, newly added items are not yet searchable by your chatbot. Run a sync on the knowledge base to generate embeddings for the new content. Learn about syncing →