Website
Synchronize readable pages from a permitted website into your Knowledge Base.
Choose suitable content
Use a website you are authorized to index. This connector reads accessible page content; it does not sign in through an interactive browser. JavaScript-only content, CAPTCHAs and authentication walls may require a different source.
Do not put passwords or session tokens in the website URL.
Configure the source
- Choose Add connector → Website.
- Enter the starting URL, for example https://example.com/handbook/.
- Use Limit to URL path to narrow the content, for example /handbook.
- Set a suitable maximum page count; start small enough to inspect the result.
- Test the connection, choose the sync strategy, review scope and assign CogLake role access.
- Save and run synchronization.
- 1Choose permitted pages
- 2Limit path and volume
- 3Save access settings
- 4Synchronize
- 5Check text and source links
Check the result
Open a few imported pages and compare their text and source links with the website. A reachable home page does not guarantee that every linked page is reachable or in scope. Crawl limits can mean that the entire site was not covered.
Later synchronization can update changed content. Check job details when pages are no longer present or reachable; do not treat a failed/incomplete crawl as proof that the source deleted everything.
Common issues
| Symptom | What to check |
|---|---|
| Blank or little content | JavaScript rendering or a sign-in wall |
| Access denied | Site policy, robots rules or network restrictions |
| Too few pages | Starting URL, path restriction, links and maximum page count |
| Redirect outside expected scope | Correct starting URL and the destination's accessibility |
| Rate limited | Website capacity; reduce frequency and retry later |
The interactive agent browser is a separate feature, not a way to grant the connector permanent access to an authenticated session.