Incremental Website Sync: Sync New or Updated URLs Without Full Re-crawl | Community
Skip to main content
Submitted

Incremental Website Sync: Sync New or Updated URLs Without Full Re-crawl

Related products:Fin & AI
  • June 19, 2026
  • 1 reply
  • 9 views
A

Currently, website source re-sync requires a full crawl of the entire website, even when only a single new page or a small number of URLs have been added or updated.

For organizations with large knowledge bases, this can be inefficient and time-consuming. In our case, we have thousands of indexed articles, and triggering a full re-sync just to make one new page available to Fin is not practical.

It would be valuable to support one or more of the following capabilities:

• Sync a specific URL or a set of URLs on demand
• Incremental sync that only processes new or modified pages since the last crawl
• Ability to submit a URL or a set of URLs for immediate indexing without triggering a full website re-sync

Benefits:
• Faster availability of new content in Fin
• Reduced crawl time and resource consumption
• Better experience for teams managing large documentation sites
• Improved agility for time-sensitive content updates

This would significantly improve knowledge management workflows for customers with large and frequently updated websites.
 

    1 reply

    S
    Intercom Team
    Forum|alt.badge.img+6

    Hi ​@Arunkumar Kumaresan, Seán here from the Fin technical support team 👋

    Thanks for taking the time to open this topic and spell out the use case so clearly. This is really helpful context, and I can see why being able to re-sync a specific URL or only newly added or updated pages would make a big difference for larger websites.

    The improvements you’ve outlined, like URL-level sync, incremental crawling, and faster indexing for time-sensitive updates, do a great job of showing both the operational challenge today and the value of a more flexible workflow.

    Hopefully others who are managing large and frequently updated knowledge bases can add their experience here as well, as that helps the product team better understand demand and review all feedback in one place.

      Terms & ConditionsAccessibility statement