Prevent search engines indexing your Help Center using Cloudfront

Looping in @Nathan Sudds @Milan @Joseph S.  in that in case they have some feedback to share 🙌

How can we update https://help.taskade.com/robots.txt as it is preventing Google from crawling our support articles for https://taskade.com/ despite turning the crawling on per https://www.intercom.com/help/en/articles/5145271-prevent-search-engines-indexing-your-help-center 

cc @mateusz.leszkiewicz 

@John Xie currently it is not possible to update robot.txt for the Help Center.

 

All Google - or any search engine - needs is to encounter a link to any page in your Help Center.

Most teams link to their Help Center from their marketing site. Google can index your whole Help Center starting with that link.

While you can submit your URL to Google Search Console directly, you don’t need to. — Google regularly crawls the web to add new sites to its index.

All our pages include metadata tags that search engines use to index content:

Our pages also include OpenGraph metadata. This is useful when sharing links on social media (X (formerly Twitter) / Facebook / Slack etc.). Search engines probably don't care, but worth mentioning.

We use the canonical link metadata tag. So if you include `utm_source` or other query string parameters in your URL, or if you have a custom domain and someone accesses your intercom.help Help Center, the search engine will know what the canonical URL is for this content:

If you are having issues with google not indexing your Help Center and you have a custom domain, ensure you have HTTPS set up correctly. You can find the troubleshooting steps here.

If you’re serving robots.txt through Cloudfront, I usually set a dedicated behavior for that path and attach a simple Lambda@Edge to force the noindex rules. For pages that need stronger control, adding an X-Robots-Tag header in the origin response works well. While tightening things up, I’ve used a Quora backlinks service to build authority for public articles, keeping the private help stuff hidden but still boosting the rest.