Q: Clarification on Data Scraping, Site Access, and Adherence to Robots.txt

Could you clarify how Abun interacts with our site data? Specifically, does the tool scrape or extract data from client websites when their domains are entered? If so, does it respect standard web protocols such as restrictions set in a site's robots.txt file? Additionally, will any data retrieved be visible to clients (e.g - server logs, etc.), and how is this information stored or managed to ensure privacy and compliance with relevant regulations?

ai_mad_219Feb 12, 2025
Founder Team
JunaidAnsari

JunaidAnsari

Feb 20, 2025

A: Hi,

Only when you enter your website domain, we scrape your website & save a summarization of the page along with the page URL. This helps us understand which page is relevant for which internal linking.

We only do this when you enter the website domain with Abun. If you do not wish to save the page URLs on our end, you can simply delete the website domain from Abun.

Share
Helpful?
Log in to join the conversation
View product details