Competitor pages that could not be read now get read

Change detection reads competitor pages directly now, which is faster, and a page that refuses a direct read is retrieved another way.

• Review sites and company profile pages — the surfaces most likely to sit behind bot protection — previously could not be read at all. They are covered now.
• A page that still cannot be read is recorded as “could not read” and keeps its last successful reading, instead of being counted as a change.
• Pages a site's robots.txt asks us not to read are left alone.
• The first scan after this change re-reads each competitor as a fresh baseline, so expect one run with no change report before comparisons pick up again.