If you’ve noticed that the sitemap generated during a scan sometimes shows more endpoints, or fewer, compared to previous scans — there are a few logical reasons this could happen.
This is a common and valid concern, and it’s rooted in a combination of how dynamic applications work, how scans are configured, and how data and access affect crawling.
Below, we outline the typical causes and what can be done to reduce variability.
Looking for more real-time endpoint discovery?
Our API Security Platform leverages OpenTelemetry-based tracing to detect your application's surface area by analyzing real-time traffic — not just what’s discoverable through crawling. This approach ensures deeper visibility, especially for APIs and dynamically generated endpoints that may not be reachable through traditional scans.
If the underlying application has changed between scans, the sitemap is expected to reflect that. This includes:
Pages or endpoints being added, removed, or modified.
Navigation structures or sitemaps being updated.
Feature flags toggling visibility of certain areas.
Some parts of the application are data-dependent and only become visible under specific conditions. For example:
In a food delivery app, if there’s no current order, the “Track Delivery” page might not appear.
If no previous orders are marked as delivered, refund-related pages might not be accessible.
In dashboard-style apps, most of the application is available only after logging in. If login is not properly configured:
The scanner will only see public pages.
Incorrect credentials, expired tokens, or login flow changes can prevent access to authenticated areas.
Additionally, if the credentials used have restricted permissions (e.g., limited user roles), the sitemap may not reflect admin-only or restricted features.
Differences in scanner settings across scans can lead to sitemap variations. Common configuration-related causes include:
Adjusting the scope of subdomain coverage.
Adding or removing URL exclusion rules.
Modifying custom headers, authentication methods, or session cookies.
For accurate comparisons, it’s best to compare scans with the same configuration.
Factors outside the application logic can also impact crawling behavior:
Server outages or temporary latency.
Firewalls or WAFs returning 403s or captcha pages.
Cookie consent banners blocking content unless accepted.
Pages taking too long to respond or intermittently failing to load.
These can prevent the crawler from reaching certain pages or lead it to pick up unexpected routes.
If you're comparing sitemaps from different types of scans, differences are expected:
A Web Crawl performs passive crawling and collects visible endpoints.
A Full Scan includes both crawling and fuzzing — which may surface additional or even false-positive endpoints (e.g., when the app returns 2XX/3XX/4XX on invalid paths).
Lightning or Emerging scans use reduced crawl depth, focusing only on high-priority areas.
To get consistent comparisons, compare sitemaps from scans of the same type.
While some variation is natural, we take several steps to reduce it and maximize coverage:
Central Endpoint Inventory
We maintain a centralized inventory of all endpoints seen across web crawls, full scans, and observability. This inventory is cumulative and is used to enrich subsequent scans.
Authentication Health Checks
If login fails during a scan due to misconfiguration or invalid credentials, we flag it in your dashboard for quick resolution.
Connectivity Resilience
We automatically retry crawling attempts in case of temporary network issues, using exponential backoff to ensure better reliability.
Sitemap Transparency
For every scan, the sitemap is available in your dashboard, along with the full central inventory. This gives you visibility into what was detected per scan and what’s known overall.
Ensure your scan configurations (login, scope, exclusions) remain consistent across runs.
Use consistent test data where possible to make dynamic content more predictable.
Periodically review your credentials and roles provided for scans.
Let us know if you plan to make major changes in your app or environment — we can help retune scanning parameters.