Why Certain URLs May Not Appear in the Scan Sitemap

Last updated: January 23, 2026

Why Certain URLs May Not Appear in the Scan Sitemap

This article explains why certain URLs (such as modules within an admin dashboard) may not appear in the sitemap after a scan, even though the scanner is correctly crawling and analyzing the application.

Why This Matters

When reviewing automated scan results, you may notice that some URLs (for example, /access-manager, /user-management, /policy-manager) are not explicitly listed in the sitemap. This can raise concerns about incomplete coverage. Understanding how the scanner detects and reports crawled content helps clarify why this happens and ensures confidence that the full application is being tested.

1. How the Scanner Crawls Your Application

  • The scanner records API calls and requests made by your application during page loads and navigation.

  • Each API call is captured in the sitemap to represent the application’s attack surface.

2. Why Some URLs May Not Show Up

  • Certain modules (e.g., dashboards, admin panels, or settings pages) may only update the browser URL path without triggering a new API call.

  • In such cases:

    • The browser shows a new URL (e.g., /access-manager), but

    • No corresponding backend request is sent.

  • Since the sitemap is generated from actual API calls, these URLs do not appear separately in the report.

3. What This Means for Coverage

  • Even if /access-manager or similar paths are not listed as standalone entries, all underlying API calls triggered when the page loads are scanned.

  • The scanner is not skipping those modules—it is analyzing the requests made by them.

Example

Suppose your admin console has a module at:

https://yourdomain.com/access-manager

When you navigate to this page:

  • The URL in the browser changes, but no new API request to /access-manager is triggered.

  • Instead, the page may reuse existing APIs (e.g., /api/users, /api/roles, /api/policies).

  • These API calls do appear in the sitemap and are included in the scan.

URLs that only change at the browser level (without new API calls) will not appear as separate entries in the sitemap.

The scanner still covers the actual API requests made by those modules.

Your application is being tested thoroughly, even if certain URLs are not explicitly listed.

4. Authentication Flow Limitations (OIDC / SSO with PKCE)

Some applications use advanced authentication mechanisms such as OIDC / SSO with PKCE, often backed by identity providers like Keycloak and fronted by API gateways.

In such cases:

  • The scanner performs authentication using an automated, headless browser.

  • Certain OIDC + PKCE flows depend on browser state, timing, or interactive redirects.

  • During automated execution, the authentication flow may expire or fail.

Common symptoms include:

  • Login works correctly during manual testing but fails during the scan.

  • Authentication callbacks return errors such as authentication_expired or temporarily_unavailable.

  • The login endpoint does not appear in the sitemap.

  • Only public or unauthenticated pages are crawled.

What this means for the sitemap

If authentication is not completed successfully, the scanner cannot reach protected areas of the application. As a result, authenticated URLs and APIs will not appear in the sitemap.

Recommended approach

  • Re-record the login flow if recent changes were made (for example, adding a landing page before login).

  • If the issue persists, use session cookies or token-based authentication instead of browser-based login recording to ensure reliable access to authenticated areas.

What To Do If You’re Unsure

  • Review the API calls listed in the sitemap for coverage.

  • If you still believe a module is being skipped, share an example URL and expected API request with our support team. We’ll verify the configuration for you.

  • If automated login is not feasible, you can reach out to our support team to import a HAR file captured from a successful manual login session. This allows the scanner to reuse authenticated requests and ensure coverage of protected areas.

Need Help?

If you have questions about your scan results or suspect a module isn’t covered, please reach out to our support team from your Astra dashboard. We’ll be happy to walk you through the sitemap and confirm coverage.