Publishing a page and having it found are two separate events, and on large sites the second one is where the delay lives. In Quebec City there is an extra complication: the same document often exists at two addresses because of a single accent.
The complaint is always phrased the same way. The page is live, it is linked, it is in the sitemap, and three weeks later it still returns nothing in a site search. Nobody blocked it, nobody made a mistake, and the fix is not a tag. What has happened is that the page joined a queue you cannot see, behind a lot of other pages that had a better claim on the crawler's attention.
This article covers what that queue is, what consumes it, and what the Indexing Hub inside the Semalt panel can and cannot do about it. It is aimed at anyone running a documentation library, a publication archive or a seasonal catalogue in the Capitale-Nationale — and at anyone evaluating a supplier here who wants to know whether the answer they are being given is a real one.
A page that exists is not a page that is found
Search engines do not hold a list of every page on the web. They hold a list of addresses they have reason to visit, and each visit costs something. Your site is one of many millions competing for a finite amount of that attention, and the allocation you receive is set by factors mostly outside a single publishing decision — how often your pages change in ways that matter, how quickly your server responds, how much of what was fetched last time turned out to be worth keeping.
That allocation is what people call crawl budget. It is not published, not configurable and not purchasable, which makes it easy to dismiss as a theoretical concern. It stops being theoretical the moment your site passes a few thousand URLs, which in this region happens faster than anyone expects.
What actually consumes the budget
Crawl budget is rarely consumed by the pages you care about. It is consumed by the ones you forgot were addressable. Every filter combination on a product listing is an address. Every sort order is an address. Every session parameter, every printable version, every calendar month going back to 2011 is an address, and a crawler that follows them is not spending that time on your new service page.
The pattern is consistent across the sectors here. A financial services firm publishes one policy document and links it from a library page, a search page, a category page and a downloads page, each with its own parameters. A university publishes a research output that appears under the author, the department, the year and the repository. A tourism operator generates a page per date. None of this is a mistake in itself. In aggregate it is a budget problem.
- Parameter variants. Sort, filter and tracking parameters multiply one document into dozens of addresses that all return substantially the same thing.
- Slow responses. A server that takes two seconds per page is crawled less than one that takes two hundred milliseconds, because the cost per fetch is higher.
- Soft failures. Pages that return a 200 status with an empty or error body teach the crawler that fetching from you is often wasted effort.
- Endless archives. Paginated listings with no ceiling and date archives with no end absorb crawl attention indefinitely and produce almost nothing.
The corrective is unglamorous and it works: reduce the number of addresses before you try to increase the attention. Every URL you remove from the crawlable surface is attention returned to the pages that earn revenue.
The accented URL and its unaccented twin
Here the address-multiplication problem has a regional variant that does not exist in most markets. French page titles carry accents, and when a content system generates a slug from a title it has to decide what to do with them. Different systems, and the same system across different versions, decide differently. The result on a site with any history is predictable: two live addresses for one document, sometimes three.
The first is the stripped form, where é becomes e and the path reads cleanly in a browser bar. The second keeps the accent, which means the character is percent-encoded whenever the address travels — in a link, in a log line, in a sitemap file. The third is the legacy one, produced by an older system that encoded the same character under a different scheme, and it usually survives because somebody bookmarked it and nobody noticed.
| Variant | Where it comes from | What it costs you |
|---|---|---|
| Stripped slug | Modern CMS transliteration | Usually the safest canonical choice |
| Accented slug | Systems that preserve the title verbatim | Percent-encoded in transit, awkward in logs |
| Legacy encoded slug | Pre-migration platform, different byte encoding | Invisible to editors, still crawled, still indexed |
| Both forms linked internally | Old menus and hand-written links | Splits signals and doubles the crawl demand |
| Both forms in the sitemap | Automated export from a mixed database | Tells the crawler explicitly to fetch twice |
The scale of this is easy to underestimate. On an archive of ten thousand French-titled documents, a mixed history can leave four or five thousand shadow addresses in circulation. That is not a five percent inefficiency, it is half your crawl demand spent confirming that two URLs hold the same text.
Sitemaps do a job, if you let them
A sitemap is widely treated as a compliance artefact: generate it, submit it once, forget it. Read properly it is the only place where you state, in machine-readable form, exactly which addresses you consider real. On a site with a duplication problem that statement is the most valuable thing you can publish.
Sitemap submission and recursive parsing
For sites whose URL inventory lives in an index file rather than a spreadsheet.
- Upload or point at a URL. Submit a sitemap file directly, or give the hub the address of one already published.
- Recursive parsing to three levels. An index of indexes of sitemaps is followed down three levels, which covers almost every real-world structure.
- Up to 1,000 sitemaps per job. Large segmented inventories can be processed as one job instead of being fed in by hand.
- Two jobs at a time, twenty queued. Concurrency is capped at two running jobs with a queue of up to twenty behind them.
Three levels of nesting is more generous than it sounds, and it maps well onto how documentation-heavy sites are actually built. A top index points at one sitemap per section; each section index points at one file per year or per product family; each of those lists the documents. That structure is worth building deliberately, because it turns a single opaque list into a diagnostic: when discovery lags, you can see which branch it lags in.
The site index
One entry per major section, so a problem can be localised before anyone opens a log.
- Services, archive, news, resources
- Keep it stable across releases
The section index
Split by year, product family or programme, at a size that stays readable when something goes wrong.
- Segment before you hit the file limit
- Name files so the split is obvious
The document list
The actual URLs, in canonical form only, with honest modification dates.
- One form per document, never both twins
- No redirects, no dead entries
What keeps it useful
A sitemap that lists addresses which redirect or 404 is a statement that you do not know your own site.
- Regenerate on publish, not quarterly
- Drop expired pages deliberately
Getting addresses in front of the crawler
Beside the sitemap route, the hub takes URLs directly. The ceilings are fixed and worth memorising, because they determine how long any real project takes. The tracker works to a daily budget of 1,000 URLs per account, and bulk submission accepts up to 10,000 URLs in a single batch — the batch is the container, the daily budget is the rate at which it drains.
The URL tracker and bulk submission
For a defined list — a migrated archive, a new catalogue, a season of pages going live at once.
- 1,000 URLs per day. The account-level daily budget, and the number every schedule has to be built around.
- 10,000 URLs per batch. The maximum size of one bulk upload, which then draws down against the daily rate.
- IndexNow delivery. Submissions go out through the IndexNow API, covering GoogleBot and BingBot.
- Per-URL logging. Each address carries its own record of bot visits with timestamps, status and error detail.
IndexNow is worth understanding for what it is: a notification protocol. It tells participating engines that an address has changed, which removes the waiting time between publication and the crawler finding out. It does not tell them the page is good, and it does not oblige them to do anything about it. What it buys you is the removal of one specific delay, and on a site that publishes frequently that delay is often the biggest one.
The status of a batch, and what it means
A submission you do not read is a submission you did not make. The hub keeps live counters for submitted, found and failed, and a per-URL log recording bot visits with timestamp, status and error details. Those three counters answer three different questions, and confusing them is the most common misreading in this whole area.
Submitted means the address left your side. Found means a bot came and fetched it. Failed means the fetch was attempted and did not produce a usable result, and the error detail on the individual record is where the actual cause sits — a timeout, a server error, an unexpected redirect chain, a status code that does not match what the page appears to show in a browser.
| What the batch shows | What it tells you | What to do next |
|---|---|---|
| Submitted high, found low | Notification worked, attention did not follow | Look at site-level crawl demand, not at the batch |
| Found high, still not in results | Fetched and judged not worth keeping | A content and duplication question, not a technical one |
| Failed clustered in one section | Almost always a server or routing fault | Open one record, read the error, fix once |
| Failed scattered thinly | Usually transient or stale addresses | Re-run the failures, do not re-run the batch |
| Found, then a redirect logged | You submitted a non-canonical twin | Correct the source list before resubmitting |
Where this bites in the Capitale-Nationale
The sites in this region that hit the wall are not the biggest ones. They are the ones with a document library attached to a small marketing site, where the URL count grew for reasons nobody tracked.
The documentation library
Policy wordings, coverage summaries, rate schedules and forms, each in several versions and each linked from multiple index pages.
- Retire superseded versions properly
- Keep one index page per document
The publication archive
Research outputs and clinical documents surfacing under author, department, year and repository, which is four addresses for one item.
- Choose one canonical view
- Let the others be navigation, not URLs
The season that expires
Programme, schedule and package pages built fresh each year, with last year's set still live and still consuming attention.
- Decide in advance what expires
- Redirect to the standing page, not the home page
The project portfolio
One completed project generating a gallery page, a case page and a tagged listing entry, repeated across several hundred projects.
- Cap the tag pages
- Consolidate galleries into the case page
Now the arithmetic, because it is the part that makes the planning conversation short. Take a financial services documentation library of 38,000 addresses after a platform migration. Submitted as they stand, at 1,000 URLs per day, that is 38 days of daily budget — and 10,000 of them can sit in a single batch, so you are looking at four batches and roughly six weeks before the last address has even been announced.
Now remove the accented twins and the superseded versions. If 40 percent of that inventory is duplication — which is a realistic figure for a French-language archive that has been through a migration — the real set is around 23,000 addresses, three batches, 23 days. You have saved a fortnight and, more importantly, you have stopped asking a crawler to fetch fifteen thousand pages it was going to discard. The cleanup is the faster route, and it is the one that keeps working after the batch finishes.
Frequently asked questions
If I submit a URL through the hub, will it be indexed?
No. Submission makes an address known and invites a fetch. Whether the page enters the index afterwards is decided by the search engine on its own criteria, and pages are routinely fetched and then not kept. The honest expectation is faster discovery, not guaranteed inclusion.
Should the canonical URL keep its accents or drop them?
Both work technically, and the stripped form is the more common choice because it survives copying, logging and pasting without percent-encoding. What matters far more than the choice is that only one form is live, that the other redirects permanently, and that internal links and the sitemap use the same one.
How deep can my sitemap structure go?
The hub parses recursively to three levels, and a job can take up to 1,000 sitemaps. In practice three levels — site index, section index, document list — is enough for a documentation archive of almost any size, and it has the advantage of showing you which branch is lagging.
Can I run several sitemap jobs at once?
Two run concurrently and up to twenty more can wait in the queue. That is generally more than enough, since the constraint on a large project is the daily URL budget rather than job concurrency.
Does IndexNow cover Google?
Submissions through the hub go out via the IndexNow API covering GoogleBot and BingBot. Treat it as a notification channel: it removes the waiting time before an engine learns that something changed. It makes no promise about what the engine does next.
What indexing work actually buys you
Handled properly, discovery work removes a category of excuse. You stop wondering whether a page is missing because of a technical fault, because the log tells you when a bot came and what it got. You stop resubmitting the same archive every quarter, because you know the ceiling is a thousand addresses a day and you plan against it. And you stop treating the sitemap as paperwork, because it has become the place where you declare which of your accented twins is the real one.
What it does not buy you is a reason to be in the index. A crawler that fetches a page it judges to be a near-copy of something it already holds will decline it, politely and permanently, and no amount of resubmission changes that. That is why the deduplication pass comes first in every plan worth following.
The hub sits alongside the campaign and analytics sections of the same panel, so a page that has been discovered and then starts collecting impressions shows up in the Search Console views without a separate export, and the indexing status of each URL stays visible next to its performance. For sites where wording passes through review before publication, the FullSEO tier at 500 USD per month per domain — roughly 685 CAD, an approximate conversion — adds a human-review mode over the same automation; our service pages set out how that fits an archive migration.
If you want to see the queue against your own inventory rather than in the abstract, open the panel and submit one sitemap. Point it at your largest section, read the found and failed counters after 48 hours, and you will learn more about your crawl situation than any audit document will tell you. Further reading continues on our blog, and the full feature overview lists the remaining views in the hub.