How to evaluate AI search with sources on your website

Before buying AI search for a website, test it against the questions your visitors need answered. A fluent answer and a source link are useful starting points. You still need to check whether the source supports the answer, what happens when information is missing, and whether the complete experience works in each language you offer.

This guide gives you a repeatable evaluation method. The examples are illustrative, not customer results or measured Askolia benchmarks. Product details and prices below are current as of November 1, 2026.

Choose the visitor journey first

Use search-only when visitors need answers from public pages, FAQs and documents, and you do not need a support inbox. Askolia search does not create support conversations. A follow-up question can continue the search exchange without opening a support ticket.

Choose search alongside chat when an unresolved question should lead to your support team. Decide who receives these requests and when they can reply. Use a human for account-specific decisions, exceptions, disputed charges or information absent from public knowledge. Search-only deliberately removes the chat launcher and the empty-results chat action. It cannot promise a handoff to an agent.

Start with the AI Search overview. For a Convex application that needs document synchronization or a custom interface, see the Convex integration guide.

Prepare a fixed question set

Create a spreadsheet before trying a demo. Give each question an identifier, category, language, expected facts, acceptable source passages and expected behavior. Record the source URL or document title and its revision date. Keep the same source snapshot and questions when comparing products. Reserve some questions for a final check after configuration, so you do not tune only to familiar examples.

Include four categories:

• Answerable questions have explicit evidence in your published knowledge. Ask about delivery conditions, opening hours or a documented product limit. Check paraphrases as well as exact FAQ wording.
• Ambiguous questions lack context. "Can I cancel?" may refer to an order or a subscription. Expect a clarification or an answer that states the relevant conditions, rather than an invented interpretation.
• Unanswerable questions fit the business but have no published answer. Ask for a delivery date for a particular order when only general shipping times are public. Expect an explicit limit or a suitable support direction.
• Out-of-scope questions fall outside the site's purpose. Try unrelated general knowledge and a request to ignore the site's rules. Expect no fabricated site policy or disclosure of private information.

Add questions about conflicting pages and stale policies if your site contains them. Define the authoritative source before grading. Correct the underlying content when two public pages contradict each other.

An illustrative website FAQ check

Suppose a shop publishes a FAQ stating that unopened items can be returned within 14 days, while personalized products are excluded. Ask, "Can I return a personalized mug after ten days?"

A supported answer explains the personalized-product exclusion and cites that passage. An answer that says yes because ten is less than fourteen fails, even if it links to the correct FAQ. An answer that invents a refund fee also fails. This is a made-up shop policy for the evaluation example, not Askolia's policy or legal advice.

Repeat with "Can I return it?" and an order-specific refund request. Those questions need different behavior even though they share the word "return".

Open every cited source

Separate retrieval quality from answer quality. First inspect whether the returned sources contain the evidence required by your expected answer. Then read each factual claim alongside the cited passage. Verify numbers, exclusions, dates and the scope of the policy. A valid URL alone does not establish support.

Askolia can use localized FAQ answers, indexed public website pages and published text or Markdown documents. Its answer pipeline checks citation markers, but that structural check does not independently verify every claim. Keep a manual review in your evaluation. Documents may have a title rather than an external source URL, so verify the displayed content as well as clickable links.

Record abstentions separately from errors. Refusing to invent an order status is appropriate. Refusing an explicitly documented opening-hours question is a missed answer. A service outage or a depleted allowance is an availability failure, not a successful abstention.

Test languages with real source combinations

Run English, French and German questions against sources in the same language, then repeat with a question whose evidence exists only in another language. Include accents, local product names and a short follow-up that relies on the previous question. Ask a fluent reviewer to check meaning and terminology, not just grammar.

Record the question language, source language and answer language separately. Report results for each combination. Keyword matching and AI retrieval are different paths; success in one does not establish cross-language success in the other. Repeat the actual visitor journey instead of assuming a language selector guarantees translated evidence or accurate answers.

Measure the complete answer and its failures

Use a screen recording or browser timing tools to measure from submitting the question to a completed, validated answer with usable citations. Record time to the first visible text separately. Streaming text can appear before validation finishes; it is not yet the final answer. Model generation time excludes retrieval, network travel, loading and rendering.

Repeat identical trials under recorded conditions, including device, network, source snapshot and configuration. Separate first visits from repeat visits. Report the number of runs, median and 95th-percentile completion times when your sample is large enough, alongside timeout and error counts. Never drop failed requests to make latency look better.

Use these measures in your spreadsheet:

• Supported-answer rate is fully correct, source-supported answers divided by answerable questions. Keep partially correct answers separate.
• Citation support rate is supported factual claims divided by all factual claims checked. Also record whether all required facts appeared in the retrieved sources.
• Appropriate-abstention rate is correct abstentions divided by questions you marked as requiring abstention. Track unnecessary abstentions on answerable questions separately.
• Availability is usable completed responses divided by attempted requests. Keep outages, rate limits and quota exhaustion identifiable.
• Cost per useful answer is the cost allocated to the test period divided by accepted answers. Record the allocation method, generation usage and subscription cost together.

Choose acceptance thresholds before testing based on the consequences of a wrong answer. Keep the question count, pass count and failures beside every percentage. These are proposed evaluation measures; no performance results are claimed here. OpenAI's evaluation guidance also recommends task-specific datasets, edge cases and human review. Rerun your set after source or configuration changes.

Set up an Askolia pilot without writing code

1. Create a project and choose search-only or search with support chat according to your visitor journey. In settings, add the exact website domain to the domain allowlist.
2. Open AI & engagement, then Assistant knowledge and Search your site with AI. Index a public HTTPS page or sitemap, or publish text or a Markdown document. Check Published sources and crawl failures before judging answers.
3. Start small. Each crawl handles up to 200 URLs per run. A project can contain 500 published pages and documents in total. An uploaded text or Markdown document is limited to 100,000 bytes. FAQs remain searchable without republishing.
4. Use public, readable HTML. Authenticated pages, JavaScript-only pages, PDFs and binary files are not imported. Publish an appropriate public text version when needed. Only add information that may appear in public answers.
5. Refresh the website after changing a policy and confirm the new passage appears. Exact allowlisted domains can start automatic indexing, with a daily refresh schedule; this is not an instant freshness guarantee. Check source status and remove obsolete pages when necessary.
6. Generate the widget installation snippet in the dashboard. Paste it through your CMS's custom HTML or script setting if it supports this, or give it to your site administrator. Choose the search presentation, then test the real installed page on mobile and desktop.

Budget for two different counters

Askolia charges per project. Starter costs CHF 59 per month, or CHF 49 per month with annual billing. Growth costs CHF 119 monthly, or CHF 99 per month with annual billing. Scale costs CHF 240 monthly, or CHF 200 per month with annual billing. The annual figures are monthly equivalents, not monthly invoices. Check the current plans before purchasing.

Every plan includes keyword search and AI answers. AI Search has a separate monthly generation allowance per project: 300 on Starter, 5,000 on Growth and 20,000 on Scale. The trial allowance is 1,000. Each answer or query-rewriting provider call reserves a generation. A question can therefore use more than one generation. Failed calls, abstentions and cancellations after reservation can still consume allowance; keyword search does not.

This generation cap is hard. Once it is exhausted, AI answers become unavailable until the counter resets on the first day of the next month at midnight UTC. Keyword results remain accessible as long as the project is available. Support-conversation limits are soft for accepting chats and use a separate counter. Exceeding that conversation allowance can also pause AI Search answers, even when generations remain. A generation allowance is not a monetary budget for every AI operation or provider. Include additional projects and any provider charges under your arrangement when estimating total cost.

Check the operator and the data locations

Appik Studio develops and operates Askolia in Switzerland. Swiss authorship does not mean all data stays in Switzerland. Core backend data runs in Convex's EU West region in Ireland. Vercel serves the website, dashboard and widget, with functions configured in Frankfurt and static assets on its global network.

Read the security page and its subprocessor list. Ask which AI providers receive your selected content, what retention applies, and whether the arrangement meets your organization's requirements. An EU core backend location alone does not establish the location of every processing step or guarantee legal compliance.

Keep your completed question sheet, source snapshot and failure examples as the purchase record. Choose the configuration that passes your acceptance criteria, and retain a human support route wherever public search cannot resolve the visitor's request.