Dify website import: how a crawl fills a knowledge base
A Dify website import uses a crawl service to fetch pages from a site and turn them into documents in a knowledge base, so a team with product docs, policies or an internal wiki can answer questions from those pages without uploading each one by hand. The crawl services Dify offers, Firecrawl and Jina Reader, change over releases, so confirm which provider is available, how it is configured and which fields it needs in the Dify documentation for the exact release you run, starting with the Dify docs on the Docker Compose deployment and the release notes on the Dify releases page. Do not copy settings from a blog post or an older version, because a crawl is only as correct as the configuration the release expects.
The crawl decides what content enters the index, but it does not decide how that content is indexed, retrieved or handled. Chunking, the index method, retrieval settings and the file and data handling rules of the knowledge base still apply to every page the crawl returns, so a page that is poorly structured or full of navigation text will be chunked and retrieved badly no matter which crawl service fetched it. Before the first crawl, review what the knowledge base will hold, including pages you did not intend to publish, and decide who can see the resulting documents, since the knowledge base access settings govern that once the content is inside.
Treat the crawl as an ingestion path with its own risks. Confirm with the vendor or in your own environment where the crawl service runs, whether page content leaves your network, and how the outbound requests it makes are routed and filtered, especially when the target is an internal site. Plan the crawl scope, meaning which domains, paths and depth are allowed, and test it against a staging knowledge base first. Freshness is a separate decision: the import creates a snapshot at the time it runs, so decide how often pages are re-imported, who owns that schedule and how removed pages are cleaned up.
What a Dify website import needs before the first crawl
A crawl touches your Dify version, your host sizing, a third-party account, network egress, and the content itself. Settle these six items before the first crawl so that a failed run points to a setting rather than to a missing approval or a broken path.
Confirm the Dify release in use
Record the exact Dify version before you plan the crawl. Minor releases have arrived roughly monthly, so a version you test in April may not match production by summer. Website import behavior, crawl provider settings and environment variables can change between minor releases, so read the release notes for the target version and test the crawl on that version first. Note the version in your runbook next to the crawl configuration.
Meet the Docker Compose baseline
The Docker Compose deployment needs Docker Compose 2.24.0 or later, at least 2 CPU cores and 4 GB of RAM. The stack includes the api, worker, sandbox and SSRF proxy services, plus PostgreSQL, Redis, Weaviate and Nginx. Crawl jobs run through the worker and write embeddings into the vector store, so treat these figures as the floor. Confirm headroom for indexing, not just for the web console, and test with a page set close to your real scope.
Hold the crawl provider account
Firecrawl and Jina Reader each require an account and an API key that your team owns, not one tied to a single engineer's personal login. Confirm with the provider which plan, request quota, rate limits and data retention terms apply to your use, and who is billed. Store the key as a secret in your environment, not in workflow exports, since DSL exports do not carry third-party API keys and you will need to re-enter them on import.
Open outbound paths through the SSRF proxy
Dify routes outbound requests through the SSRF proxy, which blocks internal and private IP ranges by design. That protects your network, but it also means a crawl of an intranet host will fail unless you deliberately allow it, and any allowance should be narrow and documented. Test reachability from the worker container to each target domain, and confirm the provider's crawler can reach public sites from its own network. Ask your network team where egress is logged.
Get written permission and check terms
Before crawling, get written approval from the content owner for each domain and path, and read the site's terms of use and its robots.txt directives. A crawl can copy pages that carry personal data, licensed material or pages behind a login that a service account can reach. Record the owner, the approved scope and the date of approval, because those records decide what you can index and what you must delete when the owner objects.
Back up before any version change
Take a backup of the PostgreSQL metadata database, the vector store and the file storage before you upgrade Dify or change crawl-related variables. The Docker Compose upgrade guidance recommends a backup before every upgrade and a review of .env.example for new variables afterward. Restoring a crawl without its matching vector data leaves knowledge bases that show documents but return no results, so back up all three stores together.
How to configure and test a website import in Dify
This sequence takes a website from first configuration to a verified knowledge base. Each step ends with a check, so you confirm the imported pages, text and retrieval quality before committing to a full crawl.
Confirm the crawl provider option in your version
The platform team confirms the Dify version running in your environment against the Dify releases on GitHub page, then opens the knowledge base creation flow and checks whether Firecrawl or Jina Reader appear as website import sources. Provider options can change between versions, so record what your instance actually offers. Done when you have a written note of the version, the provider options visible, and any settings that are missing or hidden by configuration.
Store the provider credential securely
An administrator adds the Firecrawl or Jina Reader API key through the credential settings that your Dify version provides, and limits who can view or edit it. Keep the key out of shared notes and out of exported app files, because the Dify docs say DSL exports do not include third-party API keys and you must supply them again on import. Done when the credential saves without error and the provider accepts a first request, which you can confirm in the first test crawl.
Set the target URL and crawl scope
The knowledge base owner writes down the root URL, the path prefixes to include, the paths to exclude, and the page and depth limits the chosen provider exposes. For internal sites, check first that the outbound route will work, since the Dify Compose stack includes an SSRF proxy that blocks internal and private IP ranges by design. Confirm the approach with your network team. Done when a one-page scope sheet lists the allowed hosts and paths and the owner has signed off.
Run a small test crawl on one section
Choose one bounded section of the site, such as a single documentation folder with a known number of pages, and run the import with the scope from the previous step. Keep the page limit low so the run stays easy to review. Compare the number of imported documents with the number of pages you counted by hand. Done when the document count matches the expected count and no pages from outside the allowed hosts or paths appear.
Review imported documents and their text
Open a sample of the imported documents and read the extracted text, not just the titles. Look for navigation menus, cookie banners, footers, repeated headers, duplicated pages, login prompts and empty pages, since these add noise to retrieval. Note which elements repeat across pages. Done when you have a short list of recurring noise patterns and confirm that the main body text of each sampled page is complete and readable.
Adjust cleaning and index settings before a full run
Using the noise patterns from the review, change the cleaning and chunking settings in the knowledge base, and select the index method your retrieval needs. Make one change at a time so you can attribute any improvement. Re-run the test crawl on the same section and repeat the document review. Done when the sampled pages show the boilerplate removed or clearly separated and the document count still matches the scope, which means the full run can proceed.
Confirm retrieval returns the expected pages
Before the full import, write a small set of questions whose answers appear on specific pages in the test section, then run them through the knowledge base retrieval testing view. Each question should return the page that contains its answer near the top of the results. If a question returns the wrong page or nothing, check whether the cause is the imported text, the chunking, or the question wording. Done when every test question returns its source page, and the same check is repeated after the full crawl.
Public sites and internal sites need different network paths
For public sites, the crawl is normally performed by the provider's infrastructure, so the network path that matters is outbound access from your Dify deployment to the provider's API and the provider's access to the public site. Your Dify instance only needs to call the provider endpoint, so your egress rules should allow that destination and nothing broader. Page text leaves your environment once it is sent to the provider, which makes the provider a third party in your data flow.
- Provider reach: Confirm which origin addresses the crawler uses, whether it follows redirects to other domains, and how it handles robots rules and page caps. Check these in the provider's own documentation before you set a crawl scope.
- Egress rules: Allow outbound HTTPS from the Dify API and worker containers to the provider endpoint through your firewall or proxy, and log those destinations.
- Data handling: Confirm how the provider stores, retains and processes page content, and whether it uses content for any purpose beyond the crawl. Do this before importing any site that carries customer or employee information.
- Permission to crawl: Confirm that the site owner allows automated retrieval and that the content licence permits copying it into your knowledge base.
- Credentials: Keep the provider API key in the credential settings Dify provides, and rotate it on your standard schedule.
Before the first crawl, check the provider's documented origins, your egress policy and its retention terms in writing.
Intranets, wikis and staging hosts usually resolve to private addresses. Dify sends outbound requests through the SSRF proxy configured by the SSRF_PROXY settings, and that proxy blocks internal and private IP ranges. A hosted crawl service cannot reach these hosts through that path, so the import can fail, return empty pages or index a login screen. The block protects the platform from server-side request forgery, so the first task is to diagnose the failure and then choose a documented path.
- Failure signs: Check the crawl job status and the worker logs. A refused connection to a private address points to the proxy, while an empty page usually points to authentication or rendering.
- Global disable is off the table: Turning off the proxy or allowing all private ranges would open every outbound request from Dify to your internal network, including requests that users can shape through app inputs.
- Scope of the block: Review the SSRF_PROXY entries in the environment variable reference and identify which hostnames and ranges are affected in your environment.
- Staging hosts: Staging sites often share names with production or use self-signed certificates. Confirm which hostnames are involved before you consider any exception.
- Authenticated pages: Crawlers commonly cannot complete SSO on intranet pages, so plan for pages that need a session to be excluded or exported.
Before choosing a path, list each hostname, its IP range, whether it needs authentication and who owns it.
Three paths can bring private content into a knowledge base. Each moves the trust decision to a different place, so none should be approved for convenience. Each needs a named owner, a written data flow and a security review before deployment.
- Self-hosted crawler inside the network: Run a crawler in the same network segment as the internal site, then send its output to a knowledge base through file upload or the Knowledge API. The crawler becomes a service you operate, so it needs patching, access control and logging. Security should review its credentials, which pages it can reach and whether it honours robots rules.
- Reviewed proxy exception: Add a route that permits only named hosts, ports and addresses, so the rest of the block stays in place. Security should review the exact destinations, and the exception should be re-reviewed after each upgrade because proxy and environment settings can change between releases.
- Export to a controlled source: Export the pages from the wiki or content system and upload the files. This keeps crawl scope under the source system's permissions. The review covers what is exported, who runs the export, where the files are stored before upload and how they are deleted afterwards.
Ask the security team to approve the chosen option in writing, with the hostnames and data flow attached.
Crawl scope limits to set before the first run
A crawl is only as contained as the limits set on it. Agree these four controls in writing before the first run, because they decide how many pages get indexed, what personal data reaches the knowledge base and how much noise retrieval has to sort through. Confirm each control against the options of the provider you choose, since names and behavior differ between Firecrawl and Jina Reader.
Starting URL and allowed domains
Pick one starting URL and an explicit list of domains the crawler may follow. Subdomains, redirects to third-party hosts and links to social or partner sites can pull in content nobody reviewed. This control mainly protects against unplanned indexing cost and content that was never approved for the knowledge base. Confirm whether the provider restricts crawling to the starting domain by default, how it treats subdomains, and whether it follows cross-domain redirects. Record the decision with the owner of the site.
Path prefixes to include and exclude
Define which path prefixes are in scope, such as product documentation, and which are out, such as account pages, search results, staging paths and internal tools. Exclusion patterns protect against personal data entering the index and against noisy retrieval from navigation or legal boilerplate. Write the patterns down, test them against a sample of real URLs, and confirm how the chosen provider matches patterns (glob, regular expression or prefix) and whether excludes take priority over includes.
Page and depth cap agreed in advance
Set a maximum page count and a maximum link depth before the run, and agree who approves going beyond them. Without a cap, a crawler can follow calendar, filter and pagination links into a very large set of near-identical pages. This control protects indexing cost and keeps the first review manageable. Ask the provider which limits it exposes, whether a crawl can be stopped midway and whether the run reports how many pages were skipped by each rule. Do not assume a default cap until the provider documents one.
Login-gated, media-heavy and duplicate pages
Decide how the crawl treats pages behind a login, pages dominated by images, video or scripts, and pages that repeat content under different URLs, such as print views or tracking parameters. Login-gated pages should usually stay out, because a crawler may capture private data it was never meant to store. Media-heavy pages add little text and cost extraction effort. Duplicates inflate the index and skew retrieval. Confirm how the provider handles authentication headers, JavaScript rendering, canonical tags and query parameters before you rely on any of them.
How crawled page fields map to knowledge base documents
This table compares each field a crawl returns with where it lands in a Dify knowledge base and what to verify before indexing. Rows marked for confirmation depend on the output format of the crawl provider you choose, so check a sample crawl before you build filters on them. The main takeaway is that metadata you add is the only field whose names you fully control, and those names decide how retrieval can be filtered across every document.
| Source field | Where it lands in Dify | What to check |
|---|---|---|
| Page URL | Source reference for the imported document, and the basis for identifying which page a chunk came from | Confirm the provider returns the canonical URL and that query strings or fragments do not create duplicate documents (to confirm with the provider) |
| Page title | Document name in the knowledge base list, which also appears in the document's index entry | Check for empty or generic titles on some templates, since these make documents hard to find and delete later |
| Body text | Converted into text segments according to the chunk mode and preprocessing rules set on the knowledge base | Review navigation menus, cookie banners and footers in a sample, as these add noise to every chunk (to confirm with the provider's markdown output) |
| Crawl timestamp | Not a native document field to confirm; keep it as metadata if the team needs to filter by freshness | Record the crawl date in your own metadata so you can tell which pages are stale when you refresh (to confirm how the provider exposes it) |
| Metadata added by the team (for example site section, owner, language) | Document metadata fields, scoped to the knowledge base for filtering at retrieval time | Use one naming convention across all sources, since field names and types affect filtering across every document in the knowledge base |
| Provider-specific fields such as status code or response headers | Only kept if you map them into metadata or drop them during import | Confirm the exact field names in a sample response before writing any mapping, and do not assume names carry over between providers |
| Pages that fail, redirect or return errors | Not imported as documents; they appear as failed or skipped items in the import run | Check the run log for skipped URLs so that missing content is found before an app goes live |
Keeping imported pages current after launch
An imported page is a snapshot. Once Firecrawl or Jina Reader has pulled a site into a Dify knowledge base, the answers built on it reflect the content as it was on the crawl date, and nothing updates them until someone re-crawls the source and replaces the old segments. Before launch, the team needs to decide three things: how changes reach the knowledge base, how stale or removed pages are deleted, and who owns that process. Name one owner for the crawl schedule, one for the scope rules, and one for sign-off on each refresh. Without named owners, the knowledge base drifts from the site while everyone assumes someone else is watching it.
Record the source URL and crawl date for every imported document, and keep them in document metadata or in your own crawl log so a reviewer can trace any answer back to the page and the day it was read. Schedule re-crawls on a cadence the team sets and confirms against how often the source content changes, and make the schedule part of the written scope rather than an informal habit. Decide in advance what happens to pages that now return errors, redirect elsewhere or fall outside the crawl scope, because those are the pages most likely to keep answering questions after the source has moved on. Confirm with your vendor how the chosen crawler reports removed pages, since that behavior determines how much of the deletion step you must script yourself.
Re-run the same retrieval test questions after every refresh and compare the results with the previous baseline, so a changed answer is noticed before users see it. Dify's minor releases arrive roughly monthly (1.16.0 on July 17, 2026, 1.17.0 on August 25, 2026 and 1.17.1 on September 10, 2026), and any of them can change how imports behave. Before each upgrade, take a backup and repeat a test crawl against a known set of pages, then compare the output with your baseline. Check the Dify releases on GitHub notes for changes to crawling or knowledge base handling, and treat a failed comparison as a reason to hold the upgrade until the cause is understood.
Securing crawl credentials, outbound requests and imported content
A website import touches three things that need controls: the API keys that let the crawler run, the outbound requests that reach sites on your behalf, and the content that lands in a knowledge base. The cards below cover where each risk sits and what to verify in your own environment before the first crawl.
Keep provider keys server side
Firecrawl or Jina Reader keys are third-party credentials that authorize every crawl and bill against your account. Store them in Dify's model provider settings on the server, not in prompts, workflow variables or shared documentation. The DSL export does not include third-party API keys, so an exported app will need the key re-entered after import. Check that exports shared with other teams contain no crawl credentials by reviewing the file before it leaves your environment.
Protect SECRET_KEY and what it covers
The environment reference says SECRET_KEY signs session cookies and JWTs and encrypts OAuth credentials, and changing it after deployment logs out all users. Store it in your secrets manager, set it before the first start, and record which stored credentials depend on it. Confirm with the version you run whether it also protects every stored provider credential, since that determines what a rotation or restore will break.
Keep SSRF proxy filtering on
Crawling means Dify fetches URLs that users or administrators choose, so a crawl can be aimed at internal services. The environment reference says SSRF_PROXY settings route outbound requests through a proxy that blocks internal and private IP ranges. Keep that proxy in the path for crawl traffic, and treat any allow-list exception as a narrow, documented change limited to named hosts and reviewed by the security owner.
Review pages before indexing
Public pages and internal wikis often contain names, contact details, customer data or draft material that should not become searchable by every app using the knowledge base. Before indexing a crawl, sample the pulled pages, check for personal or confidential content, and exclude paths that hold it. Record who approved each crawl scope so the decision can be traced later.
Respect site terms and robots rules
Confirm that the site owner permits automated retrieval and that the crawler honors robots rules and rate expectations for each target. Internal sites still need an owner's sign-off, and crawls of third-party content may carry licensing limits on reuse in AI answers. Ask legal or the content owner to confirm these points in writing before the crawl scope is fixed.
Restrict who edits crawled knowledge bases
Anyone who can create or edit a knowledge base can point it at new sources, change what is indexed and expose its content to apps. Limit crawl configuration to a small group of named owners, keep sensitive crawls in their own knowledge bases, and review access changes as part of the same change record. Knowledge base access settings determine who can use the results, so set them before the first import.
Common website import failures and how to fix them
This table lists common failures when importing websites into Dify knowledge bases with Firecrawl or Jina Reader, with the likely cause and a check the team can run for each. Read it in pipeline order: network access and crawl scope come first, then content quality, credentials, upgrades and capacity. Most failures trace to network policy or crawl scope, so check those before changing indexing settings.
| Symptom | Likely cause | Fix |
|---|---|---|
| Crawl returns no pages | Start URL unreachable from the Dify containers, provider key missing or out of quota, or pages rendered only by JavaScript | Test the start URL from the api container, confirm the provider key is saved and has quota, then retry one static page. Check the Dify FAQs. |
| Internal host pages never import | The SSRF proxy blocks internal and private IP ranges by design | Confirm the host resolves to a private address. Decide with security whether to allow that host through the SSRF_PROXY settings, not a broad rule. Consult the environment variables reference. |
| Crawl imports the wrong sections | Scope rules too broad, or the start URL is the site root and links into unrelated sections | Narrow the crawl to a path prefix, set depth and page limits, and review the URL list from a small run before indexing the full site. |
| Boilerplate dominates retrieval | Navigation, footers, cookie notices and repeated legal text are indexed on every page | Apply text cleaning to remove repeated elements, exclude utility paths, then compare retrieval results with real questions. |
| Provider rejects the credential | Key is wrong or expired, the wrong provider is selected, or the key was entered in another workspace | Re-enter the key in the workspace provider settings, check the provider dashboard for status and quota, and review the run log for authentication errors. |
| Imports fail after an upgrade | New release added or renamed variables, or changed service images such as the proxy or sandbox | Back up before upgrading, follow the upgrade guide for the target release, compare .env with .env.example, then rerun one small crawl. |
| Large run stalls | Upload limits, Nginx body size, or worker and sandbox capacity below the load | Split the crawl into batches by path, check worker logs and container resource use, and review the file upload limits in the Dify docs. |
Questions teams ask before importing a website into Dify
What drives the cost of importing a website into Dify?
Cost comes from three places: the crawl provider, the model that embeds the text, and staff time. A Firecrawl or Jina Reader account is billed by that provider, so the team pays it directly, and the plan, quota and rate terms must be confirmed with the vendor before scoping. Embedding costs depend on the indexing model and the volume of text, since every imported page is converted into vectors when a High Quality index is used. Staff time covers defining scope, reviewing what was captured and running retrieval tests. Size is the main driver: page count, page length, crawl depth and how often the site is re-crawled all raise recurring cost. Ask any partner for a quote that lists the provider plan, a page estimate for the agreed scope, the embedding model, and the review and testing effort as separate lines.
Can our team import a website into Dify without outside help?
Yes, for a small public site with a clear scope, a named owner and a Dify instance that already runs. The work is mostly configuration: choosing the crawl provider, entering its credential, setting the start URL and include and exclude rules, then reviewing the captured pages. Internal sites are harder. Dify's SSRF proxy routes outbound requests through a proxy that blocks internal and private IP ranges, so reaching an intranet site means a deliberate network path, proxy rules and often a firewall change approved by your security team. Authenticated pages add credential handling and session questions. If the scope includes those elements, the work moves from configuration into network and security engineering, and a planned written scope helps keep it contained.
How long does a website import take?
The duration depends on the scope, the review cycle and any network or security approvals, and no reliable figure applies across sites. A public documentation site with a few hundred clearly structured pages moves much faster than a large internal portal that needs proxy changes, credential reviews and content owner sign-off. Ask the crawl provider how long a crawl of your agreed scope takes and what limits apply to page counts and rates. Plan for at least one review round in which the content owner checks what was captured, plus a retrieval test against real questions. Internal approvals often set the pace, so start those requests first and treat them as the critical path in any schedule.
What does a website import need from our team?
The import needs decisions and access more than heavy engineering time. Plan for these inputs before work starts:
- A named owner who signs off the crawl scope, including allowed domains, paths, exclusions and page limits.
- A content owner who approves the captured pages and confirms that the material is safe to expose to the app's users.
- Credentials for any authenticated pages, issued to a service account with the least access needed.
- Network access for internal sites, including proxy rules and firewall changes approved by your security team.
- A set of test questions with the expected answers, used to judge retrieval quality before the app goes live.
Without the test questions, there is no objective way to tell whether the import helped.
Does Dify Community edition support website import?
Confirm this for the exact release you run before planning around it. The Dify pricing page lists the self-hosted Community edition as free and limited to a single workspace, and the Enterprise edition as adding features such as SSO, multiple workspaces and professional support. Whether a given crawl provider integration is available, and how it is configured, can change between releases, so check the release notes for your version on the Dify releases page and test the import in a non-production instance. Also confirm the provider's own terms, because a hosted crawler will receive the URLs and page content it fetches. A feature being present in the product does not settle whether it fits your data handling rules.
How do we keep imported pages current in Dify?
An import is a snapshot, so freshness needs a schedule and an owner. Decide how often the site is re-crawled, which pages are critical enough to refresh more often, and who checks the result. A re-crawl captures changed pages, but deleted or moved pages can remain in the knowledge base and continue to answer questions, so the process must also remove stale documents. Confirm in the Dify docs and your release how updates to existing documents are handled, and whether the Knowledge API can replace or delete documents automatically. After each refresh, rerun the agreed test questions. Recurring crawl charges, re-embedding and review time all repeat on this schedule, so the refresh frequency should be part of the quote.
What would DeployOpen deliver for a website import under a fixed scope?
Under a fixed scope, DeployOpen agrees the deliverables and timeline in writing before work starts. A typical written scope for this work would cover the agreed domains, paths and exclusions; the choice of crawl provider or self-hosted crawler, with its data handling reviewed against your rules; the Dify configuration, including any network path and proxy rules for internal sites; the indexing and chunking settings; a retrieval test against questions your team approves; and documentation with a handover to your owners. If you want ongoing refreshes or testing, that is handled as a separate retainer. The scope document, not this page, sets what is included, and the quote follows from it.
Should we use Firecrawl, Jina Reader or a self-hosted crawler?
The choice depends on where the content is allowed to travel, how much you want to pay per crawl and who will maintain the crawler. A hosted service such as Firecrawl or Jina Reader fetches pages on its infrastructure, which may be unacceptable for internal sites or sensitive documentation, and its pricing and quotas must be confirmed directly with the vendor. A self-hosted crawler keeps fetching inside your network and removes per-crawl provider charges, but your team then owns updates, rate limits, JavaScript rendering and failure handling. For public content, a hosted provider is often the simpler route. For internal content, compare the self-hosted option against the provider's data handling terms before choosing, and record the decision in the scope.
How DeployOpen scopes and delivers a Dify website import
DeployOpen plans, builds and hands over Dify website imports inside your own infrastructure, starting from a written scope that names the sites, crawl boundaries and refresh expectations. Engagements are fixed-scope, so the crawl service choice, the hosting model and the handover materials are agreed before any build work starts.
How we scope
- Inventory the sites to import, marking public pages separately from internal sites that need authenticated access.
- Define crawl boundaries such as allowed domains, path prefixes, page limits and excluded sections in writing.
- Confirm whether a hosted crawl service, a self-hosted crawler or a mix will be used, and where data may leave your network.
How we build
- Configure the website data source in your Dify instance and connect the crawl service credentials through your secrets process.
- Set outbound rules so crawl traffic passes through the SSRF proxy and cannot reach internal or private address ranges.
- Test a pilot crawl against the agreed scope, then check extracted text, page counts and chunking results before full import.
How we hand over
- Document the crawl configuration, scope rules, credential locations and the steps used to rerun or resume an import.
- Provide a refresh runbook that states how imported pages are re-crawled, removed or flagged when source content changes.
- Walk your operators through the knowledge base settings and the failure signals to watch after go-live.
How we support
- Ongoing support, upgrades and continuous testing are available as retainers, covering crawl configuration changes after Dify releases.
- Review crawl scope and refresh results with your team on the agreed schedule, and adjust rules as sites change.
- Confirm with your team which support terms, response expectations and service levels apply before the retainer starts.
