Dify annotation replies for approved answers

Dify annotation reply: how approved answers are returned instead of generated

A support lead often faces the same questions dozens of times a week, and legal or compliance has already approved the wording for each answer. Annotation reply lets a Dify app return that approved text directly. When it is enabled, Dify searches the existing annotations for a semantic match to the incoming question. If a match exceeds the configured similarity threshold, the curated response is returned. If no match clears the threshold, normal generation proceeds as it would without annotations. The decision that matters most is therefore the threshold: set it too low and a question that only resembles an approved one receives an answer written for something else, set it too high and the approved text is rarely used. Test the threshold with real phrasings from your queue, including the awkward ones, before you let customers reach the app.

Approved answers fit best where the correct response is fixed and the cost of drift is high: refund windows that policy defines, notices about outages, eligibility rules that change only through a formal review, and disclosures that must match a published document word for word. They do not replace generation for questions that need account context, judgment about a specific case, or a summary of a long document. The embedding model used for annotations also deserves a deliberate choice, because changing it regenerates all annotation embeddings and so must be planned as a maintenance event rather than a casual setting change. Confirm with your own team which embedding model you will standardize on, and which reviewer signs off each annotation before it is imported.

Plan the governance around the feature as well as the feature itself. Bulk import of question and answer pairs speeds up the first load, but every imported pair needs an owner, an approval record and a review date, because an answer that was correct last year may not be correct now. Hit tracking shows which annotations are being used and which questions still reach generation, so review those logs regularly to find gaps and stale wording. Annotation hits are a useful signal for compliance evidence, but they do not prove that a response was lawful or accurate on its own. Keep human review steps in place for anything that carries a legal or financial consequence, and confirm with your own compliance team how retained conversation records fit your retention policy.

Configuration options and when each fits

This table compares the four settings that shape annotation replies: similarity threshold, annotation embedding model, bulk import of Q and A pairs, and hit tracking. Read each row as a trade-off between returning a curated answer too eagerly and missing a question the team already answered. The main takeaway is that the embedding model choice should be settled before large imports, because changing it regenerates every embedding.

Option	What it controls	Trade-off and when to adjust
Similarity threshold	The minimum semantic match score a stored annotation needs before Dify returns its curated answer instead of generating one	A high threshold avoids returning an approved answer to a loosely related question but misses paraphrases. A low threshold catches more paraphrases but risks serving an approved answer that does not fit. Adjust it after reviewing test questions, not from intuition.
Annotation embedding model	The model that converts annotated questions and incoming user questions into vectors for matching	Different embedding models judge similarity differently, especially for domain terms, product names and short questions. Changing the model regenerates all annotation embeddings, so confirm the model and language coverage with your support content before bulk import.
Bulk import of Q and A pairs	How many approved question and answer pairs enter the annotation store at once, and how they are reviewed before they go live	Large imports make coverage fast but can carry unreviewed or outdated answers into production. Import in batches that a subject expert has signed off, and keep the source file under your own version control so each approved set can be traced.
Hit tracking	Records of which annotations matched user questions and how often, so reviewers can see what is being served	Hit data shows which approved answers are used and where matches look wrong, but it also stores user question text, which may fall under your retention rules. Review hits on a set schedule and confirm with your privacy team what is kept and for how long.

Setting up annotation replies on a self-hosted Dify

This sequence turns a reviewed set of approved answers into a working annotation reply on one Dify app, with the embedding model confirmed, the threshold tuned and hits visible in tracking before any user sees the result. It ends with evidence that matched questions return curated answers and unmatched questions still generate normally.

Choose the target app and scope
Select one chatflow or chat app that already has a defined support purpose, and confirm with the owning team which question categories it may answer from curated content. Record the app name, workspace and the person accountable for the approved answers, since that owner signs off on every later change. Done when the scope note lists the app, its question categories and its approver, and the app's current behavior on three sample questions has been captured for comparison.
Confirm the annotation embedding model
Open the annotation reply settings and check which embedding model Dify will use for annotation matching. Confirm that the model is available from the provider configured in your workspace, that it runs where your data rules allow, and that your team understands the cost and data path of calling it. Record the model name in your change log, because changing it later regenerates all annotation embeddings. Done when the chosen model is saved and recorded, and a test embedding call from the console succeeds.
Prepare the reviewed question and answer set
Assemble the question and answer pairs in a spreadsheet, with one row per approved answer and columns for question, answer, source document, reviewer and review date. Remove personal data, account identifiers and anything a subject-matter reviewer has not signed off. Keep the approved set under version control so later changes are traceable. Done when every row has a named reviewer and the file contains no content that is not approved for customer-facing use.
Import the annotations in bulk
Use the bulk import option for annotations in the app's annotation settings, uploading the reviewed file in the format the Dify interface expects. Import into a staging or test app first if your change process requires it. Then open the annotation list and compare its row count with the approved set, checking a sample of entries for exact wording. Done when the count matches, rejected rows are logged for correction, and the spot-checked answers read as approved.
Set the similarity threshold
Enable annotation reply and set the similarity threshold. A higher threshold returns curated answers only for close matches, while a lower one matches looser phrasings and raises the risk of returning an answer to a different question. Start conservatively and write down the value and the reason for it. Run the test set from the previous step at this value and record which questions match and which do not. Done when the threshold is saved, the reason is documented, and the test results show the matching behavior your reviewer expects.
Test with non-sensitive live questions
Ask the app a set of real but non-sensitive questions drawn from past support tickets that have been stripped of personal data, plus questions that are close to approved ones and questions that have no approved answer. Have a reviewer judge each response as curated, generated or wrong. Done when the review sheet shows that clear matches return the approved answer, that unrelated questions still go through normal generation, and that any wrong match has been traced to a threshold or annotation change.
Confirm hit tracking and unmatched behavior
Open the annotation hit tracking view and confirm that each test match appears as a hit against the expected annotation, and that unmatched questions do not appear there. Check that the logs record the conversation as normal generation when no match exceeds the threshold. Confirm with your operations team which logs are retained and for how long under your privacy policy. Done when hits for the test matches are visible, unmatched questions generated normally, and the hit records are exported or noted as evidence for the change record.
Record the approval and the rollback path
Document the final model, threshold, annotation count, test results and approver in the change record, and state how to disable annotation reply or revert the import if a wrong answer appears in production. Confirm the rollback steps with the owner of the app before enabling it for users. Check whether your deployment's backup routine covers the annotation data before any future upgrade, since Dify's upgrade guidance recommends a backup beforehand. Done when the approval record is signed and the rollback steps have been rehearsed once in a test app.

Where approved answers help and where they do not

Annotation reply fits best where the answer is stable, already approved and the same for every customer. Examples include business hours, the location of a published policy page, how to reset a password through a supported channel, or the list of documents a customer needs to upload. Because the curated answer is returned verbatim when the similarity match is strong enough, the approved wording is what the customer reads, and generation does not rewrite it. Generation still handles anything the annotation does not cover, so the team should expect a gradual shift in coverage rather than a single switch.

Clear ownership: name one owner for each approved answer and record who approved the wording and when.
Narrow scope: keep annotations to facts that change rarely, and link out to the live policy page where one exists.
Threshold testing: test paraphrases of each question, since a loose threshold can pull an answer into a neighboring topic.
Change triggers: tie annotation review to policy releases so stale answers are removed the same day the policy changes.
Hit review: check hit records for answers that match often but were written for a narrower question.

Before relying on this tab, confirm with your compliance owner which facts count as stable and which policy pages customers are directed to.

Security and governance considerations for curated answers

Annotation replies turn approved wording into an answer path that bypasses generation, so the people who create, approve and retain those entries carry the risk. The cards below cover who holds those permissions, how imported content is checked, what must stay out of annotations, and where the data physically sits in your deployment.

Who may create and approve

Decide before rollout which roles can add, edit and delete annotations, and who signs off that an answer is correct for customers. Dify's built-in workspace roles are the first control to check, but confirm in your own environment exactly what each role can change in the annotation settings, since those permissions are not a substitute for a written approval step. Assign ownership of each annotation set to a named team so that a wrong answer has an accountable owner.

Review before bulk import

A bulk import of question and answer pairs puts many unreviewed answers into a path that returns them without generation. Run the import from a staging copy, have a second reviewer check a sample of entries against current policy and product terms, and remove duplicates or outdated wording before going live. Keep the source file under version control so each batch can be traced and reversed.

Keeping personal data out

Annotation text is stored in the deployment and is returned verbatim to any user whose question matches closely enough, so a name, account number or health detail typed into an answer can surface for a different person. Write answers as general guidance, reject entries that contain identifiers in your review checklist, and test with realistic but non-sensitive questions before enabling the feature for end users.

Data path and embeddings

Annotations and their embeddings are stored inside your Dify deployment. The embedding model you choose determines whether vectors are produced by a cloud provider or a model you host, and the vector store determines where they are indexed, with Weaviate as the default. Map these choices against your data residency rules, and confirm that changing the embedding model regenerates all embeddings, which also moves that data through the new model.

Link to conversation log retention

Questions that match an annotation are still recorded in conversation logs, so a retention rule for logs also governs the user questions that trigger curated answers. Set the log retention period to match your privacy policy, and review whether the annotation text itself has a separate deletion path when an answer is withdrawn. Confirm both behaviors in a test instance, because cleanup of logs and of annotations may not run together.

Secrets and deployment access

Annotation data is protected only as well as the host it runs on. The SECRET_KEY setting signs sessions and encrypts OAuth credentials, so treat it as a managed secret and plan for the logout effect of changing it. Restrict administrative access to the Dify console and the underlying database to the staff who manage annotations, and confirm backups of annotation data are encrypted and access-controlled.

Limits to confirm before relying on annotation replies

Annotation replies return stored answers only when a question matches closely enough, so their value depends on your own question set, your app versions and how you maintain the answers over time. The cards below list what to test and confirm before a regulated support team depends on curated answers.

Threshold behavior on your questions

Dify returns a curated answer only when a semantic match exceeds the similarity threshold, and otherwise falls through to normal generation. Build a representative set of real support questions, including paraphrases, typos and near-miss questions with different policy outcomes. Record which ones hit an annotation, which ones generate, and whether any near-miss returns the wrong approved answer. Confirm the threshold on your own question set before go-live.

Embedding model changes

Changing the annotation embedding model regenerates all annotation embeddings. Treat that change as a re-validation event: rerun the question set, compare hit and miss results against the previous baseline, and confirm the regeneration completes on the size of your annotation library. Schedule model changes outside peak support hours and record who approved each change.

Behavior across app versions

Dify ships minor releases roughly monthly, so an annotation setup tested on one version may behave differently after an upgrade. Ask your team to confirm, in a staging copy, that annotation settings, thresholds and hit tracking survive each upgrade. Check the upgrade guide for the target version and take a backup before every upgrade.

Exports do not carry annotation content

Dify DSL exports include app configuration, workflow nodes, model parameters, prompts and knowledge base connections, but not knowledge base content or usage logs. Annotation handling therefore needs its own procedure: confirm with a test import whether your annotation set moves with the app, and keep a separate, versioned copy of every approved answer with its owner and review date.

Maintenance effort for curated answers

Approved answers go stale when policies, product terms or fee schedules change, and each stale answer is served with the same confidence as a current one. Define who owns each answer, how often it is reviewed, and how retired answers are removed. Ask any implementation partner to quote this ongoing review work separately from the initial bulk import of question and answer pairs.

Hit tracking and audit evidence

Hit tracking shows which annotations answered live questions, which is the evidence reviewers will ask for. Confirm the tracking data is retained long enough for your review cycle, that it can be exported without exposing end-user content beyond your policy, and that it links each served answer to the approved version that was in force at the time.

Questions teams ask before enabling annotation replies

How much effort does building a reviewed Q and A set take?

Effort depends on how many questions you want to cover, how many distinct answers already exist, and how much review each answer needs before it is approved. Start by pulling the questions your support team answers most often from ticket history and chat logs, then group near-duplicates so each approved answer covers a real cluster of wording. Drivers that raise effort include answers that differ by product version, region or customer tier, answers that need legal or compliance sign-off, and content that is scattered across wikis and old tickets. Most of the cost is people time from subject-matter experts and reviewers, not platform work. Ask any partner to show how they will source, deduplicate and approve the set before importing it.

How long does a pilot take, and what does it need from our team?

A pilot length depends on the size of the approved set, the number of channels you test, and how long your reviewers need to sign off. It needs a named business owner for the question set, one or more reviewers who can approve or reject answers, an engineer who can run a Dify instance and manage its model and embedding settings, and access to real but sanitized questions for testing. Plan for a test period long enough to see both hits and misses across normal traffic. Confirm the schedule, deliverables and acceptance criteria in writing before the pilot starts, since most DeployOpen engagements are fixed-scope.

Can an internal team run annotation replies without a partner?

Yes, a team with Dify administration skills and someone who owns the content can run annotation replies alone. The work is mostly configuration and content governance: setting the similarity threshold, choosing the annotation embedding model, importing question and answer pairs, and reviewing hit tracking. The harder parts are usually process, such as deciding who approves answers, how answers are retired when policies change, and how misses are fed back into the set. A partner is most useful when the team lacks Dify operations experience or needs the work documented as evidence for an assessment. Decide based on available skills and reviewer capacity.

What changes when we switch the annotation embedding model?

Switching the annotation embedding model causes Dify to regenerate embeddings for all annotations, so the change affects every stored question and answer pair, not only new entries. Before switching, confirm that the new model is approved for the data it will process, that it runs where your data governance rules allow, and that its cost and rate limits are acceptable for the size of your set. Re-test the similarity threshold afterward, because match scores from a different model are not directly comparable with the old ones. Record the change, the date and the test results in your change log so reviewers can trace why hit behavior moved.

Where do approved answers reduce risk in regulated support use?

Approved answers reduce risk when the same question must always receive the same reviewed wording, such as a disclosure, a fee explanation or a policy statement. They are less suitable for questions whose correct answer depends on account-specific data, because a stored answer cannot see that context. Treat every annotation as a controlled record: keep an owner, an approval date, a source reference and a retirement process. Confirm with your compliance team whether a curated answer counts as the approved text under your own policies, and whether your retention settings for conversation logs match your privacy notices.

How do we know whether annotation replies are working?

Use hit tracking to see which questions return an annotated answer, which ones fall through to generation, and which annotated answers receive negative feedback. A low hit rate usually means the set is missing common wording, the threshold is too strict, or the embedding model handles your language poorly. A high hit rate on the wrong question usually means the threshold is too loose or two answers are too similar to separate. Review a sample of hits and misses each week during the pilot, and tie each change to a recorded test. Your own evaluation set, built from real but sanitized questions, is the best measure.

When is it worth bringing in a partner for annotation replies?

A partner is worth considering when you need a fixed-scope build with written deliverables, a documented pilot with acceptance criteria, or help with Dify hosting and security settings that your team has not run before. DeployOpen implements, configures, hardens, documents and supports self-hosted open-source platforms inside your own infrastructure, and it prepares evidence for compliance assessments without issuing audit reports. It does not decide what your approved answers say. Ask for a written scope covering the content workflow, the configuration, testing, handover and any ongoing support, then compare that against your internal capacity.

Keeping curated answers working through Dify upgrades

Plan for frequent minor releases. The Dify releases page shows minor versions arriving roughly monthly (1.16.0 on July 17, 2026, then 1.17.0 on August 25, 2026), so an annotation reply set that works today will meet new code within weeks. Each upgrade should follow the upgrade guide for the target version, and the Docker Compose documentation recommends taking a backup before every upgrade. Because annotations and their vectors live in the same database and vector store as the rest of the application data, that backup is the only clean rollback point if matching behavior changes. After upgrading, compare the new .env.example against your .env file, since new variables can appear in a release and be missing from your deployment.

Treat the annotation embedding model as a separate change control item. Changing it regenerates every annotation embedding, so an upgrade that also swaps the model changes two things at once and makes a regression hard to trace. Before any upgrade or model change, export a fixed test set of questions with the answers you expect to return and whether each should hit an annotation or fall through to generation. After the change, run that set against the same similarity threshold and record which entries matched, which missed and which matched the wrong approved answer. Compare the results with the pre-upgrade baseline before you release the change to users.

Write the retest into the runbook as a gate, not a courtesy check. Assign an owner for the test set, keep it under version control with the approved answer text and its review date, and confirm with your compliance lead which changes require re-approval of answers that no longer match cleanly. Hit tracking gives you a before-and-after signal in production, so review it for a few days after each upgrade. Your own environment, not a generic schedule, should set how long you monitor and who signs off. Confirm the exact annotation behavior for your target version in the Dify annotation documentation before you rely on it.

How DeployOpen scopes and delivers annotation reply work

DeployOpen plans, implements, customizes, tests and documents self-hosted Dify in your own infrastructure, including the annotation reply configuration that returns approved answers. Engagements are fixed-scope, so the answer set, threshold, embedding model and review ownership are agreed in writing before any build starts.

How we scope

Inventory the questions that should return approved answers and the ones that must still go to generation
Agree in writing which Dify apps, workspaces and owners will hold annotation reply settings
Record the embedding model choice, its hosting location and the process for regenerating embeddings
Define the review and approval roles that sign off each Q and A pair before import

How we build

Configure annotation reply per app with a similarity threshold tested against sample questions from your team
Prepare bulk import files of Q and A pairs in a format checked for duplicates and outdated answers
Set up hit tracking so reviewers can see which curated answers are returned and how often
Test fallback behavior so questions outside the curated set still reach normal generation as designed

How we hand over

Document the threshold rationale, embedding model, import procedure and the approval workflow for owners
Deliver an export of app configuration in DSL form, noting that annotation content must be backed up separately
Walk through how to review hit reports, retire stale answers and rerun imports after content changes
Provide a runbook covering what to check after Dify upgrades, since minor releases arrive roughly monthly

How we support

Ongoing support, upgrades and continuous testing are available as retainers after handover
Retainers can cover periodic review of hit tracking and retirement of answers that no longer match policy
Regression checks of annotation behavior can be repeated when the embedding model or Dify version changes
Support scope, response expectations and fees are confirmed in the written agreement before work begins