How Dify knowledge sync from GitHub and S3 works
A team that wants repository and bucket content searchable by a Dify app inside its own infrastructure can do it with two official plugins from the Dify Marketplace: a GitHub datasource plugin, which covers repositories, issues, pull requests and wiki pages, and an AWS S3 Storage datasource plugin. Both are published by langgenius in the datasource category. A datasource plugin brings documents from an external system into a knowledge base for retrieval. That is the job here. The plugin docs distinguish it from a *trigger plugin*, which starts a workflow when an upstream event happens. A trigger can tell a workflow that a pull request was opened, but it does not by itself put that content into a knowledge base. If the goal is searchable content, the datasource plugin is the component to plan around, and a trigger is only worth adding if the team also wants workflows to react to events.
The sync runs as a knowledge pipeline. The datasource plugin reads from GitHub or S3, the pipeline processes the documents, and Dify writes them into the knowledge base index. Before building, confirm in the marketplace listings which content types each plugin actually reads in the Dify version you run, since the GitHub plugin's coverage of issues, pull requests and wiki pages is the feature set the listing describes. Also confirm how each plugin handles deletions and re-syncs, because those behaviors decide whether the index drifts from the source over time. Your own environment, the plugin's configuration form and a test repository or bucket are the right places to check.
The credentials need the same care as the content. For S3, the plugin should use an IAM principal scoped to the specific bucket and prefixes the knowledge base needs, with read-only actions, not an account-wide key. For GitHub, use a token limited to the repositories in scope. Because Dify encrypts stored credentials with the workspace key and outbound requests can pass through the SSRF proxy, record which hosts the plugin must reach and test that the rules allow them and nothing else. Plan the rotation and revocation path before the first sync, and check the plugin's documentation for the authentication method it supports.
What you need before connecting GitHub or S3
Knowledge sync from GitHub and Amazon S3 depends on four things lined up before any datasource is configured: a supported Dify version, the marketplace plugins, source credentials with narrow scope, and network paths from Dify to the source APIs. Check each one in your own environment before the first sync run.
Self-hosted Dify and Docker Compose
The Docker Compose deployment starts the api, worker, plugin daemon and supporting services, and the self-hosting docs set a minimum of 2 CPU cores, 4 GB of RAM and Docker Compose 2.24.0 or later. Confirm that your host meets these numbers with room for indexing jobs, since embedding large repositories or buckets adds load to the worker. Pin the Dify version you test against, because the datasource plugins are tied to the plugin architecture that Dify runs.
GitHub and S3 datasource plugins
Install the GitHub datasource plugin, which covers repositories, issues, pull requests and wiki pages, and the AWS S3 Storage datasource plugin, both published by langgenius in the datasource category. These are datasource plugins, so they bring documents into a knowledge base for retrieval. They do not start workflows on repository or bucket events, which is the job of trigger plugins. Confirm the installed plugin versions in the Marketplace listing before you plan the sync schedule.
GitHub identity with read access
Decide whether the sync runs under a GitHub App installation or a personal access token, and grant it read access only to the repositories you intend to index. Issues, pull requests and wiki pages each need their own read permission, so a token that reads code may still fail on wiki content. Repository owners should confirm the exact token type and permission names against the plugin's current listing, and should record who owns the credential so rotation does not break indexing.
AWS bucket and dedicated IAM principal
Create a bucket for the documents and a separate IAM principal used only by Dify, with list and read actions limited to that bucket or prefix. Avoid reusing an administrator or application role that also writes to the bucket, because a broad grant turns a retrieval connector into a path to other data. Confirm the exact actions the AWS S3 Storage plugin requires in its current listing and keep the access key in your secret store, not in workflow definitions.
Network paths from the plugin daemon
The plugin daemon and the API must reach GitHub and the regional S3 endpoint for your bucket over outbound HTTPS. If you route outbound traffic through a proxy, the SSRF_PROXY settings can block private address ranges, so a destination inside your network will fail unless you add an allowlist entry for it. Test reachability from the running containers, not from your workstation, because the containers may sit behind different egress rules.
Sync owners and data classification
Name an owner for each source and a reviewer who confirms that the repositories and buckets may enter a knowledge base that other workspace members can query. Issues and wiki pages often contain personal data, customer details or security notes that were never meant for a general assistant. Settle these decisions before the first run, since removing content from an index later requires a deliberate cleanup.
Configure the GitHub and S3 datasource plugins in order
This sequence takes a self-hosted Dify instance from installed plugins to a knowledge pipeline that imports GitHub and S3 content, with each step closed by a check the team can run. Work moves from the source systems to the Dify workspace, so read-only access is in place before any document leaves its origin.
Install the GitHub and AWS S3 datasource plugins
A platform administrator installs the GitHub datasource plugin and the AWS S3 Storage datasource plugin from the Dify Marketplace into the self-hosted workspace. Both are published by langgenius in the datasource category, so confirm the publisher and the plugin type before installing. The step is complete when both plugins appear in the workspace plugin list as installed and enabled, and the datasource category shows them as available for knowledge pipelines.
Create a read-only GitHub credential
The GitHub administrator creates a token or app credential limited to read access on only the repositories that will feed the knowledge base. Confirm with the GitHub documentation which permission scopes cover issues, pull requests and wiki pages in your organization, and avoid write scopes. The check is a test read against one approved repository outside Dify, which should succeed while a write attempt is refused.
Create a scoped IAM identity for the bucket
The cloud administrator creates an IAM identity whose policy allows only listing and reading objects under the approved bucket and prefixes, with no write, delete or policy permissions. Record the bucket name, region and prefix list in the change ticket. The check is an access test with those credentials that lists the approved prefix and is denied on an unapproved bucket or on a write call.
Add both credentials in the Dify workspace
A workspace administrator opens the datasource settings for each installed plugin and enters the GitHub credential and the S3 access values, then saves them. Dify stores these values as workspace credentials, so restrict who can view or edit them. The step closes when each datasource shows a successful connection test and the credential is listed as configured, without the secret displayed in plain text.
Build a knowledge pipeline with the datasources
A knowledge owner creates a knowledge pipeline in Dify and adds the GitHub datasource and the S3 datasource as sources, then selects the repositories and the content types, such as issues, pull requests or wiki pages. For S3, select the bucket and prefixes rather than the whole bucket. Save the pipeline only after reviewing which items it will ingest. The check is a preview list that matches the approved repositories and prefixes exactly.
Run the first import and record counts
The knowledge owner runs the first import and waits for indexing to finish, then records the number of documents imported from each source and any items that failed. Compare these counts with the expected numbers from the source systems, noting that some items may be excluded by the selected content types. The step closes when the counts reconcile with the source listings and the failure list is understood or resolved.
Verify retrieval with cited sample queries
The knowledge owner runs a set of known questions against the knowledge base, using one question per source, and checks that the answers cite chunks from the expected repository file, issue, or S3 object. Keep the questions and expected sources in a written test list so later syncs can be compared. The step is complete when each sample query returns cited chunks from the correct source, and any missing or stale result is logged for follow-up.
What the GitHub plugin imports and what it leaves out
The GitHub datasource plugin from the Dify Marketplace covers four content types: repository content, issues, pull requests and wiki pages. Each one becomes a document that a knowledge pipeline can chunk, embed and index, but each arrives in a different shape. Repository content is the source files and documentation in the branch you point the plugin at, so its value depends on which branch, paths and file types you select. Issues, pull requests and wiki pages are conversational or narrative records, and their usefulness to retrieval depends on whether titles, bodies, labels and discussion threads are kept together. Plan the knowledge base around these four shapes before you enable the connector, because mixing them in one dataset makes retrieval results harder to judge.
Do not assume that every field, comment or attachment on those objects is carried into Dify. Before you rely on any content type, read the plugin's own documentation on the GitHub datasource plugin listing and confirm, for each type, whether metadata such as authors, dates, labels and state is stored, whether review comments and issue comments are included, how images and binary attachments are handled, and whether closed or merged items are imported. Run the connector against a small test repository that contains a closed issue, a merged pull request with review comments and a wiki page with an attachment, then inspect the resulting documents in the knowledge base. Record what is missing, because your access reviews and your answers to users will depend on that record.
Treat the gaps as a design input, not a surprise at go-live. If comments or attachments are dropped, decide whether a second source (for example, the repository itself or a document export) must cover them, or whether users should be told the content is out of scope. Confirm the plugin version you install against the Dify Marketplace listing, since behavior can change between releases. Also check that private repositories and issue content inherit the access you intend, because a document imported under a service credential is visible to anyone who can query that knowledge base in the workspace.
Mapping sources, permissions and knowledge base fields
This table traces each GitHub and S3 source object to the Dify knowledge base document, the credential that reads it, and the access app users receive. Read each row as a chain: the source object, the credential, the Dify field, then the access decision you must make. The main takeaway is that source-side permissions do not carry into Dify automatically, so access must be designed in the app model.
| Source object | Credential that reads it | Dify knowledge base field | Access Dify grants to app users |
|---|---|---|---|
| GitHub repository (code, README) | GitHub token scoped to the repository, held by the datasource plugin | Document per selected file or path, with the repository as the source | Set by the knowledge base and app access settings, not by GitHub repository visibility |
| GitHub issues | Token with issue read scope on the chosen repositories | One document per issue, including title, body and comments as indexed | Anyone with knowledge base access can retrieve private issue text unless the knowledge base is restricted |
| GitHub pull requests | Token with pull request read scope, separate from write scopes | One document per pull request, including review discussion as indexed | Review comments may hold sensitive detail, so restrict the knowledge base to the groups allowed to see them |
| GitHub wiki pages | Token with access to the wiki of the chosen repository | One document per wiki page, updated when the page changes at sync | Wiki access in GitHub does not map to Dify roles, so define who may query these pages |
| S3 bucket and prefix | IAM user or role limited to s3:ListBucket on the bucket and s3:GetObject on the prefix | Documents per object under the chosen prefix, one knowledge base per data classification | Bucket policy does not reach Dify app users, so bucket scope and app scope must match |
| S3 object (file) | Same scoped IAM credential, read only, no write or delete actions | One document per object, with object key kept as the document name for traceability | Object-level tags or prefixes are not enforced in Dify; separate knowledge bases where access differs |
| Dify app and workspace | Plugin credential stored in Dify, encrypted under the workspace secret settings | Knowledge base attached to one or more apps | Workspace members and app access settings decide who queries the index; confirm the Community edition limits with the docs |
Scoping GitHub tokens and IAM credentials for read-only sync
Both datasource plugins reach external systems with credentials that outlive any single sync run, so their scope decides what Dify can read if something goes wrong. The cards below cover token and IAM scoping, where secrets live inside a self-hosted Dify, and the outbound controls that sit between Dify and the internet.
Read-only GitHub token per repository set
Create the GitHub credential for the datasource plugin with read-only repository permissions and limit it to the specific repositories whose issues, pull requests and wiki pages belong in the knowledge base. Avoid organization-wide access tokens for convenience. Before you start, confirm with the plugin's Marketplace listing which permissions it requests, since the set it needs for wiki pages may differ from the set for issues. Revisit the repository list whenever a team's repositories change.
IAM access limited to named prefixes
Give the S3 datasource an IAM user or role that can call ListBucket and GetObject only, and scope both actions to the prefixes that hold documents meant for retrieval. A broad read policy on the whole bucket exposes every object to indexing. Ask the plugin maintainers which actions and resource formats it expects, then test the policy with a sample prefix before connecting production data. Denying writes and deletes is the simplest way to keep the sync read-only.
Credentials in Dify's encrypted storage
Enter datasource credentials through the Dify console so they are stored by the platform, not written into the .env file or a Compose override that may be committed to version control or copied to backups. Restrict who holds the workspace role that can view or edit plugin credentials. Confirm with your Dify version how credentials are encrypted at rest, and document who can rotate them.
SECRET_KEY as a protected value
The SECRET_KEY setting signs session cookies and JWTs and encrypts OAuth credentials. Generate a strong value before first start, store it in your secrets manager, and do not change it after deployment, because the environment reference says changing it logs out all users. Treat any copy of this key as sensitive, since it protects the stored connection data that the datasource plugins depend on.
SSRF_PROXY and endpoint allowlisting
Outbound requests from Dify can pass through the SSRF_PROXY settings, which block internal and private IP ranges. Expect datasource traffic to leave through this proxy. Before you allowlist any destination, confirm with each plugin's documentation which endpoints it calls, such as the GitHub API or the regional S3 endpoint, and open only those. Do not widen the proxy to internal hosts to make a sync work without first reviewing why it needs them.
Keeping the index current after the first import
The first import is the easy part. The harder question is what happens when the GitHub repository or S3 bucket changes after launch. A re-sync does not simply refresh the whole knowledge base. Depending on how the datasource plugin detects changes, it may re-embed only the objects it sees as new or modified, or it may re-process everything under the configured path. Each re-embedded chunk costs model calls on your embedding endpoint, so the sync mode you choose sets both the recurring cost and how long stale content lingers. Confirm in the GitHub datasource plugin and the AWS S3 Storage datasource plugin documentation whether each supports incremental runs, what it uses to detect a change (commit or object metadata, for example), and whether it can be scheduled or must be triggered manually.
Deletions and renames are where indexes go wrong. A file removed from a repository or an object deleted from a bucket may leave its chunks in the knowledge base, so the assistant keeps answering from content that no longer exists. A renamed file can look like a deletion plus a new document, which can produce duplicate chunks if the old entry is not removed. Before relying on any sync, run a controlled test: add a marked test file, run the sync, delete or rename it at the source, run the sync again, and check the document list and retrieval results in Dify for both the old and new names. Repeat the test for an issue or pull request closed at the source if you index those.
To set sync frequency, start from how fast the source changes and how much a stale answer costs. A policy document in a bucket that changes quarterly does not need the same schedule as a wiki that changes daily. Weigh the embedding spend of each run against the staleness window you can accept, and record the decision with the owner of the knowledge base. Re-check the schedule after any Dify upgrade, since release notes can change how documents are processed, and keep a backup of the knowledge base before changing sync behavior.
Why a GitHub or S3 sync fails and how to fix it
This table pairs the failures teams most often hit when a GitHub or S3 datasource feeds a Dify knowledge base with the likely cause and the fix for each. Check the cause column against the symptom you see in the sync logs, then apply the fix in the credential, network or plugin layer before re-running the sync.
| Failure | Likely cause | Fix |
|---|---|---|
| Repository sync returns authentication errors | The GitHub token has expired, was revoked, or lacks read access to issues, pull requests or wiki content | Issue a token scoped to the approved repositories with the read permissions the datasource needs, store it in the workspace credential, and confirm the expiry date with your GitHub administrator |
| Bucket listing returns empty or access denied | The IAM policy grants object reads but not list permission on the bucket or the chosen prefix | Add s3:ListBucket on the bucket with a prefix condition, add s3:GetObject for the same prefix, and test with the same credentials outside Dify first |
| Datasource cannot reach GitHub or S3 | Outbound requests are blocked by the SSRF proxy, which rejects internal and private address ranges or hosts not allowed by your settings | Review the SSRF_PROXY settings in your environment file, confirm the proxy path to the API endpoint, and allow only the destinations your sync needs |
| Datasource appears in Dify but cannot run | The GitHub or AWS S3 datasource plugin is not installed or not loaded by the plugin daemon | Install both plugins from the Dify Marketplace, check the plugin daemon logs after install, and restart the service if the plugin is not listed |
| Sync stops partway through a large repository | GitHub API rate limits or very large issue, pull request or wiki volumes exceed what one run can process | Narrow the scope to specific repositories or content types, schedule syncs outside peak hours, and confirm rate limit behavior with the plugin documentation |
| Answers cite pages or files that no longer exist | Deleted issues, renamed objects or removed S3 keys stayed in the index because the sync only adds or updates documents | Confirm how the datasource handles removals, re-run the sync, delete stale documents from the knowledge base, and check retrieval results after each change |
Questions teams ask before syncing GitHub or S3 into Dify
What drives the cost of syncing GitHub or S3 into Dify?
The cost is driven by who does the work, how much content enters the knowledge base, and how often it is re-indexed. Plugin licensing is not priced here, so confirm it with the Marketplace listings before budgeting. Infrastructure is the second component: the Docker Compose stack needs at least 2 CPU cores and 4 GB of RAM, and vector storage and object storage grow with the documents you index. Embedding cost recurs every time content changes and is re-embedded, which makes the number of repositories, buckets and file types the main variables. Ask any partner for a quote that separates one-time work (datasource configuration, mapping, testing) from recurring work (re-syncs, upgrades, access reviews), and that names the repositories and buckets in scope.
How long does it take to get the first import running?
The first import depends on how many sources you connect, how clean their structure is, and how much testing your retrieval quality needs. A single repository with a clear issue and wiki layout is simpler than several buckets with mixed file formats and inconsistent naming. The steps that usually take the most time are credential provisioning, deciding which content should be excluded, and checking that answers cite the right documents. No duration is set here. Ask for a written timeline tied to a defined scope, and confirm which parts depend on your identity and cloud teams.
What does our internal team need to provide before the work starts?
Your team needs to provide four things before the datasource plugins can be configured:
- A GitHub token or app credentials with read access to each repository in scope, and the name of the account owner who approves them.
- A list of repositories, branches, issue and pull request ranges, and wiki spaces to include or exclude.
- The S3 bucket names, regions, prefixes and an IAM role or user scoped to read only those prefixes.
- A named access owner for each source who can confirm which content may enter the knowledge base.
Also confirm where the Dify instance runs, because outbound calls to GitHub and S3 pass through the SSRF proxy configured in your environment.
Can one in-house engineer set up GitHub and S3 sync alone?
One experienced engineer can configure the datasource plugins if they already operate Docker and have authority over the cloud account and the Git hosting settings. The work becomes harder when the engineer must also negotiate access with repository owners, design IAM policies with a security team, and test retrieval quality across sources. Plan for at least one reviewer who can check the scoped credentials and one owner who signs off on indexed content. Ask a partner which parts they would do with you and which parts you keep, and get that split in the written scope.
How does the Community edition single-workspace limit affect this design?
The Community edition is free and limited to a single workspace, so every GitHub and S3 source you connect lands in that one workspace. Teams that need separate knowledge bases for different business units, or separate access boundaries, must plan for that before the first sync. Because the Enterprise edition adds multiple workspaces and SSO, a design that splits sources by owner may need Enterprise licensing, which is custom priced. Confirm the workspace model with Dify's pricing page and your licensing contact before you assign datasource credentials to people.
How do we keep knowledge base indexes current after the first import?
Indexes stay current only when you decide how changes are picked up and who checks the result. Each re-sync re-processes changed content, so the volume of commits, issue updates and object uploads determines the recurring embedding and storage work. Decide which repositories and prefixes need frequent updates and which can refresh less often, and record that schedule in the runbook. Upgrades of Dify also need a backup before each change, as the Docker Compose documentation advises. Ask for a written re-sync and monitoring plan as part of the scope.
Which access risks should we review before connecting GitHub or S3?
The main risks are over-broad credentials and indexing content that should not leave its original access boundary. Use read-only GitHub access limited to the repositories in scope, and an S3 policy limited to the specific bucket and prefixes instead of the whole account. Remember that Dify's knowledge base is shared inside the workspace, so anyone with access to that knowledge base may query indexed issues, pull requests or documents. Review private repositories and confidential prefixes with their owners, and confirm that the SSRF proxy keeps internal network addresses out of reach of the datasource calls.
How DeployOpen scopes and delivers a knowledge sync
DeployOpen plans, implements, hardens, documents and supports self-hosted open-source platforms inside your own cloud accounts, data centers or air-gapped networks. For Dify knowledge sync from GitHub and Amazon S3, that means agreeing the source scope, credentials and index behavior in writing before any connector is configured.
How we scope
- Inventory which repositories, issue and pull request types, wiki pages and S3 prefixes the knowledge base should hold, and which must stay out.
- Confirm the Dify version and deployment layout, including which vector store and metadata database the knowledge base will use.
- Agree in writing the IAM policy shape, GitHub token type, sync frequency and deletion behavior before work starts, since engagements are fixed-scope.
How we build
- Install and configure the official GitHub datasource and AWS S3 Storage datasource plugins in a staging workspace before production.
- Create scoped read-only S3 credentials and GitHub access for the agreed sources, and store them in Dify without exposing them in exports.
- Test ingestion, re-indexing and removed-file handling against sample repositories and buckets, and record the results for review.
How we hand over
- Deliver a runbook covering each sync source, its credentials, owners, schedule and the steps to rotate keys or rebuild an index.
- Document the outbound network paths the Dify instance needs to reach GitHub and S3, including any SSRF proxy allowlisting you must approve.
- Walk your platform and security teams through the configuration and the checks that confirm the index matches the source.
How we support
- Review the knowledge pipelines after each Dify minor release, since releases arrive roughly monthly and plugin behavior can change.
- Ongoing support, upgrades and continuous testing are available as a retainer, scoped separately from the original build.
- Ask any partner, including DeployOpen, to confirm plugin versions, IAM and token scopes, and sync costs in writing before you approve the plan.
