OpenCTI connectors setup

When an OpenCTI import runs slowly, the connector is often not the problem. Connector output reaches the platform as messages, the platform places them on RabbitMQ queues, and Python workers consume those messages to perform the asynchronous writes into the data store. A connector can read its source quickly and still stall if too few workers drain the queue, because the backlog grows and the data does not appear in the platform until the workers catch up. Before you tune the connector, check the queue first.

The documented starting point is at least one platform node for every three workers. Use it as a minimum for planning, then adjust from observed behavior. Count workers per deployment, not per connector, because all connectors that write through the same queues compete for the same consumers. Confirm for the release you run how worker concurrency is configured, how many messages each worker prefetches, and whether queues can be assigned to specific worker groups. These settings change between releases, so read them from the documentation for your version instead of copying values from an older deployment.

Three signals tell you whether to add workers. Queue depth shows how many messages are waiting; a depth that rises during a feed run and does not fall back before the next run points to too little capacity. Worker backlog, meaning messages delivered but not yet acknowledged, shows whether workers are busy or stuck. Write latency, the time between a message being queued and the object appearing in the platform, shows what the analyst actually experiences. If depth is high and workers are fully busy, add workers. If depth is high but workers are idle or slow, the bottleneck is likely the search cluster or Redis, so check their load before adding anything.

Heavy import connectors and enrichment connectors have different timing needs, so they should not share a single path by default. A large scheduled feed can fill the queue with thousands of messages, and an enrichment request an analyst is waiting on then queues behind them. Decide the split by source volume and urgency: schedule bulk imports for quiet periods, and give enrichment its own worker capacity or queue assignment where your release supports it. Ask whether the connector list for your deployment has been reviewed for this, because the answer determines how many workers you need, not only how fast each connector runs.

Queue and memory settings must be confirmed against the release in use before you change them. Relevant items include worker concurrency and prefetch, RabbitMQ memory and disk alarm thresholds, worker memory limits, and the heap settings of the search cluster. The documented minimums for RabbitMQ (3.11 or later) and Redis (7.1 or later) are a floor for compatibility, not a tuning guide. Record your queue depth and write latency before and after each change, and repeat the check after every platform upgrade, since a release can change defaults that you depend on.

Connector data volume in OpenCTI is driven less by how many feeds you enable than by how much of their content overlaps. Two sources that describe the same malicious domain, file hash or threat actor will each push an observable, an indicator and relationships into the platform. If those objects do not share consistent identifiers and are not modeled the same way in STIX2, OpenCTI may store near-copies that look like separate intelligence to analysts and grow the Elasticsearch or OpenSearch index without adding value. Deduplication therefore starts with the connectors' output format, not with a cleanup job run later. Before you enable a second source for the same entity type, confirm in each connector's documentation how it generates identifiers and whether it merges with existing objects on import.

Several controls should be decided before the first production connector runs. Set a source priority so that when two connectors report conflicting scores, labels or descriptions for the same object, the team knows which one should win and which should be treated as corroboration. Use import filters to limit what each connector brings in, for example by indicator type, confidence threshold, label or age, so that a broad feed does not flood the graph with low-value indicators. Tag or mark the output of every connector with a dedicated label or marking definition, so the team can trace which source created an object and remove everything that source contributed if it proves noisy or is retired. Without this traceability, bulk cleanup becomes a manual investigation.

Retention and cleanup policy is the control that keeps volume stable over time. Indicators are time-bound by nature, so decide how long an indicator stays active after its last sighting, when it is revoked, and when it is removed from the platform altogether. Keep these rules separate for indicators, observables and reports, because analysts often need historical context for the latter long after the former have expired. Record the policy in writing, assign an owner, and check that the platform's own expiration and cleanup behavior matches it in your version. Confirm the exact settings and default behavior in the OpenCTI documentation for your release, since these features change between versions.

Test every source in a non-production instance before it reaches analysts. Run the connector against a representative time window, then measure how many objects it creates, how many duplicates it produces against existing data, how many relationships it adds, and how quickly the index grows. Check that the tags and markings appear as expected, that removal by source works, and that import filters behave as you intended. Repeat the test after a connector update, because changes to mapping logic can alter deduplication results. Ask your implementation partner to include this test in the written scope, and to document the observed results so the decision to enable a source rests on evidence rather than on the connector's description alone.

Before any connector feeds production, confirm that it imports what the source actually sends and that a failure in one connector cannot stall the rest of the platform. Start with a sample window, such as a few days of a feed you already know well. Compare the number of entities OpenCTI creates for each STIX object type against the count in the source export or the vendor's own dashboard. Expect differences from filtering, revoked indicators and skipped malformed objects, and record each one so the explanation exists before an analyst asks. Run the same comparison for a feed that overlaps another one, because overlap is where silent inflation starts. Next, check the relationships. For connectors that map to MITRE ATT&CK, open a sample of attack patterns and confirm that the "uses" and related relationships resolve to the expected techniques and that no objects are left pointing at placeholder or unknown references. Then test deduplication deliberately. Feed the same indicator or report through two connectors, or replay one file twice, and confirm that OpenCTI merges the objects under a single identity instead of creating copies. Check the result through the GraphQL API as well as the interface, since your later automation will read the API. Failure behavior needs its own test. For each enrichment connector that calls an external service, block that service at the network level in a test environment and submit a batch of observables. The expected result is that enrichment requests fail, the connector records errors and the queue keeps moving, so that imports and analyst work continue. If a failed enrichment stalls the import workers or fills RabbitMQ, the connector's configuration or your worker allocation needs to change before go-live. Ask your DeployOpen team or the connector documentation to confirm which timeout and retry settings apply to each connector. After go-live, monitor four signals: queue depth in RabbitMQ, connector run status and last successful run, error rates per connector, and growth in stored data. Queue depth that rises and does not recover indicates that workers are undersized or that a connector is producing more messages than expected. Data growth should track the volume you planned for in your sizing. Alert on sustained deviation rather than single spikes, and review the thresholds after the first few weeks, when the baseline becomes clear. Back up the connector configuration, including the list of enabled connectors, their scopes and their schedules, together with an inventory that records the owner, source, purpose and version of each one. Store both outside the platform. Rollback should be routine. To remove a misbehaving connector, disable it first so no new messages are produced. Then decide whether to delete the data it created or move it to a quarantine marking that analysts cannot use for decisions. Quarantine is usually safer because it keeps evidence for review. Restoring configuration and inventory is only useful if it has been tested, so rehearse the full restore in a non-production environment, timing each step and recording the gaps, before you depend on it.

DeployOpen starts every OpenCTI connector engagement with a written scope, because the connector list drives most of the decisions that follow. The scope names each source (for example a MITRE ATT&CK import, a MISP instance, a commercial feed or an enrichment service), the OpenCTI environment it lands in, the connector type it needs (import, enrichment, export or stream), the account or API key it uses, and the acceptance checks that prove it works. Connectors that consume the same feed are grouped so duplicate data is planned for before anything is switched on. The scope also records who owns each source, since a connector is only as trustworthy as the feed behind it and someone must decide whether a source is allowed in at all.

The build follows a fixed sequence. DeployOpen first confirms the platform and its dependencies are running in the client's infrastructure: Elasticsearch or OpenSearch, Redis, RabbitMQ and S3-compatible storage at the versions OpenCTI documents. Worker capacity is then set for the planned connector count, with the documented guidance of at least one platform node for every three workers used as a starting point rather than a ceiling. Connectors are enabled one at a time, not in bulk. Each one is run against a test scope, its import volume and object counts are measured, and deduplication behavior is checked by re-running the same feed and confirming that existing entities are updated rather than duplicated. Only then is the connector moved to its scheduled interval. Connectors that produce noisy or low-value data are either filtered or left off, and the decision is recorded in the scope.

Handover is a set of documents and results the client keeps. These include the connector inventory (name, source, connector type, schedule, owning team and the worker pool it runs on), runbooks for enabling, disabling and re-running each connector and for diagnosing a stalled queue, and a credential register listing every API key or service account, where it is stored, who can rotate it and when it was last changed. The verification results show each acceptance check with the date it was run and the outcome. Clients should ask for the credential register to hold references to secret stores, not the secrets themselves, and should confirm that the runbooks name the people who can approve a new source.

Ongoing work is optional and usually sold as a retainer. Common retainer items are monitoring of connector health, rotation of credentials, review of data volume as new sources are added, and testing of connectors against new OpenCTI releases before an upgrade. Whether a retainer is needed depends on how many sources the team expects to add and how quickly the platform will change. DeployOpen's work is normally fixed scope: scope, deliverables and timelines are agreed in writing before the build starts, and a quote follows from that written scope. Clients should ask for the quote to list each connector, each acceptance check, each handover document and any retainer hours separately, so the cost of each source is visible when they decide what to add next.