Open edX external graders with XQueue and Codejail

Once the event bus is enabled, every learner submission to an external grader follows the same path, and each hop is a place where it can stall. The LMS stores the submission and publishes it as an event on the bus. The learner's submission is recorded before any code runs, so the LMS does not wait on a grader in the request itself. An XQueue watcher then consumes that event and takes responsibility for grading it. Your team should trace one test submission through each of these stages, from the LMS record to the recorded score, before any learner uses the course. Confirm the exact queue and topic names in the official how-to, because the watcher configuration must match them exactly.

The watcher hands the learner's code to the grader, which executes it inside Codejail, the sandbox that runs untrusted code separately from the rest of the grading process. The Tutor plugins index lists a Codejail plugin by eduNEXT for enabling this execution, and an Xqueue plugin by eduNEXT for external grading. Before you rely on either, confirm which plugin versions match your Open edX release and how the sandbox is configured in your environment. The watcher authenticates to the platform as a Django superuser in the xqueue group, which means its credentials carry broad privileges. Treat that account as a high-value identity and decide in advance where its credentials are stored and who can read them.

After grading finishes, the watcher posts the result back through the edx-submissions API, and the LMS records the score against the learner's submission. This final hop is where a failure is easiest to miss. Because the LMS stores and publishes the submission before any watcher reads it, a submission can sit in the record as received while no watcher is consuming the event. The learner sees a pending result, the LMS shows no error, and the fault sits on the consumer side of the bus. Plan monitoring for the watcher itself, covering whether it is running, whether it is connected to the bus, and how many events are waiting for it. Alert on events waiting for a consumer so that a stalled queue does not surface first through learner complaints.

Before go-live, your team should answer questions that the documentation leaves to the official how-to and to your own environment. Which broker carries the events in your deployment, and which consumer group does the watcher join? What happens to a message when grading raises an error: is it redelivered, retried or parked? How does a failed grade appear to the learner and to course staff? Who owns the watcher service, its restart behavior and its credentials? Record the answers in the runbook for the course team. Then run a failing test submission on purpose, so the failure path is tested before a learner reaches it.

Test the whole chain before any learner submits code. The chain runs from the LMS problem, through the event bus, the XQueue watcher and the Codejail sandbox, to the result stored through edx-submissions and the score in the gradebook. A small set of cases covers the failure modes that matter. Write the expected outcome of each case down before you run it, so a pass means the system matched a target you set in advance, not a target you adjusted after seeing the output. Keep the test course separate from any live course so test scores never reach real learner records.

Start with a known-good submission and a known-bad one. The known-good answer should receive full credit, and the score should appear on the learner's progress page and in the instructor gradebook. The known-bad answer should receive the score your problem definition assigns, usually zero or partial credit. The course team writes the test cases, and a reviewer who can judge expected output checks each one independently. Ask that reviewer to cover edge cases such as empty input, extra whitespace and unusual formatting, because a grader that passes the obvious cases can still fail these in production.

Next, submit code that loops forever and code that uses far more memory or time than the problem allows. Codejail should stop both within the limits configured in your deployment. Confirm those limits in your own environment rather than assuming defaults, and check the eduNEXT Codejail plugin documentation for which controls it exposes. The learner should see a clear error, and the watcher should keep consuming the queue. Submit a normal answer straight afterward. If it is graded correctly, the sandbox recovered and the watcher was not left blocked by the hostile submission.

Then test recovery. Stop the XQueue watcher, submit an answer as a learner, and confirm the submission is stored and shown as pending rather than lost or silently dropped. Start the watcher and confirm the queued submission is graded and its score appears once, with no duplicate entries in the gradebook. During this test, confirm which service account the watcher uses. It authenticates as a Django superuser in the xqueue group, so verify that its credentials are stored securely, that access is limited to what the watcher needs, and that rotation is part of your operating procedure.

Finally, check the logs. Search the LMS, watcher and sandbox logs for the source code and output strings from your test submissions. Only the fields your logging policy permits should appear, such as submission identifiers, timestamps and scores. Confirm the exact list with the person who owns the policy, then record the results with the test date and the reviewer's name. Repeat this check after every configuration change, since a debug setting can start writing learner code to logs without any visible error.