Post Mortem – Scheduled workflows not executed (7–12 October 2026)

22 Oct 2026 · @Daan (Production Issue Coordinator)

Summary

Between the evening of Wednesday 7 October and the evening of Monday 12 October 2026, no scheduled workflows were carried out in Luscii for any organisation. This affected scheduled status changes, scheduled programme, group or protocol changes, and scheduled automatic messages to patients.

The cause was a small data inconsistency introduced during a planned move of the Luscii platform to a new hosting environment on 7 October. A handful of scheduled actions pointed to programme content that no longer existed (they had been deleted before the migration). The component that runs scheduled actions stopped at these "deleted" workflows and could not get past them, so nothing behind them in the queue was processed. The problem was reported to us by a customer on Monday 12 October and fixed the same evening.

As a direct consequence of the fix, all 4,630 delayed scheduled workflows were carried out at once on Monday evening. This was an unintended side effect, not a conscious decision. We recognise that it created additional clinical risk and extra work for care teams, and we are changing our process so that the clinical and customer impact of a fix is assessed, and customers are consulted, before a fix of this kind is applied.

Timeline

Impact

An overview of the affected programmes per organisation has been shared with each organisation by its Impact Specialist.

Clinical impact

• During the five days, patients did not receive their scheduled messages, status changes or protocol/programme/group changes.

• After the fix, all delayed actions were carried out at once. Patients received messages, two to five days later than intended and in a context that no longer matched their situation (for example, advice tied to a specific moment in a treatment programme). This additional risk was caused by the way the fix was applied, not by the original fault.

The issue was classified as a Blocker (priority 1) when it was found, and this classification was confirmed after the fix.

What we heard from you

Customers told us that patients receiving a call to action days later, in a completely different context, was not acceptable and created extra work for care teams. Customers also asked how we decide which actions to take during an incident, and asked to be consulted before such decisions are made in future. Both points are addressed in the improvements below.

Cause

What went wrong

A scheduled action in Luscii is essentially a timer: it records when something should happen and points to the programme content that defines what should happen. During the move to the new hosting environment on 7 October, a small number of these timers were copied across while the content they pointed to was not (a deletion that had not yet been carried over at the moment of the switch). Under normal operation our system makes this situation impossible, but those safeguards had to be temporarily switched off to move the data across. This type of move is very rare.

When the component that runs scheduled actions reached one of these broken timers, it stopped. It then restarted from the beginning of the queue, reached the same timer, and stopped again. As a result, no scheduled action was completed and nothing behind the broken entries was ever carried out.

Why it took five days to notice

The errors were visible in our monitoring from the first evening. We did not act on them, for three reasons:

• Our error monitoring contained a lot of recurring, low-priority noise, so a new error did not stand out.

• The first burst of errors on 7 October was investigated and marked as resolved. When the error returned later that evening, it was registered as a repeat of a closed issue and was not re-flagged to the team.

• Responsibility for watching this monitoring was not clearly assigned, and the problem started on a Wednesday evening ahead of the weekend. In hindsight, it should have been caught on Thursday.

How we resolved it

On Monday evening 12 October, our developer removed the broken timers that could not be processed. Scheduled actions immediately started running again. No workaround was available to customers during the five days.

The unintended consequence

Once the blockage was cleared, the system worked through all 4,630 delayed actions within about twelve minutes. This included scheduled messages to patients, which were delivered two to five days late and in a context that no longer matched the patient's situation.

Releasing this backlog was not a conscious decision. It was a technical consequence of the fix that the people involved were not aware of at that moment. At the time of the fix:

• it was not yet certain that this was the full cause of the problem;

• the team believed that only status changes and programme changes were affected, and not messages. This turned out to be wrong;

• it was after office hours and the issue was a priority-1 Blocker with a correction-time commitment.

Our clinical risk assessment and customer impact roles were not consulted on how the fix would be applied. In hindsight they should have been, and the improvements below address this directly.

Response times

The report reached us through the 2nd line support form. For urgent issues, calling Luscii Support remains the fastest route and is the channel we ask customers to use (this is also mentioned in the 2nd line support form).

What we are changing

The improvements below follow from our internal retrospective of 15 October. They cover four themes: noticing problems sooner, preventing this type of fault, protecting patients from overdue content, and involving the right people before a fix is applied.

Noticing problems sooner

• Clean up our error monitoring so that recurring, low-priority errors are resolved or silenced and new errors stand out.

• Make sure that an error which returns after being marked as resolved is flagged to the team again.

• Assign clear ownership for watching error monitoring, so it no longer depends on an individual developer happening to notice.

Preventing this type of fault

• Change the component that runs scheduled actions so that one broken entry can no longer stop all other scheduled actions.

• Carry out data consistency checks before and after any large data move: compare the source with the moved data and verify that all safeguards are back in place. This applies explicitly to the upcoming move of our clinical engine.

• For large technical changes, define up front what success looks like, what we will monitor afterwards, for how long, and what counts as done.

Protecting patients from overdue content

• Investigate an expiry date on scheduled messages, and other safety nets in the product, so that overdue clinical content is never sent automatically.

Involving the right people before a fix

• Add a clinical risk assessment step between identifying the cause and deciding on the fix. This runs in parallel with the technical investigation, also for Blockers outside office hours, and includes a check-in with the customer impact role.

• Align our impact and communication roles on priority and urgency during priority-1 and priority-2 issues, including outside office hours.

• Set up a dedicated internal coordination channel for every priority-1 and priority-2 issue, with the Production Issue Coordinator accountable for doing so.

Clinical safety review and questions

This post mortem was shared with our External Clinical Safety Officer on 15 October 2026. Their feedback has been received and did not lead to additional improvements beyond those listed above.

If you have questions about this incident or about the impact on your organisation, please contact your Impact Specialist or Luscii Support.

We have now personally informed all affected organisations. The incident was caused by a technical change we made on 7 October. Because all delayed actions were executed at once after the fix, patients also received some scheduled messages later than planned, at a moment when the content no longer always matched their situation.

We have carried out a thorough internal review and defined improvement actions. We understand that this incident has affected your patients and care teams, and we sincerely apologise. We take this very seriously and are fully committed to preventing it from happening again. Thank you for your patience and for the valuable feedback you have shared with us.

We have collected all data on the affected patients. For each patient, we can see which programme and protocol they are in, when the workflow should have run and the total delay. For each workflow, we can also see what it should have done: send a message, or change programme, protocol or group. No workflow actions were lost. Due to an error they were queued, and they were all executed yesterday evening. Depending on the workflow, we ask you to assess which patients need additional action.

We understand the seriousness of this situation and apologise for the inconvenience. We are now starting to send the data to each organisation and will publish an update once this is complete.

The issue has been resolved. It concerned automatic protocol changes and patient status changes. Scheduled messages were sent as expected. All workflows that should have run during the affected period have now been processed. We have identified the patients for whom scheduled changes were delayed. Depending on the type of programme, an additional manual check may be needed for these patients. We will share advice on this tomorrow morning, followed later in the day by the details of the affected patients.