Post Mortem – Scheduled workflows not executed (7–12 October 2026)

22 Oct 2026 · @Daan (Production Issue Coordinator)

Summary

Between the evening of Wednesday 7 October and the evening of Monday 12 October 2026, no scheduled workflows were carried out in Luscii for any organisation. This affected scheduled status changes, scheduled programme, group or protocol changes, and scheduled automatic messages to patients.

The cause was a small data inconsistency introduced during a planned move of the Luscii platform to a new hosting environment on 7 October. A handful of scheduled actions pointed to programme content that no longer existed (they had been deleted before the migration). The component that runs scheduled actions stopped at these "deleted" workflows and could not get past them, so nothing behind them in the queue was processed. The problem was reported to us by a customer on Monday 12 October and fixed the same evening.

As a direct consequence of the fix, all 4,630 delayed scheduled workflows were carried out at once on Monday evening. This was an unintended side effect, not a conscious decision. We recognise that it created additional clinical risk and extra work for care teams, and we are changing our process so that the clinical and customer impact of a fix is assessed, and customers are consulted, before a fix of this kind is applied.

Timeline

Impact

An overview of the affected programmes per organisation has been shared with each organisation by its Impact Specialist.

Clinical impact

• During the five days, patients did not receive their scheduled messages, status changes or protocol/programme/group changes.

• After the fix, all delayed actions were carried out at once. Patients received messages, two to five days later than intended and in a context that no longer matched their situation (for example, advice tied to a specific moment in a treatment programme). This additional risk was caused by the way the fix was applied, not by the original fault.

The issue was classified as a Blocker (priority 1) when it was found, and this classification was confirmed after the fix.

What we heard from you

Customers told us that patients receiving a call to action days later, in a completely different context, was not acceptable and created extra work for care teams. Customers also asked how we decide which actions to take during an incident, and asked to be consulted before such decisions are made in future. Both points are addressed in the improvements below.

Cause

What went wrong

A scheduled action in Luscii is essentially a timer: it records when something should happen and points to the programme content that defines what should happen. During the move to the new hosting environment on 7 October, a small number of these timers were copied across while the content they pointed to was not (a deletion that had not yet been carried over at the moment of the switch). Under normal operation our system makes this situation impossible, but those safeguards had to be temporarily switched off to move the data across. This type of move is very rare.

When the component that runs scheduled actions reached one of these broken timers, it stopped. It then restarted from the beginning of the queue, reached the same timer, and stopped again. As a result, no scheduled action was completed and nothing behind the broken entries was ever carried out.

Why it took five days to notice

The errors were visible in our monitoring from the first evening. We did not act on them, for three reasons:

• Our error monitoring contained a lot of recurring, low-priority noise, so a new error did not stand out.

• The first burst of errors on 7 October was investigated and marked as resolved. When the error returned later that evening, it was registered as a repeat of a closed issue and was not re-flagged to the team.

• Responsibility for watching this monitoring was not clearly assigned, and the problem started on a Wednesday evening ahead of the weekend. In hindsight, it should have been caught on Thursday.

How we resolved it

On Monday evening 12 October, our developer removed the broken timers that could not be processed. Scheduled actions immediately started running again. No workaround was available to customers during the five days.

The unintended consequence

Once the blockage was cleared, the system worked through all 4,630 delayed actions within about twelve minutes. This included scheduled messages to patients, which were delivered two to five days late and in a context that no longer matched the patient's situation.

Releasing this backlog was not a conscious decision. It was a technical consequence of the fix that the people involved were not aware of at that moment. At the time of the fix:

• it was not yet certain that this was the full cause of the problem;

• the team believed that only status changes and programme changes were affected, and not messages. This turned out to be wrong;

• it was after office hours and the issue was a priority-1 Blocker with a correction-time commitment.

Our clinical risk assessment and customer impact roles were not consulted on how the fix would be applied. In hindsight they should have been, and the improvements below address this directly.

Response times

The report reached us through the 2nd line support form. For urgent issues, calling Luscii Support remains the fastest route and is the channel we ask customers to use (this is also mentioned in the 2nd line support form).

What we are changing

The improvements below follow from our internal retrospective of 15 October. They cover four themes: noticing problems sooner, preventing this type of fault, protecting patients from overdue content, and involving the right people before a fix is applied.

Noticing problems sooner

• Clean up our error monitoring so that recurring, low-priority errors are resolved or silenced and new errors stand out.

• Make sure that an error which returns after being marked as resolved is flagged to the team again.

• Assign clear ownership for watching error monitoring, so it no longer depends on an individual developer happening to notice.

Preventing this type of fault

• Change the component that runs scheduled actions so that one broken entry can no longer stop all other scheduled actions.

• Carry out data consistency checks before and after any large data move: compare the source with the moved data and verify that all safeguards are back in place. This applies explicitly to the upcoming move of our clinical engine.

• For large technical changes, define up front what success looks like, what we will monitor afterwards, for how long, and what counts as done.

Protecting patients from overdue content

• Investigate an expiry date on scheduled messages, and other safety nets in the product, so that overdue clinical content is never sent automatically.

Involving the right people before a fix

• Add a clinical risk assessment step between identifying the cause and deciding on the fix. This runs in parallel with the technical investigation, also for Blockers outside office hours, and includes a check-in with the customer impact role.

• Align our impact and communication roles on priority and urgency during priority-1 and priority-2 issues, including outside office hours.

• Set up a dedicated internal coordination channel for every priority-1 and priority-2 issue, with the Production Issue Coordinator accountable for doing so.

Clinical safety review and questions

This post mortem was shared with our External Clinical Safety Officer on 15 October 2026. Their feedback has been received and did not lead to additional improvements beyond those listed above.

If you have questions about this incident or about the impact on your organisation, please contact your Impact Specialist or Luscii Support.

Inmiddels hebben we alle organisaties die geraakt zijn persoonlijk geïnformeerd. De oorzaak van het incident lag in een technische wijziging die we op 7 oktober hebben doorgevoerd. Doordat alle uitgestelde acties na de fix in één keer zijn uitgevoerd, ontvingen patiënten sommige ingeplande berichten later dan gepland, op een moment dat de inhoud niet altijd meer aansloot.

We hebben intern een uitgebreide evaluatie uitgevoerd en verbeteracties opgesteld. We begrijpen dat dit incident impact heeft gehad op uw patiënten en zorgteams, en bieden hiervoor onze oprechte excuses aan. We nemen dit zeer serieus en zetten ons volledig in om herhaling te voorkomen. Bedankt voor uw geduld en voor de waardevolle feedback die we van u hebben ontvangen.

We hebben alle gegevens over de betrokken patiënten verzameld. Voor elke patiënt kunnen we zien in welk programma en protocol ze zich bevinden, wanneer de workflow had moeten draaien en de totale vertraging. Voor elke workflow kunnen we ook zien wat deze had moeten doen: een bericht versturen, of programma, protocol of groep wijzigen. Er zijn geen workflowacties verloren gegaan. Door een fout stonden ze in de wachtrij en zijn ze allemaal gisteravond uitgevoerd. Afhankelijk van de workflow vragen we u te beoordelen welke patiënten extra actie nodig hebben.

We begrijpen de ernst van deze situatie en bieden onze excuses aan voor het ongemak. We beginnen nu met het versturen van de gegevens naar elke organisatie en zullen een update publiceren zodra dit is voltooid.

Het probleem is opgelost. Het probleem betrof automatische protocolwissels en statuswijzigingen van patiënten. Het gepland versturen van berichten is zoals verwacht verlopen. Alle workflows die in de geraakte periode hadden moeten afgaan, zijn inmiddels alsnog verwerkt. We hebben inzichtelijk bij welke patiënten geplande wijzigingen waren vertraagd. Afhankelijk van het type programma is er eventueel een extra handmatige controle nodig voor deze patiënten. We komen daarover morgenochtend met een advies en later op de dag met de gegevens van de getroffen patiënten.

The issue with connecting the Omron M4 when starting an AFib measurement on Android has been resolved in app version 5.76.1. Patients using an Android device are advised to update the app to version 5.76.1. After updating, they can connect their measurement device again and complete the AFib measurement as usual. iOS users do not need to take any action. Thank you for your patience.