Luscii - Notice history

This incident has been resolved.

The Post-Mortem will be shared via this page.

We have rolled back to the previous stable version in the production environment and are closely monitoring the situation. Today’s “Overdue” alerts are being generated correctly again.

Unfortunately, we cannot retroactively generate the alerts that were missed on Wednesday afternoon and Thursday.

In addition, we will proactively reach out to organizations with more than 100 missed alerts: today, over the weekend, and no later than Monday.

If your organization has not received an email by Monday evening, you can assume there were fewer than 100 missed alerts. You are, of course, still welcome to request the data manually via support.

We will continue to monitor the situation closely.

We are working on a solution, which is currently being thoroughly tested.

Missing “Overdue” alerts can be requested via support@luscii.com. Please specify which program(s) you would like the data for.

Summary

On January 14, 2026, we introduced an infrastructure change related to logging (recording technical events). This change was not compatible with our Vitals service, causing some requests to fail. As a result, about 10% of requests during the incident window returned an error in the application.

Importantly: no patient alerts were missed, and no patients were left unmonitored due to this incident.

Timeline

Incident window: Jan 14, 2026 15:24 → Jan 15, 2026 18:15

• Jan 14, 2026 15:24 – Released to production

• Jan 14, 2026 17:45 – Internal signal: increased error rate

• Jan 15, 2026 09:28 – Reported and escalated as urgent

• Jan 15, 2026 09:52 – Hotfix 1 deployed

• Jan 15, 2026 14:27 – Hotfix 2 deployed

• Jan 15, 2026 18:15 – Rollback to a stable version (issue resolved)

Impact

What users experienced

• During the incident window, approximately 10% of requests failed and displayed a frontend error.

• All organizations were potentially affected (broad scope), though impact varied by usage and timing.

Severity classification

• Critical (because it caused visible errors and degraded stability), despite no clinical harm.

Workaround

No user-side workaround was available. Resolution required changes on our side.

Cause

The issue originated from a change in how logs were sent through our infrastructure. Our Vitals service could not process the new format correctly.

A secondary effect amplified the problem:

• When a request failed, our system attempted to log the failure.

• Because logging was already broken, that logging attempt failed too.

• This created a loop of repeated failures, increasing error volume, degrading performance, and reducing visibility via lost log data.

This resulted in:

• Repeated “broken pipe” logging errors per request

• Degraded application performance

• Partial loss of log data during the window

Resolution

• We rolled back to the last stable production version.

• We also removed the mechanism that allowed logging failures to trigger cascading/recursive errors in VitalsAPI v3.29.102.

Communication

External

• Impacted customers were contacted.

• Status was updated on InStatus.

Internal

• Relevant roles and Slack channels were informed during the incident.

Improvements (prevention)

1. Broader and faster internal signaling

   When potential production issues are detected, we will notify multiple roles earlier via shared channels (e.g., product/operations).

2. Reduce alert fatigue with clearer accountability

   We will update alert ownership and response expectations so alerts are consistently actioned.

3. Improve status communication tooling

   We will evaluate using a Datadog status page as a replacement for Instatus to improve reliability and speed of external updates.

To fix this issue permanently an maintenance window will be introduced, at 17:15 UTC our systems will be unavailable for a small window of time (expected to be approximately 20 mins) so we can roll back some infrastructure changes. After this window its expected that all issues will be resolved.

Unfortunately, the issue has returned.

The previous fix had a temporary effect, but the problem builds up over time. That’s why it seems resolved at first, but comes back later.

Because this only happens under heavy user activity, we’re unable to fully reproduce it in our test environment, which doesn’t reflect the scale of real usage.

We’re now working on a new type of approach to solve this more permanently. More updates will follow soon.

The new fix has just been deployed.

We expect this to resolve the issue completely.

We’re closely monitoring the situation to make sure everything remains stable.

Questions? Feel free to contact support@luscii.com or support@luscii.co.uk

We’ve found the cause of the issue.

• A recent change in how error messages are handled inside our system caused it to become overloaded. This triggered new errors and slowly brought the problem back.

• Because it builds up gradually, the system seemed fine at first but started failing again later.

Our team is working on a fix now. As soon as it’s ready and thoroughly tested, it will be deployed. We’ll keep you informed.

We’re seeing that the issue with loading data in Luscii has returned in some areas (web-version, HiX).

• Data doesn’t always load immediately and may show an error.

• In many cases, refreshing the page helps as workaround.
  

We understand this is inconvenient during your work. Our team is actively investigating again and working on a stable fix. Submitted measurements are still processed correctly.

The issues on the Luscii platform have now been resolved:

• Pages in the web environment and integrations sometimes didn’t load properly, this now works as expected.

• Patients sometimes saw an error when sending in measurements via the app and had to retry.

• Successfully sent measurements were correctly processed in the alert generation system. No alerts were missed on send in measurements.

Currently, there may still be a slight delay in automatic workflows and in how fast measurements appear in the app or dashboard. This is common during peak usage and recovers automatically.

A Post-Mortom will be published.

This morning Luscii had issues like slow loading and error messages. These were caused by a technical problem in the system.

A fix has now been implemented. Luscii is already performing better: pages load faster and unexpected errors are decreasing fast.

We are actively monitoring the situation.

Update: Patient creation issue: we’re working on a fix

The issue with creating new patients is caused by part of our system (ActiveMQ) being full. We are currently cleaning this up to restore normal functionality.

Other components are not affected by this issue.

If you have any questions, please contact support@luscii.com