DemandSphere
  • Status
  • Events
  • Monitors

America/New_York

powered by openstatus.dev

DemandSphere

Degraded Performance

Oct 17, 2025

Delay on reporting data

  • App

Monitoring

The secondary vacuuming process has completed successfully, so we are running full speed to catch up on all data deliverables as quickly as possible.

We will keep this incident open in Monitoring status until things are caught up.

Identified

We had caught up on most data until 10/14/2025.

A secondary process kicked in for our primary Postgres cluster needing vacuuming, which occurred due to a buildup from the events of the last few days.

This is an unavoidable process in order to avoid extended application downtime so we are experiencing a new delay while this process completes.

It is currently 80% complete, but it is not clear yet when it will finish. The vacuuming process has already been running for 24 hours.

Data is currently delayed for 10/15/2025 and we expect further delays for 10/16/2025.

Based on the backlog as a result of this, we will be working through this weekend to back fill data.

There are no issues with fetching data, it is a reporting and UI delay only.

Identified

We have completed reporting data for 10/12 and are focused on completing everything for 10/13 next. Our goal is to prioritize the most recent data first as this is what reflects client's latest website visibility.

We are aware that there are still gaps for some sites on 10/08-10/11. We will continue to backfill these as the system catches up.

Competitor Discovery and GA4 data are missing for most of the affected date range (10/08-10/13). We will backfill these once the core reporting subsystems are populating normally.

Identified

As an update, we have identified the issue (which we will explain below) that caused the delays to occur.

We have been fetching since this incident began without incident so we do not anticipate any missing data, we are working to resolve the delay in the display and availability of this data.

The following dates are still showing delayed or missing data. Once we complete the data recovery process, these dates will have complete data, there is simply a delay in their processing.

  • October 8th
  • October 9th
  • October 10th
  • October 11th
  • October 12th

On October 8th, our primary distributed job queue manager cluster began to experience errors. We made several attempts over the next few days to repair it but it became clear that we would need to perform a larger upgrade. Specifically, we added additional redundancy and created separate queue manager clusters per subsystem (divided into fetching, parsing, reporting, and ui).

This, combined with further optimization of our parsing process, has improved overall stability and performance greatly. Unfortunately, during this time, the reporting and display of this data fell behind, leading to the delays currently showing in the app. We are working to back fill and have the entirety of this data caught up as quickly as possible. We will continue to provide updates until it is resolved.

Please reach out to your Account Manager with any questions or if you need data for specific dates, in the meantime.

We apologize for this inconvenience.

The DemandSphere Team

Identified

Due to an issue with our worker processing infrastructure, there will be a partial delay on data for October 8th, and a complete delay on data for October 9th-10th.

We are making the necessary upgrades and improvements to resolve the issue and we will provide further updates as we get closer to resolution.

We apologize for the inconvenience.

App

99.85%

45 days ago

Today