We are pleased to report that the data reconciliation is complete with all missing data now available in the Measure Effect & Insights platform. We have performed all relevant checks and QA in the system and can confirm that all dashboards are now up to date.
The data and incident management team will carry out a full post mortem on this issue to understand lessons learned and how we can improve moving forward,
If you have any questions or queries regarding this incident, please reach out to me directly.
Kind Regards, Jason
Posted May 27, 2024 - 17:09 BST
Monitoring
Afternoon All,
Please be advised that we have now reconciled missing data from the 21st to 24th May within the Measure Effect/Insights platform.
The data team will continue to work through the reconciliation with a view for this to be completed tomorrow: Monday 27th May at 16:00 UK Time
If you have any questions, concerns or issues, please reach to me directly.
Regards, Jason
Posted May 26, 2024 - 18:57 BST
Update
Afternoon All,
Please be advised that the data team are still working through the reconciliation process
Incident Name: Delays in Data being received through to Measure Effect Data Affected: Monday 20th May to Thursday 23rd May
Due to the size of the data set, this will take a few days to process. By Monday, missing data should start to filter through to measure effect with a view for full reconciliation to be completed by Wednesday.
If you have any questions on this, please reach out to me directly.
Kind Regards, Jason
Posted May 24, 2024 - 14:16 BST
Update
Morning All,
Please be advised that we have identified the root cause of this ongoing data log issue and are currently analysing the logs for May 22nd to finalize our findings.
Data Affected: Monday 20th May 2024
Our preliminary analysis indicates that this problem has stemmed from changes in the Flix tracking JS (schema changes), which altered the way logs and data are captured, resulting in data pipeline run failures.
Please note: After thorough analysis, we can confirm that this issue is not related to Limelight
The data team are now working to reconcile the missing data. Our next update to be provided tomorrow morning latest.
Kind Regards, Jason
Posted May 23, 2024 - 12:01 BST
Update
We are continuing to work on a fix for this issue.
Posted May 22, 2024 - 09:59 BST
Identified
Morning All,
Please be advised of the following incident flagged by our data team this morning:
Incident Name: Delays in Data being received through to Measure Effect
Description: Today, during our surveillance we found an issue related to "shopper logs data did not contain the UA-Client-Hint HTTP headers " after receiving the data from the Limelight Server. In response, we promptly halted our main ETL process. I will give an update by today 16:00 BST BST and tonight 20:00. Once the issues is sorted out by Limelight. We will start the manual build. For reference will send the impact analysis
Data Affected: Monday 20th May 2024
Actions taken: 1. The routine pipelines have been paused 2. A support ticket has been raised with Limelight 3. The Limelight team are now investigating this.
We will be sure to let you know as soon as we have received anything further back from the Limelight team If you have any questions in the meantime, please reach out to myself or Aravind in the Data Team directly.