End-of-day reconciliation processes create data challenges that are constraining buy-side firms from achieving efficiencies from new technologies and AI initiatives.
The challenges posed by reliance on long-established reconciliation processes come as the buy-side undergoes a transformation of its operating models to accommodate new data management, investment and settlement strategies.
This challenge was highlighted in a recent study, which found that more than two-thirds of buy-side firms still rely mainly on end-of-day reconciliations, and about half said data mismatches and timing differences were the chief cause of discrepancies in trade flows. The survey of senior buy-side leaders was conducted at a round-table gathering convened by data solutions firm Smartstream.
The data reconciliation challenge facing buy-side institutions has been made more acute as they have sought to move to multi-asset trading, as regulatory scrutiny has intensified, and as they are increasingly expected to move to a T+1 settlement cycle.
New efficiencies
As improved research and trading technologies make it more difficult for them to compete on investment returns, firms are trying to compete on fees. To do so, they have sought to find operational efficiencies to improve throughput and reduce costs. One way of doing that has been to outsource their data acquisition processes, but that is also weighing on their operations, said Robin Hasson, head of reconciliation at Smartstream.
“The data challenges in reconciliation tend to be the breadth of data, complexity, and inconsistency,” Hasson told Data Management Insight.
“There are far more data points in modern formats, and if everyone used them the same way, things would be much simpler. But that’s not the reality.”
The Smartstream survey, entitled “Smart Reconciliations: The Buy-Side Perspective”, found that 59 per cent of respondents said external data dependencies were their primary risk.
“The more synchronisation you need between systems, the greater the risk, the more fragile things become, and the higher the cost of change,” said Hasson.
More use cases
Reconciliations are important in providing checkpoints for data entering and moving through an organisation’s systems. As the use cases for data have multiplied, reconciliations now need to work across entire organisations – not just the trading activity – as information is transferred from team to team. That has to happen in real-time and without loss of data integrity, adding pressure on data chiefs to get the process right.
The most critical need for ensuring data quality is so that it can be safely used to train and power AI models – which can generate erroneous outputs if the data fed into them isn’t accurate and trusted.
“That’s become one of the most important considerations,” Hasson said. “If you have access to high-quality, well-described data, your AI models can learn what the data really represents. That becomes truly insightful and can drive significant business benefits.”
Accumulating challenges
Executing those vital checks at the end of the day, however, means data irregularity and breaks can accumulate beforehand, creating workflow difficulties and errors. AI solutions are helping accelerate and automate the processing of issues, preventing inconsistencies which could cause damage before the end-of-day reconciliation.
The solution to this is an always-on reconciliation strategy, Hasson said. A control layer over the data management architecture can help identify and rectify irregularities and breaks immediately, and lessen the risks they pose.
“As soon as you spot a breakdown in data quality, as soon as it starts to drift, you can address it,” said Hasson.
“With AI solutions, you can be far more proactive, often spotting pattern changes and identifying risks before breaks occur. You can see deviations in data flows and recommend how to correct them.”
Centralised systems
The traditional backbone of reconciliations in most use cases has been master data management, which creates a single source of truth. This has become more important as organisations have begun moving to a centralised data management strategy to better oversee their increasingly complex data processes.
“There is a drive towards standardising data storage so you have a single source of truth. Not only to support AI model training and audit, but also to drive insights, trend analysis, and broader business intelligence,” said Hasson.
That has traditionally been hobbled by fragmented legacy data architectures. This isn’t as acute a problem for some operations, but consolidating disparate systems has generally been the rule of thumb for data managers. While AI can help by more efficiently linking those independent structures using Model Context Protocol (MCP) tools, it is no substitute for full-on modernisation, said Hasson.
“It may be masking the underlying problem,” he said. “If you solve issues tactically on a case-by-case basis, is that really improving operational control over the next two to three years? The answer is: probably not. The quality achieved will never match that of a strategic data programme.”
