How ASX’s blockchain dream went up in smoke
The review by Accenture that killed the DLT project details the problems behind ASX’s CHESS replacement.
When Australia Stock Exchange (ASX) appointed Helen Lofthouse as its new CEO in August this year, the first thing she did was appoint the consultancy Accenture to review the project to replace the exchange’s post-trade system, CHESS.
Lofthouse (pictured, center) announced last week the results of the report, and the end of the CHESS replacement project, which was supposed to shift Australia’s securities market to a platform based on distributed-ledger technology.
Scrapping the project also meant taking a A$245-255 million ($163-169 million) write-down.
This ends ASX’s dream of using blockchain technology to leapfrog other markets while reducing costs across the lifecycle of securities trading.
What went wrong and what can other institutions learn?
The story began in 2015 in New York, where a respected and charismatic J.P. Morgan executive, Blythe Masters, joined Digital Asset Holdings (DA), then one of several new DLT vendors.
These companies were pitching financial institutions on the glories of the newfangled blockchain. But instead of exposing institutions to the risk and volatility of Bitcoin, they would design closed, clubby systems using distributed-ledger technology to provide benefits such as instant settlement.
Masters (pictured, right) was a regular on stage worldwide, got herself on the cover of Bloomberg BusinessWeek, and became the face of enterprise blockchain.
One of the people she charmed was Elmer Funke Kupper, the CEO of ASX. Funke Kupper was leaving the exchange, and his successor, Dominic Stevens, was looking to make his mark. He had a challenge, wanted to do something bold, and decided blockchain was it.
ASX prided itself on being the first global securities exchange to dematerialize, in the 1990s. That project had led to CHESS, for post-trade processing of cash equities. It went live in 1996, at the dawn of the Internet era, and without the database connectivity that would become best practice. By 2015 the system was very much out of date. Why not replace it with yet another pioneering move?
In 2016, ASX announced it would be replacing CHESS with a DLT platform built by Digital Asset.
Stevens (pictured, left) did more than just hire DA: he took an 8 percent stake in the vendor and a board seat (since diluted after DA conducted other funding rounds), with a vision of being part of the technology that would redefine securities markets worldwide.
For Australia, ASX and DA said the replacement platform would cut costs for fund managers, brokers and custodians because it would eliminate the need for reconciliation, which is the lengthy and often manual process of ensuring everyone’s numbers match. The ASX’s system would synchronize and update everyone’s ledgers automatically. This in turn would create shared data that would enable participants to design new business models, such as transparent reporting.
The news set off a media firestorm in Australia, which led ASX officials to downplay the project’s radical side. ASX deputy CEO Peter Hiom assured the Australian Financial Review and others that the project was simply about “enhancing database architecture”. (Hiom represented ASX on the DA board.)
More importantly, the move angered many domestic participants, for whom upgrading to the DLT involved high costs and plenty of uncertainty. ASX responded by assuring them it would also allow users to integrate via more traditional means, using an upgraded ISO20022 standard for messaging.
- Read more:
- More to ASX’s DLT extension than Covid-19 chaos
- FinClear launches DLT-based private-markets platform
- Yuval Rooz wants language to connect blockchains
This concession, however, helped doom the project. DLT’s benefits derive from mutualizing processes, but if most domestic firms chose to stick with traditional ways, then the system would never gain the mass adoption it needed.
Initially the ASX said the replacement would be completed by 2020. That deadline began to slide almost immediately.
The first blow came in 2018 when Masters abruptly quit DA under murky circumstances. She claimed at the time she needed more time with family, but she immediately went on to amass board directorships and the following year launched Motive Partners, a private-equity firm.
Her departure coincided with DA making a strategic shift, pivoting from a blockchain vendor such as rivals R3 or Hyperledger, and focusing on writing smart contracts using its own computer language, Daml.
This shift led ASX to bring on VMWare to build the settlement layer of its DLT. Digital Asset would remain the primary partner but under a more technical, less glamorous leadership. And now its most important project involved three players instead of two.
COVID and crashes
The second blow occurred in spring 2020, when the COVID-19 pandemic swept the globe. This created two stresses on financial institutions. First, they had to scramble to enable remote working. Second, February through April witnessed record-breaking market volatility.
In March, ASIC, the securities regulator, begged high-volume stock traders to cool their activity, for fear it would bring down ASX’s systems. By now the replacement program was four years running, with little to show. CHESS wasn’t getting any younger but ASX hadn’t invested in shorter-term measures to support the tech.
This represented an existential threat to the entire securities industry. ASX holds a monopoly on clearing and settlement in Australia, so even rival exchanges like Cboe (previously Chi-X) and Sydney Stock Exchange outsource post-trade processing.
ASIC’s nightmare looked plausible: in November 2020, ASX’s trading system crashed, although CHESS was not interrupted.
By now Dominic Stevens’ vision of cutting costs had led to costly overruns. Peter Hiom was suggesting that ASX’s clients should stump up to pay for the CHESS replacement. Industry players were wondering why CHESS hadn’t been simply upgraded with a modern relational database, while Digital Asset and VMWare didn’t have actual systems to put into production.
Many local players told DigFin they struggled to understand the smart contracts being written, and that COVID meant they had fires to put out and needed more time to digest information. So while the industry badly needed CHESS replaced, it was also asking ASX to slow things down.
Dominic Stevens stepped down as CEO in February, leaving the CHESS replacement project two years behind schedule and, according to local media, five times over its original budget (ASX disputes the claim that the project was over budget). He had managed to deliver good performance for ASX shareholders, but of course ASX is a quasi-monopoly.
Critics said he was refusing to clean up the mess he had made with the CHESS replacement; Stevens said the exchange could do with fresh thinking, but that the project was still on track to go live in April 2023 – a claim repeated by Damian Roche, the chairman of ASX.
His departure saw the ASX’s own stock decline by 3.8 percent that day, wiping out A$640 million of value – a testament to market fears that the ASX was headed for a big write-down, and the possibility that regulators will want to end its monopoly on clearing and settlement.
Although the leading candidate to replace Stevens was Tim Hogben, an ASX lifer in charge of securities and payments who was also day-to-day leader of the replacement project, they chose Helen Lofthouse. She is an ex-UBS and ex-J.P. Morgan banker with experience in Europe and political ties in Canberra. She had joined ASX in 2015 to run its markets business, sufficiently removed from the CHESS debacle.
When she appointed Accenture to review the replacement project – which was now being delayed again, to 2024 – she left all options on the table.
Accenture defined its mandate as looking at the project’s architecture, design, implementation practices, ways of working and project governance.
It identified several problems.
Latency: distributed systems mean slower transaction processing, especially as data has to flow twice, from the client’s node to the ledger and back.
Concurrency: On the one hand, concurrency (letting processes operate simultaneously) can help scale processing. But it becomes a problem when multiple trades involve the same data set, such as a security identifier, or the same broker. ASX and DAH sought to deal with this by introducing batch processing, which is a backward step and doesn’t deal with all situations where data is in potential conflict.
Batch processing, aka transaction grouping: Putting lots of transactions together for a single batch of processing. Accenture says this not well tested and faces constraints, such as practical limits in Daml’s APIs or VMWare’s transaction messaging.
The conclusion is, under this setup, ASX would struggle to update transactions focused on the same firm or security on its DLT.
Accenture also says the current design is contributing to challenges in achieving scalability, resiliency and supportability. Workflows weren’t being tailored for a DLT environment, designs were haphazard, and adding new functionality would require major tech work and migrations of core systems and APIs.
More fundamentally the consultant also questioned the need for a decentralized solution when ASX is clearly the central market operator.
Accenture praised the quality of Daml, noting the language’s smart contracts could still be used effectively. But the design connecting ASX and other parties using smart contracts needed a lot more thinking to avoid the need for costly adjustments in the future.
Any system needs constant maintenance and upgrades, and Accenture identified this as a vulnerability: the coding required is very specialized and few programmers are available to do it.
That risk looms even larger given the next area of focus for Accenture. It found that the client/vendor coordination between ASX and DA was poor. Teams were not aligned, communication was absent or unclear, and the two parties had different ideas of delivery, scope and resourcing.
These strategic problems led to day-to-day struggles around testing, reporting outcomes, and using metrics. The staff at ASX and DA expressed a desire to work together, but frustration of their siloed cultures, which Accenture says can only be overcome with strong leadership.
But Accenture found several critical problems that it did not think could be salvaged. These include holding data (the common data model used across workflows such as settlements), which was at risk of outages; as well as concerns regarding batch and bulk processing.
Accenture didn’t outright recommend scrapping the DLT approach but it argued ASX needs to reconsider its long-term strategy, better understand DLT’s role in the overall technology stack, and perhaps modernize other parts of the process using smart contracts so as to take greater advantage of what DLT offers.
The project as it stands, however, is dead: ASX wouldn’t have accepted a material loss otherwise.
Lofthouse did leave the door ajar: “To be clear,” she said announcing the project’s halt, “the derecognition charge [that is, ASX’s write-off of its investment in the CHESS replacement program] reflects the uncertainty of the future value of the current solution design. It does not prevent us from using parts of what we have already built if we determine there are adjustments we could make to our current design, which will enable it to meet ASX’s and the market’s high standards.”
She has appointed Tim Whiteley as project director for the next phase of replacing CHESS. He is not a blockchain person; he is a bank tech-project leader, with long stints at Commonwealth Bank and Westpac.
ASX chair Damian Roche said, “We have concluded that the path we were on will not meet ASX’s and the market’s high standards. There are significant technology, governance and delivery challenges that must be addressed.
“On behalf of ASX, I apologize for the disruption experienced in relation to the CHESS replacement project over a number of years.”
ASX’s deputy CEO and DA board member Peter Hiom left the exchange in July 2021, in the wake of its systems crashes. That month he got a new job: partner at Motive, Blythe Master’s PE shop.
Dominic Stevens’ retirement package wasn’t announced, but he was earning A$4 million in salary. He recently bought a A$21 million waterfront mansion in an affluent Sydney suburb, next door to Lachlan Murdoch.