Not so long ago, during my times as a Consultant / Forward Deployed Engineer (before FDE was a thing) I was part of a full blown rewrite for a mobile application, everything was being rebuilt, from the mobile client to the backend services powering and serving the applications. This was for a large retail bank here in the Middle East.

When I was starting my career, I used to ran away from Banks and Insurance Companies, I thought their technical stacks were boring, old and their ways of working were also off-dated and rigid. Little did i know I would end up enjoying this project so much.

Banks are one of the oldest organizations of the modern world, the concept of banking can be traced back to ancient Mesopotamia, where merchants gave grains to traders who traveled the world. The banking system as we know it today, started to develop during the Renaissance in medieval Italy around the 14^th century, with the Medici family being one of the most famous banking families, hence, the most influential dynasty in Florence.

Today, banks constitute and integral component of the world’s economic infrastructure providing essential services for business and individual customers to enable them to manage their finances while also being the main source of credit for businesses and consumers, contributing to the economic growth and stability.

As we can imagine, a lot of things have changed in the way banks function since the 1400’s in terms of processes and operations. The most impactful wave of changes started to happen in the middle of the 20^th century with the inception of computers around the 1950s to enable a faster and more efficient way of managing checks and other banking transactions in an automated fashion that required a tremendous manual effort before. By the end of the 50s, checks and accounts management were fully automated with the inception of the ERMA (Electronic Recording Machine, Accounting), developed at the Stanford Research Institute.

By the 1970s with the widespread of computer networks, it was possible to build interfaces accessible to the public, which enabled automated ATM operations. Technology has continued to evolve ever since, with faster and smaller machines, more efficient servers, distributed systems and more efficient software and systems architectures and, today, we have microservices and cloud computing, new banks and fintech have been taking advantage of all these modern technologies while traditional banks seem stuck to all those legacy technologies, frozen in time.

Executing a big-bang migration from the old systems installed in the 1970s to a more modern software and hardware stack would be a very expensive thing to do, not to mention a never-ending complex project due to the amount of data the banks have accumulated through decades of operations, not to mention new systems means new ways of managing the existing processes so it would require not only a technical migration but also a full-blown organizational transformation all at the same time.

I believe there needs to be a balance, a middle point which allows rigid organizations such as traditional banks dip their toes into the new waters of cloud computing and microservices without changing too much of their entire stack and on-going processes, but we need to seize the opportunities as they appear because not every project is the same and we need to assess on a case by case basis.

Mobile application revamp

During 2023 and part of 2024, I was part of a huge project to revamp a banking mobile application for a large retail bank in the Kingdom Saudi Arabia. This revamp looked more like a full rewrite for both, the mobile application itself, as well as all the backend systems powering that app. It involved a massive digital transformation and the introduction of Agile ways of working, but for the scope of this article, I'll focus only on the technical aspects of part of the implementation I was overseeing.

When the team landed at the client, we started exploring their technical landscape in terms of architecture and technical stack and iterated quickly to come up with an initial architecture to cater to our use case and leave the design open and flexible to serve other use cases in the future. The idea was to produce an omni-channel backend solution, capable to support not only the mobile app, but also other digital channels used by the customers to interact with the bank’s online services such as internet banking and phone banking.

Below, you will find a simplified version of the architecture we produced, it doesn’t show all the components by name and vendor, but it displays all the layers involved in the design:

Our build would sit on top of three foundational layers built before our engagement, hence, out of the project’s scope, in a nutshell, each out-of-scope layer did the following:

• Core Baking: our client was a very old bank, founded in the 1950s, so this layer was composed by multiple servers running very old software from the 1960s and 1970s, most of them inside one of those mythical Mainframes. This layer had most of the legacy systems running inside. The core system on the mainframe was a simple and effective code built using Cobol and potentially controlled by JCL (Job Control Language) and using file stored hierarchical structure making it a very fast machine for transactions processing or accessing requested record. 
• API Connect: it’s as an integration layer between the core banking systems and the more modern web layers sitting on top. We can think of it as an API Gateway with extended integration capabilities.
• Omni Channel: this is a set of multiple services built to integrate and translate requests from the upper customer-facing channels into something the core banking systems can understand and process as well as translating the responses from the core banking systems into something the modern web layers can interpret.

The core banking and api-connect layers were already there, and the Omni layer was built right before the project started as a way for the new backend systems for the new mobile app could speak to the core baking systems and request financial data for the users.

For the parts the team had to build, it was straightforward:

• Mobile app: the application running on the customer’s phone, we would build that from scratch
• API Gateway + CIAM: although we did not build an API gateway nor a CIAM solution as there are many great off-the-shelf options, we had to set it up properly and with special care as this could be a public entry point to the bank’s backend services and a potential security hole. All the requests going through the API Gateway would have the be authenticated and authorized by a CIAM (Customer Identity and Authorization Management) solution or rejected.
• BFF: this is the outer web layer for the mobile app which serves only the mobile app, BFF stands for Backend For Frontend and the idea was to have one BFF layer per customer channel.

Because the OMNI layer was supposed to serve all the other channels, if we zoom into this part of the architecture and add more channels, it would look like this:

The Problem

While testing the first phase of the project before going live, we noticed a big latency problem on many endpoints, causing the mobile app to run slower than acceptable. The BFF services were running with an acceptable response time, except when there were downstream API calls to the OMNI layer to request data from the Core Banking systems, there was a latency of up to 15s in average, but we could see up to 120s in some requests, which is way beyond what’s acceptable in terms of response time and result in the requests to time out and return an error.

Being an old bank, founded in the 1950s, the Core services were running on Mainframes, modern ones though, but there was no intermediate layer between the OMNI layer and the Core Banking. The increased traffic to the mainframe coming from the mobile app was adding pressure on the Mainframe, not only increasing the response times, but also the resources consumption on the Core Banking layer and, with that, their usage bill for the mainframe.

This meant we had to move that traffic somewhere else for us to:

1. Keep the core resources consumption in check by separating queries for data by moving them to data layer outside core and commands that would be executed against the core when updates to the data were needed.
2. Largely reduce the latency in our customer-facing mobile app

This also meant we had to move the data we needed from the Core Banking into a more accessible place with lower latency, so we started to think how we could export the data from the Core layer in a way that keeps exported data up to date and doesn’t add more pressure on the Mainframe.

On modern web applications, the natural way of thinking for a problem like this would be to add a caching layer between the BFF and the OMNI services using some in-memory caching software like Memcached or Varnish, or a mixed in-memory/disk solution like Redis. However, there were multiple concerns about this because of the large amount of data we would need to cache. We started to look at the problem, not as a web application issue, but rather as a general issue in the way the bank’s technology was setup. We had a modern software and hardware layer sitting on top of hardware running legacy software. By no means we are saying that legacy software is not good, but we need a different approach to interacting and integrating with this type of technologies. One of the best concrete and real life examples of legacy and modern living together is the Hearst Tower in Manhattan, it was built in the 1920s and at some point, in modern times, they had to expand it, so they built a modern building on top of it, preserving the old building so people can still appreciate it while also having all the benefits of modern engineering and architecture on the new building.

This client was the same, their technology stack was built in the 1950s and now it had to be modernized, not by tearing it down and starting over but rather by using the legacy as a foundation and finding creative ways for integrating both technologies.

We came up with the idea to incorporate a Low Latency Data Store (LLDS) layer that would contain a read-only copy of the Core Banking data we need for the mobile app in a more business-friendly semantics and data model. Not only that, but it would also have to be updated in near real time to comply with requirements from other teams. This would remove the need to make direct requests to the core banking and offload that traffic to the LLDS.

Introducing a Digital Data Platform

The data from the core systems was very fragmented, the client used different software for different services, this means, for example, the data for the Debit Card transactions was in one place and the data for the Current and Savings accounts were stored in a different place. Simply exporting the data and dumping it on a different data store was going to help with the latency and the load on the mainframe, but certainly we could do more. The data, just as it was, was very difficult to work with as there was no way to link records together from one system to another, it would be better to process, enrich and transform the data into a format that was easier for the mobile app backend services to process and understand. The question was, how can we publish data from the core systems in real time? we did not have access to those services and could not modify their configuration.

The core layer looked like the following diagram.

Most of the core systems were backed by Relational Databases and a few, those located on the mainframe, were stored in hierarchical structures persisted in files, enabled as VSAM (various types of indexing used depending on the data). All the data we need is being pumped into the core’s storage layers, call it Database or VSAM, everything was there and we could easily get it through a technique called Change Data Capture (CDC), there are multiple CDC software tools and connectors for virtually any Database Management System, including VSAM.

We can think of CDC software as a surveillance system for databases, whether the data is updated, added or deleted, the CDC tool logs those events and shares them so they can be utilized by other systems interested in that data. In our case, we needed to transform and store the transformed data somewhere else.

This solved one part of the puzzle, getting the data out of the mainframe. Our client’s VSAM files were encrypted and compressed, so after reading the changed record we needed to decompress and (if needed) decrypt it.

Once the data is out, expanded and decrypted, we had to process it. For that, the best way is to have a data stream for each data class we needed. Apache Kafka has become the de-facto standard for high-throughput low latency data publishing and subscribing in the tech industry. It can accept, store and publish data in any format, but doesn’t enforce it. Therefore it is best if paired with a schema tool so that we ensure consistency in the serialization format and avoid rejections while processing the data. Apache Avro is one of the most popular and space saving serialization frameworks that is often used alongside with Kafka for this purpose.

This leaves us with an architecture like this:

As shown, for example, if a Credit Card transaction happens, it would be processed by the Credit Card system and eventually stored in its Database, where the CDC software will be monitoring the logs and as soon as it registers the new record it would be picked up and sent to the Kafka Producer. Our Kafka Producer will essentially fetch the Avro Schema from the Schema Registry to serialize the Credit Card Transaction data and publish it to the corresponding Kafka topic. On the other side of there will be a Kafka Consumer waiting to process the data, enrich it with data from other topics, published by other systems and:

1. Store it in the LLDS for later consumption by the mobile app backend services, the analytics engine and any other system which needs this data.
2. Push the processed and enriched data to another Kafka topic for real time consumption by the upstream services powering the mobile app backend.

This way, anything happening on the core banking systems is extracted via CDC in real time and served to the mobile app so that the bank customers are always aware of the state of their finances via the different modules integrated with the app. At the same time, this Customer Data Platform enables other verticals from the bank to take advantage of the data to move faster, by integrating into the processed data kafka topics or the LLDS instead of talking to the mainframe directly.

The Customer Data Platform now can serve as a single integrated hub for the bank’s most recent client data instead of consuming fragmented pieces stored in different format and different places, democratizing the data and enabling the organization to move at a faster pace in terms of their technology transformation and business decisions.

Conclusion

Having a huge legacy stack doesn’t mean it’s not possible to run modern tech, legacy software is oftentimes too rigid, difficult to change and very expensive to replace. But with some creativity and the right team it can be turned into a solid foundation to support more the new pieces of a modernized tech stack. We must be very careful to produce an elegant solution design that can scale well and serve the client for many years so that, just like the Hearst Tower, it becomes a piece of art of modern architecture.

If you like what you've read so far or resonate with my opinions, subscribe to my mailing list, it's free and you'll get all my writings twice a month on your inbox.