Will moving workloads from on-prem mainframes into the cloud scare employees and push them into quitting their jobs?
Probably not, judging by the results of a collaboration between AWS and the New York Times, which named the claim that "staff retention will be a challenge" as one of three "myths" that were "busted" during the migration of the world famous newspaper's systems.
The story starts almost ten years ago when The NYT used an IBM Z mainframe to run a key application named CIS, which took care of business-critical functionality including billing, invoicing, customer account management, delivery routing, and financial reporting (accountancy, not journalism).
This infrastructure was expensive to run and needed modernisation to drive down costs whilst enabling the convergence of its digital platform with the COBOL-based system, which managed the delivery of papers to readers' homes.
In 2015, it partnered with Modern Systems, an AWS Partner Network (APN) Select Technology Partner, to transform CIS into a Java-based application that runs on the AWS Cloud and is now called Aristo. In the first year after its deployment, Aristo billed more than half a billion dollars in subscription revenue and processed nearly 6.5 million transactions, AWS enterprise support lead Chris Milkosky and the New York Times' Padmanabha Rao Chillara, wrote in a blog looking back at the project.
Clients access it via an API Management Platform, while external vendors use a UI for updates, and internal systems send batch feeds to update data. A batch scheduler processes inputs from APIs, UI, and batch systems, generating data for downstream systems like billing, payments, and reporting.
Before migration, clients used CIS Online Screens and an ETL tool for metadata transformation. Post-migration, clients transitioned to middleware-based APIs, INK Services (REST-based) replaced complex SOAP calls, and direct data posting streamlined processes, eliminating ETL dependency.
Aristo also offered easier integration with internal and third-party applications, increased observability, and better integration processes. Previously, integrations relied on complex ETL transformations and screen scraping due to mainframe constraints like compressed data fields.
In AWS, teams quickly developed cost-efficient microservices to access data directly, while an in-house billing application replaced external dependencies. Observability improved significantly with tools like Sumo Logic and Datadog, enabling dashboard creation, API monitoring, batch job tracking, and SLA alerts, providing enhanced visibility and operational insights.
It wasn't all plain sailing. In 2020, The Times identified key areas requiring optimisation to address potential challenges. Filesystem storage retention was a significant issue due to the mainframe’s generation of large, redundant files from batch processes. To manage this, the team implemented mechanisms to limit file growth by reducing retention periods, archiving non-essential data, and cleaning up unnecessary files.
The migration to a relational database introduced fragmentation that slowed system performance over time. While not immediately critical, the team anticipated future impacts and developed defragmentation tools using Oracle APIs to address the issue during system downtime. Database size growth was also managed by implementing retention policies to remove unnecessary data.
The legacy database schema, inherited from the mainframe, also lacked modern design principles, causing occasional consistency issues under high traffic. The team optimised code, improved process scheduling, and enhanced monitoring to mitigate these problems.
To ensure long-term maintainability, comprehensive documentation was created, including diagrams, auto-generated Java documentation, and tools to manage mainframe file formats. This gave new staff clear insights into the system’s architecture and processes.
Operational maturity was also enhanced by optimising job schedules to accommodate peak traffic, automating low-severity issues, and decoupling analytical data from operational processes, ensuring smoother performance and reliability.
Cloud migration myths
The project ended up challenging three myths, including fears that the cloud will impact staff retention.
"The Times was able to retain staff that supported Aristo on the mainframe," AWS wrote in its blog. "They were given training on using Java for future software development and AWS to help them support the system running in AWS.
"The Times strives to employ “T-Shaped” engineers, technology professionals with a broad base of understanding in several areas along with a particular area of deep expertise. This helped the team support the Aristo monolithic application and allowed them to develop new features that the business required."
The project also challenged a myth that suggests cloud cannot achieve the same scalability and performance as a mainframe.
When planning the migration, care was taken to ensure AWS resources matched or exceeded the mainframe's performance capabilities.
"Enhanced performance was observed with Aristo’s batch jobs by using preload steps in the jobs," AWS wrote. "The preload process takes data from database tables and loads it into cache memory. That cache is then used for job operations. At the end of the job, changes are committed to the actual database.
"This process drastically increased job performance. For example, a “Home Delivery Liability” job that used to run more than 20 hours has been reduced to 4 hours."
The final myth was that cloud could not be as reliable as the mainframe.
"After reviewing availability metrics over time, The Times confirmed that service availability was maintained after the migration," AWS confirmed.
After busting those three myths, the NYT is now working to migrate key Aristo databases from Oracle on Amazon EC2 to Amazon RDS PostgreSQL using AWS Database Migration Service, reducing costs while maintaining performance and features.
