Search has been the cornerstone of finding relevant content since the dawn of the digital age. But the traditional method of search, a text-based recommendation engine based on ‘tagged’ keywords, can be unhelpful if the tags are not the aspects of the content you are predominantly interested in. AI is now set to meet the growing desire for more serendipitous content discovery.
Specifically, vector search has emerged as a tool for recommending content based on semantic similarities rather than matching keywords. It does this by converting pieces of content into numerical high-dimensional vectors using machine learning algorithms and then calculating the distance between them, with those closest together being the most relevant recommendations.
The Financial Times, one of the world’s oldest and leading business news organisations, sees the future of media not entirely in vector search but rather hybrid search. This new, AI-driven approach to search combines traditional text-based search with the more advanced capabilities of vector search to present recommendations which are contextually and semantically relevant.
“Especially nowadays with AI becoming more popular for solving different problems, the ability to do vector search alongside the standard keyword search is crucial. The problem is it's a new territory,” says Dimitar Terziev, Technical Director for FT Core Platforms at the Financial Times.
Uncharted territory
Remaining a market-leading source of business information for well over a century has meant evolving with the times, and the Financial Times is acutely aware that, as with other sectors, AI is now set to revolutionise how media is both discovered and consumed in the years ahead.
All media brands have found they have less time and more competition to keep audiences engaged in the digital world, but subscribers to the Financial Times, which include a high concentration of business decision-makers and high-net-worth individuals, are particularly time poor. They turn to the FT to keep on top of news events and understand complex topics quickly.
Predominantly known for its print and online newspaper, the FT also produces other products including events, B2B content and specialist titles such as The Banker and Investors' Chronicle.
“We might have various products, but at their core is our gold star journalism,” says Terziev. “Being able to slice, dice and surface the content in different ways is critical. [But] as a nascent area, we weren’t able to speak with anyone advanced in their journey to building hybrid search. We felt like we were the first to embark on this and develop a hybrid search that can work at scale.”
Leap of faith
Tasked with developing a hybrid search solution in just a few months, Terziev’s platforms team within the FT looked close to home for a partner to support the build and rollout. Already the backbone of its content and metadata platform, which stores over a million pieces of FT content, MongoDB was the natural choice for supporting the company through its hybrid search journey. The database provider enables its customers to efficiently deploy full-text search and vector search capabilities, as well as combining both capabilities for unified results in a hybrid search.
The FT’s existing relationship with MongoDB as well as its search technology were major drivers in the vendor selection for the hybrid search deployment, but what sealed the deal was the wraparound support and expertise offered from the MongoDB Professional Services team. These engagements were especially important as the FT is an early adopter of hybrid search.
“It was a bit of a leap of faith [but] we decided to invest in MongoDB not only because of the technology but because of the partnership we have established with them,” says Terziev. “It's important to know that you can rely on good support. Search is a complex space. It encompasses areas such as information retrieval, statistical analysis, AI, ML and user experiences. Some areas we felt confident with, others we wanted to get external expertise.
“We gained a lot through being able to engage through MongoDB Professional Services. They were very, very helpful, especially at the later stage of the project which is the critical part in terms of tinkering. And the fact we already had proficiency with MongoDB, the team didn't need to learn something new, was a major benefit. It made quite a lot of sense to go with MongoDB.”
Leading the charge
The initial deployment milestone, releasing hybrid search for web users on FT.com, was completed in just one and a half quarters, going live at the end of September. This has enabled the FT to take a leading role in hybrid search for the media sector, with readers benefitting from hyper-relevant article suggestions which they don’t need to take time to search for themselves.
The rollout of hybrid search will now expand to the FT’s mobile applications before turning to its specialist titles so that all of the company’s users can benefit from the capabilities. The ultimate goal is for content from all FT platforms to be available via the same hybrid search mechanism.
“We are at the beginning and we expect to improve it further. It's probably not the fastest search engine out there, but the fact it incorporates so many additional benefits makes it very lucrative,” Terziev adds. “There is no one magical configuration you can apply across your search indexes and everything will be working. Building hybrid search requires domain expertise, understanding the content that's been searched and also the use case that you are optimising and building for.
“It is a journey which will probably be a long one. In certain cases, we might consider that there is no end destination for this journey because the search needs evolve as users demands evolve, especially now we are living in this AI-enabled world. So being able to surface content in the most efficient way that is very relevant to your readers will continue to be a focus for us.”
Delivered in partnership with MongoDB
