About Tencent and ABetterChoice
Tencent, owns and manages some of the gaming industry's most popular franchises including the blockbuster hit "League of Legends". With 200 million worldwide users and growing, Tencent Games has established themselves as a leader in gaming analytics, developing entirely new ways for their players to engage with the games they love. In 2022, leveraging the company's robust experimental expertise, the Tencent A/B Test team launched ABetterChoice, a new commercial A/B testing SAAS offering for the gaming industry.
Challenges
Developing ABetterChoice wasn't without its hurdles. From the start, engineers were forced to grapple with the inherent challenges that came with transitioning from an internal framework to a cloud-based global service:
-
Data Warehouse Dependent: The original internal framework was dependent on ingesting into a proprietary data warehouse for query acceleration, a costly solution that undermined data governance and data freshness.
-
Limited Support for Complex JOIN Queries: Essential for minimizing the costs and rigidity associated with denormalization pipelines, Tencent required a query engine that could not only rapidly execute JOIN operations but also dynamically scale to optimize costs.
-
Handling Multi-Cloud and Open Formats: Managing both Tencent's internal needs on Tencent Cloud and the diverse requirements and environments of their global customer base was demanding on team resources.
-
Elevated Storage Costs: Implementing a scalable and cost-efficient architecture would be crucial when it came to managing increased storage demands from a growing customer pool without incurring excessive costs.
Solution
Because their multi-cloud solution needed to serve both Tencent's internal and commercial users, ABetterChoice needed to be designed to run on two data lake solutions on Tencent Cloud along with other public clouds. This influenced the Tencent teams approach in the following ways:
-
ABetterChoice integrated Tencent Big Data Service (TBDS) on Tencent Cloud to support Tencent's internal A/B testing workloads.
-
ABetterChoice adopted Databricks on various public clouds, such as AWS, Azure, and GCP, to provide external independent game studios and companies with a data lake environment based on Delta Lake.
ABetterChoice architecture diagram
Additionally, the Tencent team selected StarRocks as its data lakehouse query engine for Delta Lake and Apache Iceberg which enabled all query workloads to run on one copy of data on the data lake. With StarRocks' Data Cache, engineers were able to minimize network overhead and ensure consistent performance by bypassing the data lake performance limitations they were encountering. Together with its optimization for large-scale JOIN queries, ABetterChoice was able to deliver high concurrency low latency queries directly on the data lakehouse, eliminating the need for a proprietary data warehouse for query acceleration.
Result
Today, ABetterChoice is helping Tencent better understand and innovate in the gaming industry, integrating with major titles such as Honor of Kings, PUBG Mobile, and Ubisoft's The Division. The impact of ABetterChoice's new architecture has been notable:
-
A Simplified Architecture: Unifying all workloads on the data lakehouse has eliminated the cost of maintaining a data warehouse alongside its ingestion pipelines, data model designs, and governance challenges.
-
Enhanced Reusability: Adopting StarRocks as their exclusive computing engine standardized their experimental computing layer and unified SQL computations, enhancing the reusability of upper-layer application services.
-
Reduced Storage Costs: The shift to cloud object storage for data persistence significantly reduced storage costs thanks to its superior scalability and cost-effectiveness.
-
Flexibility: The new architecture, deployed across various public clouds, supports multiple data lake table formats. This flexibility addresses a wide range of internal and external use cases, exclusively using open-source technologies.
What's Next For Tencent and ABetterChoice
Having met its initial goal in delivering a more robust data lakehouse architecture with StarRocks, The team behind ABetterChoice is planning to invest deeper into the following areas:
-
Further optimizing the query performance of StarRocks on Delta Lake, especially for multi-dimensional ad-hoc queries in A/B testing.
-
Contributing to the StarRocks project with query dispatcher optimizations for high concurrency workloads.
-
Customizing StarRocks' integrated lakehouse architecture to comply with additional data regulations and standards.
