New York Post

Founded by Alexander Hamilton in 1801, the New York Post is the oldest continuously-published daily newspaper in the United States and one of the nation’s premier digital destinations for news and information. The expansive digital presence boasts 93.6 million monthly unique visitors¹ across their network of sites (their flagship NYPost.com, PageSix.com, and Decider.com) and nearly 150,000 daily print circulation².

In September 2021, New York Post launched Post Sports+, a premium membership experience. New York Post was interested in promoting Sports+ to their users who are most interested in premium sports membership. One method of engaging potential Sports+ subscribers is to show a flyout encouraging conversion to a subscription to the most interested users.

The New York Post leveraged machine learning decisioning to combine batch and real-time signals to promote Sports+ using a flyout. By leveraging real-time signals in particular, New York Post was able to target new cohorts of users including anonymous and first-time users.

Challenges

The New York Post marketing team designed a high-quality flyout which promotes the Sports+ subscription. The natural question arose around whom to target with the flyout across the large customer base of 93.6 million monthly unique users¹. In order to maintain the highest quality consumer experience, New York Post wanted to identify and present the flyout only to users who would be most likely to join Post Sports+.

Machine learning provides numerous benefits for this use case including –

1. Machine learning and associated automated feature engineering can identify the user activities most likely to determine who will convert across billions of behavioral activities
2. Machine learning will adapt over time to changing consumer behaviors
3. Machine learning, when used with real-time features, is able to promote the flyout to anonymous, real-time, and recurring users

Real-time machine learning decisioning was a natural solution for the New York Post technical and marketing teams given that (a) real time signals enable the targeting of anonymous users and (b) machine learning decisioning frameworks allowed New York Post to decide, for each user, whether to show the subscription flyout.

Implementation

The New York Post leveraged various machine learning pipeline modules in order to accelerate the deployment of this solution including –

• Continuous model and feature retraining to ensure the machine learning pipeline adapted to changes in consumer behavior over time
• Automated merging and cleaning of various raw data sources across batch and real-time
• Automated feature engineering techniques to identify the most predictive features and generate these features at scale
• Automated AutoML model selection techniques to identify the highest performing models

There were four aspects of the implementation for the New York Post team –

1. Batch data ingest into Cortex
2. Real-time data ingest
3. Integrating Decisioning
4. Visualizing performance within Cortex

Batch Data Ingest

Batch data was provided to Cortex in the form of daily static files and corresponding to extensive user behavioral activity.

Real-Time Data Ingest

The Decisioning SDK provided a framework for the New York Post to both send real-time behavioral data into Cortex and also an API to obtain real-time decisions on user behavior. Sending real-time data into Cortex was straightforward and included in-session behaviors like clicks and pageviews.

Decisioning

The Decisioning SDK exposes various decision endpoints which indicate whether a user should see the flyout or not based off of a real-time inference engine which takes both batch and real-time data as input. These endpoints were used to decide, for each user, whether they should see the subscription flyout at a particular moment.

Visualizing Performance

An important aspect of the implementation was to expose performance information to the technical, marketing, and business teams in order to evaluate any increase in conversions. In conjunction with the Vidora team, New York Post customized various visualizations which showed daily conversions and increases in conversions over a rule-based “metered” approach. These visualizations were monitored daily and used as discussion points during weekly calls.

Business Value

The New York Post leveraged Cortex visualization tools to analyze performance on a daily basis. The New York Post was interested in two primary metrics – (1) the increase in conversions using a machine learning approach over a rule-based approach and (2) the increase in performance which resulted from using real-time features in conjunction with batch features.

Performance Increase Over Metered : 3x Increase in Flyout Conversions

The New York Post implemented a rule-based approach to targeting users for conversions based on how often a user read “sports” content on the New York Post. New York Post compared the conversion rate for users exposed to the rule based approach and the machine learning approach. The Machine Learning approach resulted in a 3x increase in the flyout campaign conversions over the rule-based approach.

Performance Increase using Real-Time Features : 40% Increase in Flyout Conversions

Next, The New York Post compared the conversion rates when combining real-time features and batch features to experiences which leveraged only batch historical features based on historical user engagement. Note that real-time features benefit conversions by both increasing the accuracy of the models (the models are now working with more data, in particular real-time data) and also increasing the number of users which can be targeted (the models can now target anonymous and first-time users). Using real-time features increased the conversion rate by 40% over using only batch historical features.

Conclusion

Increasing conversions to Sports+ subscriptions and other products is a key goal for the New York Post technical, product, and marketing teams. In this initial campaign utilizing Vidora machine learning decisioning, the New York Post demonstrated that machine learning techniques far outperformed rule-based engines for predicting who will convert to a Sports+ subscription. In addition, the New York Post demonstrated the real-time features and real-time machine learning result in a large increase in conversions over batch techniques based on only historical data.

¹ Comscore Media Metrix November 2021
² Total average circulation excluding affiliated publications as reported by AAM media statement September 2021