FlyRank ML Internship · Capstone Research Artifact

Search Intelligence & Content Opportunity Scoring

A research-grade investigation into whether leakage-controlled search-demand, competition, freshness, content-production, and contextual signals contain generalizable information about observed content momentum.

Research Lane: Refresh / Content Opportunity Grouped Validation Leakage Controlled Reproducible
Abstract

A disciplined study of predictive signal, not a manufactured success story.

This study investigates whether a leakage-controlled set of content, search-demand, competition, freshness, and content-production signals can explain observed content momentum and support a transparent prioritization framework for human content review.


The analysis deliberately separates predictive modeling from exploratory opportunity ranking. Identifiers are excluded from the feature matrix, target-derived variables are removed, missing-value treatment is learned inside pipelines, and grouped train/test validation is used as the primary estimate because observations belong to anonymized clients.


The resulting evidence does not support a claim that the tested machine-learning models provide generalizable predictive improvement over the mean baseline under grouped held-out validation. The operational contribution is therefore framed as an auditable decision-support layer rather than an autonomous forecasting system.

30,000
Dataset rows
44
Original variables
26,612
Modeling observations
12
Predictive features
32
Anonymized clients
Research Design

Methodological safeguards were treated as first-class research objects.

The capstone is intentionally structured around validity, reproducibility, interpretability, and disciplined inference.

01

Target Audit

The target variable is audited before modeling to establish what is actually being measured.

02

Leakage Control

Variables that directly encode or temporally entangle the target are excluded.

03

Grouped Validation

Client groups are separated between training and testing to prevent optimistic leakage across related observations.

04

Baseline Comparison

A mean dummy regressor is mandatory, preventing weak models from being presented as useful merely because they are technically non-trivial.

05

Pipeline Integrity

Missing-value treatment and preprocessing are learned inside modeling pipelines after the split.

06

Untouched Test Evaluation

Final predictive metrics are evaluated on held-out data rather than training observations.

07

Predictive / Exploratory Separation

Predictive modeling and opportunity ranking are explicitly treated as different analytical tasks.

08

Claim Discipline

The analysis does not claim causality, Google-algorithm knowledge, or guaranteed future performance.

Central Finding

The negative result is itself a research result.

Under grouped held-out validation, no tested machine-learning model improved upon the mean baseline. Ridge Regression was the strongest non-dummy model by MAE, but its MAE remained above the baseline. The appropriate interpretation is therefore limited predictive signal under the tested design—not failure of the research.

This distinction is fundamental: a rigorous empirical study should preserve evidence even when the evidence does not produce an impressive predictive score.

Measurement Validity

Target construction was independently audited.

Confirmed

Target reconstruction

The notebook reconstructs trend_pct from the relationship between impressions in the last 30 days and the previous 30 days.

26,612 rows were tested. The median absolute difference was approximately 0.022 percentage points, with 100% of audited rows matching within 0.1 percentage point.

Excluded

Leakage variables

  • impressions_last_30d
  • impressions_prev_30d
  • trend_direction
  • period-comparison click/session variables
  • 90-day performance aggregates
  • temporally entangled engagement metrics
  • content_id and client_id
  • the target itself
Methodology

From raw warehouse extract to defensible evidence.

The workflow moves sequentially from data audit to target validation, feature construction, grouped modeling, diagnostic analysis, and finally a transparent opportunity index.

01

Ingestion & Audit

Load the anonymized content warehouse extract and establish its schema, missingness, identifiers, and dimensional structure.

02

Leakage Investigation

Determine whether available fields mathematically or temporally encode the target and remove them from predictive modeling.

03

Grouped Modeling

Train Dummy, Ridge, Random Forest, and Extra Trees models under client-grouped held-out validation.

04

Decision Support

Construct a transparent opportunity score using demand, staleness, competition, and lifecycle signals.

Predictive Feature Matrix

Twelve features survived the methodological filter.

search_volume Numeric
competition Numeric
cpc Numeric
word_count Numeric
char_count Numeric
content_age_days Numeric
days_since_last_update Numeric
competition_level Categorical
content_type Categorical
main_intent Categorical
provider_used Categorical
model_used Categorical
Empirical Results

Grouped held-out validation does not support predictive superiority over the baseline.

The mean dummy regressor remains the strongest model by MAE. Ridge Regression is the best non-dummy model, but its error is still higher than the baseline.

Model MAE RMSE R² Interpretation
Dummy Regressor 65.4752 295.9562 -0.0005 Baseline
Ridge Regression 71.3596 297.5412 -0.0113 Best non-dummy model
Random Forest 88.6801 309.0226 -0.0908 Limited predictive signal
Extra Trees 88.9691 309.3702 -0.0933 Limited predictive signal


Statistical interpretation

Relative MAE improvement vs. dummy: -8.99%

Because the best non-dummy model has higher MAE than the baseline, the machine-learning system should not be presented as a validated predictive engine for future content momentum.

Empirical Figures

Figures reproduced directly from the capstone notebook.

The section below reads the embedded PNG outputs directly from capstone.ipynb. Keep the notebook in the same directory as this HTML file so the original analytical figures can be rendered without redrawing or approximating them.

Loading original notebook figures…
Exploratory Signal

Permutation importance identifies where the exploratory model finds sensitivity.

These values are diagnostic rather than causal explanations. They describe the change in model scoring loss under permutation within the exploratory analysis.

Feature Importance Mean Importance Std
word_count 10.925458 0.520312
search_volume 10.855314 0.973300
char_count 5.447605 0.413041
competition 3.428014 0.381740
days_since_last_update 2.959371 0.262951
cpc 2.074102 0.207187
content_age_days 1.845638 0.945165
provider_used 0.022146 0.001481
competition_level -0.088470 0.027482
main_intent -0.440405 0.189133
content_type -0.531634 0.037549
model_used -4.921439 0.301586
Decision-Support Layer

When prediction is limited, transparent prioritization can still create analytical value.

The opportunity engine is explicitly not presented as a forecast. It is an interpretable review-priority index designed to help human analysts decide which content deserves attention first.

Opportunity Score = 0.40 × Demand Signal + 0.30 × Staleness Signal + 0.20 × Competition Signal + 0.10 × Lifecycle Signal
40%

Search Demand

Percentile-ranked search volume. Higher demand increases review priority.

30%

Staleness

Percentile-ranked days since the content was last updated.

20%

Competition

Percentile-ranked competition signal from the available dataset.

10%

Lifecycle

Percentile-ranked content age as an additional review signal.



High Priority Review

Top 20 opportunities

The ranking system sorts eligible observations by opportunity score and assigns a human-review priority.

The notebook identified 27,532 rows eligible for opportunity scoring.

Human-in-the-loop

No autonomous action

Reason codes such as "Stale Content", "High Search Demand", and "Mature Content" are deliberately diagnostic.

They do not imply causality or guarantee future traffic, ranking, click, or business outcomes.

Subgroup Analysis

Opportunity signals differ across content and search intent.

These comparisons are descriptive rather than causal. They characterize how the opportunity index behaves across groups.

Content Type

Content Type Pages Mean Score Median Score Mean Staleness Mean Demand
keyword article 26,835 50.6286 49.2899 47.8334 163.0091
comparison article 697 25.8707 27.1389 18.0100 0.0000


Search Intent

Intent Pages Mean Score Median Score Mean Staleness Mean Demand
Transactional 5,691 59.9529 59.6194 52.9181 90.6783
Commercial 4,561 58.5345 58.4936 50.6895 166.9458
Navigational 46 54.0099 57.8683 41.0217 63.0435
Informational 17,092 44.5520 43.5909 44.2206 181.0180
Robustness Analysis

Validation design materially changes the apparent result.

A secondary random row split was performed as a sensitivity analysis. Grouped validation remains the primary estimate because observations are nested within anonymized clients.

Validation Design MAE RMSE R²
Grouped by client 88.9691 309.3702 -0.0933
Random row split 64.7979 317.9670 -0.0128

Why this matters

Random splitting can create a more optimistic picture of generalization.

The difference demonstrates why grouped validation is retained as the primary design rather than selecting whichever split produces the most flattering metric.

Research Conclusion

What the evidence actually supports.

"The resulting system should be presented as a research-backed decision support layer for content review, not as an autonomous optimizer or a validated forecast of search performance."

The capstone establishes two principal contributions.

First: it tests whether historical content, search, competition, freshness, and production attributes contain generalizable predictive information about observed content momentum.

Second: it converts available search-intelligence context into a transparent, auditable opportunity-review system without overstating predictive or causal capability.

The study therefore prioritizes methodological defensibility over artificially impressive model performance.

Reproducibility

The research artifact is designed to be rerun.

The notebook uses a fixed random seed of 42 and exports public-safe analytical artifacts for inspection and repository versioning.

01
schema_audit.csv Dataset schema and missingness audit
02
target_summary.csv Target distribution summary
03
model_comparison.csv Grouped model evaluation metrics
04
permutation_importance.csv Exploratory feature sensitivity
05
top_20_content_opportunities.csv Highest-ranked review opportunities
06
validation_sensitivity.csv Grouped vs random validation comparison
07
content_type_summary.csv Content-type subgroup analysis
08
search_intent_summary.csv Search-intent subgroup analysis
09
content_opportunity_rankings.csv Public-safe ranking export
Research Access

Inspect the complete analytical workflow.

The website is the presentation layer. The notebook and repository remain the primary reproducibility artifacts.