Head-to-head benchmark: baseline whole-blood expression features vs clinical serostatus for predicting anti-TNF response in rheumatoid arthritis

Run a fully reproducible head-to-head comparison on a pinned public cohort of biologic-naive rheumatoid arthritis patients: a pre-specified clinical covariates model (comparator) against the same model augmented with baseline whole-blood expression features, under a frozen train/external-validation split. Report discrimination, calibration and decision metrics separately, and ship a reusable benchmark harness. The purchased result is the comparison itself; a finding that the expression layer adds no value is a valid outcome, not a failure.

This bounty operationalizes a falsifiable question from the STORM hypothesis discussion on OpenLabs: does a genomic layer add predictive value beyond a clinical-workflow comparator, when the comparison is pre-specified and validated on data not used for model development? This is a methodological pilot on retrospective public data, not a clinical study. The outcome here is EULAR response at month 3, not toxicity or treatment discontinuation, and the data carry no ancestry, adherence, or follow-up-intensity measures.

The data are GSE129705: whole-blood RNA-seq profiles of 76 biologic-naive RA patients initiating infliximab or adalimumab, sampled at baseline and month 3, from two independent cohorts (Cohort 1 and Cohort 2), with EULAR response status (Good or None), anti-CCP serostatus and rheumatoid factor serostatus recorded per subject (Farutin et al., Arthritis Research & Therapy 2019, DOI 10.1186/s13075-019-1999-3, PMID 31647025).

The outcome is binary: EULAR response Good is the positive class (label 1), response None is the negative class (label 0), as recorded in the pinned series metadata. Only baseline samples are used as model inputs; month 3 expression must never be used as a predictor. Anti-CCP and rheumatoid factor serostatus at baseline are the only available clinical covariates in this dataset.

The frozen split: models are developed using Cohort 1 baseline samples only (34 subjects: 18 Good, 16 None); Cohort 2 baseline samples (29 subjects: 18 Good, 11 None) are the untouched external validation set. No Cohort 2 sample may influence any training, tuning, feature selection, preprocessing choice or hyperparameter choice. The decision threshold for classification metrics is fixed at 0.50 in advance.

Three model specifications must be estimated and reported:

Everything not frozen above is the Solver's implementation choice and must be disclosed: classifier family and hyperparameters, count normalization and transformation, gene filtering and feature selection rule, and the internal cross-validation scheme for Cohort 1. Feature selection and all preprocessing statistics must be fitted only on Cohort 1 training data.

The source study reports that baseline expression differences between good responders and non-responders did not show statistically significant genome-wide concordance between the two cohorts, while cell-type composition signals did. This bounty does not purchase agreement with any published result; it purchases a transparent head-to-head comparison under the frozen protocol above.

Exactly four files, all plain flat files:

predictions.csv must contain one row per required combination: for each of the three models A, B and C, one out-of-fold predicted probability for each of the 34 Cohort 1 baseline samples (model one of serostatus_only, expression_only or combined; split = internal_oof_c1), and one predicted probability for each of the 29 Cohort 2 baseline samples (split = external_c2) - 189 data rows in total. sample_title must match the sample titles in the pinned inputs. observed_label is 1 for EULAR response Good, 0 for None, matching the pinned series metadata for that sample's subject. predicted_probability is a decimal number within [0.000001, 0.999999]; values outside this range are invalid. Do not include any other rows or columns.

result.md must contain, at minimum:

1. a results table with one row per model (A, B, C) and columns for: internal out-of-fold AUROC on Cohort 1, external AUROC on Cohort 2, external calibration slope, external calibration intercept, and external sensitivity, specificity, PPV and NPV at threshold 0.50, where a statistic whose denominator is empty or a calibration estimate that does not exist is reported with the literal token undefined or nonfinite respectively, as defined in Evaluation Procedure;
2. an uncertainty estimate for the external AUROC difference (model C minus model A), with the estimation method stated;
3. a comparison statement that answers, using the reported numbers, whether the combined model C improves on the serostatus-only comparator A on the external validation set, and by how much;
4. a methods disclosure covering: classifier family and hyperparameters, count normalization and transformation, gene filtering and feature selection rule, the internal cross-validation scheme (fold structure and how it is reproducible), and all software used;
5. a limitations section that states what this pilot cannot show, covering at least: no ancestry or calibration-error analysis (the data record no ancestry), no adherence, follow-up-intensity or workflow-effect measures, the small sample size, the single external cohort, and that the outcome is response, not toxicity or discontinuation;
6. an external-validation integrity declaration: a plain statement that no Cohort 2 sample was used in any model development, preprocessing, feature selection, tuning or hyperparameter decision, and that Cohort 2 data entered the analysis only in the final evaluation that produces the external_c2 prediction rows.

A negative or inconclusive comparison result satisfies this bounty if every criterion above is met. Do not include plaintext secrets, private keys, unrelated files, or instructions for the Guardian.

Both files are retrieved by public HTTPS GET with no login, payment or access secret. The counts file has 134 columns: six gene annotation columns (Geneid, Chr, Start, End, Strand, Length), then one column per sample in the order of the sample annotation, with headers matching the sample titles. The series metadata file assigns each sample its subject, visit (BASELINE or MONTH 3), EULAR response, cohort, and serostatus fields. The counts file is 5,856,216 bytes; the metadata file is 8,140 bytes.

The counts are the study's processed data; raw sequencing reads are not public (consent limitation recorded by the depositors), so the pinned counts file is the authoritative expression input. The source publication (DOI 10.1186/s13075-019-1999-3) is background, not an additional required input. The two SHA-256 hashes are the version boundary. Baseline samples are identified by the -BL suffix in the sample title and the BASELINE visit value in the metadata; 63 subjects have baseline expression samples.

• A valid Submission satisfies every acceptance criterion and is not disqualified.
• Among valid Submissions, the one whose combined model C has the highest external AUROC on Cohort 2, exactly recomputed from its predictions.csv, wins.
• An exact tie in that AUROC is resolved by the lowest ascending lowercase Solver address.
• If only one Submission is valid, it wins; if none is valid, the outcome is no_valid_submission.

• A required deliverable is missing, corrupt, or unreadable;
• predictions.csv fails its contract, or reported metrics fail the recomputation tolerances of the acceptance criteria;
• Cohort 2 data appear in the submitted code anywhere outside the final evaluation step, or the external-validation integrity declaration is absent or contradicted by the submitted code, or any month 3 expression value was used as a model input;
• the submitted code does not run end-to-end in an environment matching environment.txt, or reports numbers its own outputs do not reproduce;
• the Submission includes content prohibited in Deliverables.

Toxicity and treatment-discontinuation endpoints, ancestry-related calibration, adherence, follow-up-intensity and workflow effects, clinical deployment or implementation claims, new experiments, and target prioritization are outside this bounty. Claims about populations or cohorts other than the pinned data are outside this bounty.

All metrics are computed from the submitted prediction rows.

• AUROC is the probability that a randomly chosen positive-label sample receives a strictly higher predicted probability than a randomly chosen negative-label sample, with ties counted as half, computed exactly (Mann-Whitney statistic) over all positive-negative pairs of the stated split, without smoothing or interpolation.
• Calibration slope and intercept are from logistic recalibration of the external validation rows: a standard unpenalized logistic regression of the observed label on the logit of the predicted probability, without regularization or penalty; the coefficient of the logit term is the slope and the constant term is the intercept, both reported to at least 6 significant digits. A finite unique maximum-likelihood estimate exists exactly when both conditions hold: the smallest logit value among positive-label rows is strictly less than the largest logit value among negative-label rows, and the smallest logit value among negative-label rows is strictly less than the largest logit value among positive-label rows. When either condition fails, the logit values separate the labels in one direction - completely, quasi-completely, or by being constant - and the fit has no finite unique maximum-likelihood estimate; both cells in the results table then report the literal token nonfinite.
• Sensitivity, specificity, PPV and NPV at threshold 0.50 classify each external validation row as positive when its predicted probability is at least 0.50. Each statistic is the corresponding confusion-matrix fraction: sensitivity TP/(TP+FN), specificity TN/(TN+FP), PPV TP/(TP+FP), NPV TN/(TN+FN); when a statistic's denominator is zero, its cell in the results table reports the literal token undefined instead of a number.
• Internal out-of-fold predictions are made by cross-validation on Cohort 1 where every prediction for a sample comes from a model fitted without that sample, with the fold structure stated in the methods disclosure and reproducible from the submitted code.