{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# 文本因果推断 Lab · Text-as-Data Causal Starter\n\nStatsPAI Summer Bootcamp 配套 notebook。配合知识页 `/courses/summer-bootcamp/topics/text-causal-inference`。\n\n用一份可复现的合成语料 `text_corpus.csv`（文本里编码了一个同时影响处理与结果的\"主题\"混淆），走一遍：\n1. 朴素差分（被文本混淆，偏大）\n2. 把文本向量化（TF-IDF）\n3. 交叉拟合 DML：用文本表示扣掉混淆，恢复真实效应（true tau = 1.0）\n\n> 依赖：`pandas`, `scikit-learn`。安装：`pip install pandas scikit-learn`\n> 三条红线：测量误差→衰减；后处理文本泄漏；不要用结果监督文本表示再在同一批样本估计。"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "import pandas as pd\n\ndf = pd.read_csv(\"text_corpus.csv\")\nprint(\"shape:\", df.shape)\nnaive = (df.loc[df.treated == 1, \"outcome\"].mean()\n         - df.loc[df.treated == 0, \"outcome\"].mean())\nprint(\"naive diff-in-means =\", round(naive, 3), \" (true tau = 1.0, confounded by text theme)\")\ndf.head(3)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 1. 把文本向量化（TF-IDF）\n文本是混淆 X：它同时驱动处理 treated 和结果 outcome。"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "from sklearn.feature_extraction.text import TfidfVectorizer\n\nX = TfidfVectorizer(max_features=40).fit_transform(df[\"text\"]).toarray()\nY = df[\"outcome\"].to_numpy()\nD = df[\"treated\"].to_numpy()\nprint(\"TF-IDF matrix:\", X.shape)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 2. 交叉拟合 DML：用文本表示扣掉混淆\n- 在折外预测 E[Y|text] 与 E[D|text]\n- 残差化后回归，得到对文本混淆稳健的处理效应\n- 关键：nuisance 必须折外预测，避免把结果信息泄漏进估计"
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": "import numpy as np\nfrom sklearn.linear_model import LinearRegression, LogisticRegression\nfrom sklearn.model_selection import KFold\n\ny_hat = np.zeros(len(df)); d_hat = np.zeros(len(df))\nfor tr, te in KFold(5, shuffle=True, random_state=1).split(X):\n    y_hat[te] = LinearRegression().fit(X[tr], Y[tr]).predict(X[te])\n    d_hat[te] = LogisticRegression(max_iter=500).fit(X[tr], D[tr]).predict_proba(X[te])[:, 1]\n\nyr, dr = Y - y_hat, D - d_hat\ntheta = float(np.sum(dr * yr) / np.sum(dr ** 2))\nse = float(np.sqrt(np.mean((yr - theta * dr) ** 2 * dr ** 2) / (np.mean(dr ** 2) ** 2 * len(df))))\nprint(f\"naive (text-confounded) = {naive:.3f}\")\nprint(f\"text-adjusted DML       = {theta:.3f} +/- {1.96 * se:.3f}   # true tau = 1.0\")"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 3. 接下来\n- 把合成语料换成你自己的文本（政策文件、年报、新闻、开放式问卷）。\n- 想清楚文本扮演的角色：处理 D / 结果 Y / 混淆 X —— 角色不同，做法不同。\n- 真实项目：用更强的编码器（嵌入 / LLM），但仍要样本分裂、信度检验、并防后处理泄漏。\n- 用 StatsPAI 的 `dml` / `dml_diagnostics` 做正交估计与诊断。\n\n**作业**见同目录 `text_causal_assignment.md`。"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.x"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}