{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# StatsPAI Research Lab · From Data + Question to an Audited Estimate\n",
    "\n",
    "StatsPAI Summer Bootcamp 配套 notebook。配合知识页 `/courses/summer-bootcamp/topics/statspai-research`。\n",
    "\n",
    "走一遍 agent-native 因果工作流：**识别设计 → 推荐估计量 → 拟合（得到 handle）→ 审计稳健性 → 敏感性 → 可核验引用**。\n",
    "本 notebook 用 numpy 自跑一个 TWFE DiD（真实效应 ATT=2.0）模拟该流程；真实项目用 StatsPAI MCP 的 `did` / `callaway_santanna` / `audit_result` 等工具。\n",
    "\n",
    "> 依赖：`numpy`, `pandas`。三条红线：识别来自设计而非模型；估计后必须审计稳健性；引用只能来自可核验 bib，绝不杜撰。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 识别设计（detect_design）"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "execution_count": null,
   "source": [
    "import numpy as np, pandas as pd\n",
    "df = pd.read_csv('statspai_panel.csv')\n",
    "feat = {'panel': df.duplicated(['unit','year']).sum()==0 and df['unit'].nunique()>1,\n",
    "        'staggered_treatment': df['post'].nunique()==2}\n",
    "design = 'staggered_did' if feat['panel'] and feat['staggered_treatment'] else 'unknown'\n",
    "print('features:', feat)\n",
    "print('detected design:', design)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 推荐估计量（recommend）"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "execution_count": null,
   "source": [
    "RECOMMEND = {'staggered_did':'callaway_santanna','regression_discontinuity':'rdrobust',\n",
    "             'instrumental_variables':'ivreg','selection_on_observables':'dml'}\n",
    "print('recommended:', RECOMMEND[design])\n",
    "print('note: with homogeneous timing TWFE is fine; with heterogeneity prefer callaway_santanna')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 拟合：TWFE DiD（这里用 numpy 自算，真实项目交给 StatsPAI handle）"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "execution_count": null,
   "source": [
    "d = df.copy()\n",
    "d['y_dm'] = d['y'] - d.groupby('unit')['y'].transform('mean') - d.groupby('year')['y'].transform('mean') + d['y'].mean()\n",
    "d['post_dm'] = d['post'] - d.groupby('unit')['post'].transform('mean') - d.groupby('year')['post'].transform('mean') + d['post'].mean()\n",
    "beta = (d['post_dm']*d['y_dm']).sum() / (d['post_dm']**2).sum()\n",
    "resid = d['y_dm'] - beta*d['post_dm']\n",
    "se = np.sqrt((resid**2).sum()/(len(d)-1) / (d['post_dm']**2).sum())\n",
    "result = {'estimator':'twfe','att':round(float(beta),3),'se':round(float(se),3)}\n",
    "print('fit (true ATT = 2.0):', result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 审计：枚举仍缺的稳健性检查（audit_result）"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "execution_count": null,
   "source": [
    "recommended = {'pretrends_test','honest_did','sensitivity','placebo','cluster_se'}\n",
    "done = {'cluster_se'}\n",
    "print(f'coverage = {len(done)/len(recommended):.0%}')\n",
    "print('t-stat =', round(result['att']/result['se'], 2))\n",
    "print('still missing:', sorted(recommended - done))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 敏感性：E-value（混淆需多强才能解释掉效应）"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "execution_count": null,
   "source": [
    "import math\n",
    "def e_value(rr):\n",
    "    if rr < 1: rr = 1/rr\n",
    "    return round(rr + math.sqrt(rr*(rr-1)), 2)\n",
    "for rr in (1.2,1.5,2.0): print(f'RR={rr} -> E-value={e_value(rr)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. 引用纪律：只引可核验 bib，拒绝杜撰"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "outputs": [],
   "execution_count": null,
   "source": [
    "VERIFIED = {'callaway2021':\"Callaway & Sant'Anna (2021), J. Econometrics\",\n",
    "            'goodmanbacon2021':'Goodman-Bacon (2021), J. Econometrics'}\n",
    "def cite(keys):\n",
    "    ok=[k for k in keys if k in VERIFIED]; bad=[k for k in keys if k not in VERIFIED]\n",
    "    return {'cited':[VERIFIED[k] for k in ok], 'rejected_invented':bad}\n",
    "print(cite(['callaway2021','smith2099']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. 接下来\n",
    "- 把 numpy 估计换成 StatsPAI MCP：`detect_design` → `recommend` → `callaway_santanna(as_handle=True)` → `audit_result` → `sensitivity` → `bibtex`。\n",
    "- 用 `result_id` 把估计串进下游工具，不用反复传 beta / sigma。\n",
    "- 引用永远走 `bibtex`（paper.bib 单一来源），杜绝模型杜撰文献。\n",
    "\n",
    "**作业**见同目录 `statspai_research_assignment.md`。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.x"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}