Agentic Empirical Research Workflow: From Research Question to Auditable Evidence Bundle

This minimal scheduler records action, observation, and gate status at every step. Failures go to repair paths instead of being narrated as success.

from dataclasses import dataclass, field
from typing import Callable

@dataclass
class State:
    task: str
    data_ready: bool = False
    model_ready: bool = False
    report_ready: bool = False
    trace: list[dict] = field(default_factory=list)

def clean_data(state: State) -> dict:
    return {"rows": 1280, "missing_rate": 0.018, "artifact": "data/clean.parquet"}

def estimate_model(state: State) -> dict:
    if not state.data_ready:
        return {"error": "clean data missing"}
    return {"coef": 0.12, "se": 0.04, "artifact": "models/did.json"}

def render_report(state: State) -> dict:
    if not state.model_ready:
        return {"error": "model not verified"}
    return {"artifact": "outputs/report.md", "claim": "effect is positive"}

def gate(tool_name: str, obs: dict) -> str:
    if "error" in obs:
        return "fail"
    if tool_name == "clean_data" and obs["missing_rate"] < 0.05:
        return "pass"
    if tool_name == "estimate_model" and abs(obs["coef"] / obs["se"]) > 1.96:
        return "review"
    if tool_name == "render_report":
        return "pass"
    return "fail"

TOOLS: dict[str, Callable[[State], dict]] = {
    "clean_data": clean_data,
    "estimate_model": estimate_model,
    "render_report": render_report,
}

def choose_action(state: State) -> str:
    if not state.data_ready:
        return "clean_data"
    if not state.model_ready:
        return "estimate_model"
    return "render_report"

state = State(task="Build an auditable DID research note")

for _ in range(5):
    action = choose_action(state)
    observation = TOOLS[action](state)
    status = gate(action, observation)
    state.trace.append({"action": action, "observation": observation, "gate": status})

    if action == "clean_data" and status == "pass":
        state.data_ready = True
    elif action == "estimate_model" and status in {"pass", "review"}:
        # Human review is required before the claim is externally used.
        state.model_ready = True
    elif action == "render_report" and status == "pass":
        state.report_ready = True
        break

print("report ready:", state.report_ready)
print("trace:")
for row in state.trace:
    print(row)

• A student or researcher wants an agent to complete a policy-evaluation project: clean raw data, construct a panel, estimate DID, generate tables, write interpretation, and prepare a replication bundle.
• The low-quality version asks a model to write conclusions directly. The robust version builds a DAG: raw data, clean data, features, model, diagnostics, tables, figures, manuscript, and review notes.
• Every node needs a gate: sample counts, duplicate keys, convergence, parallel-trends figure, table provenance, and whether prose cites the right output.
• The deliverable is not a single answer. It is an auditable set of artifacts: processing logs, commands, tables, screenshots, human confirmations, and still-open identification assumptions.

References

• Yao et al. (2022), ReAct: Synergizing Reasoning and Acting in Language Modelshttps://arxiv.org/abs/2210.03629
• Project TIER Protocolhttps://www.projecttier.org/tier-protocol/
• ACM Artifact Review and Badginghttps://www.acm.org/publications/policies/artifact-review-and-badging-current
• AEA Data and Code Availability Policyhttps://www.aeaweb.org/journals/data/data-code-policy