Phase 1 & 2: Baseline experiments + AGoT implementation + reasoning graph analysis

## Added ### Core framework - src/llm_client.py — Ollama API wrapper for local Docker inference - src/samples.py — Embedded benchmark datasets (GSM8K/MATH/AIME + Chinese) - src/run_all.py — Unified experiment runner ### Phase 1: Baselines - src/baseline/benchmark.py — IO, CoT, CoT-SC, ToT evaluation with Chinese support ### Phase 2: AGoT - src/agot/agot.py — AGoT core algorithm (6 agent types, recursive decomposition) - src/agot/graph_utils.py — DAG graph data structure with cycle detection - src/agot/prompts.py — 6 agent prompt templates (EN + ZH) - src/agot/run.py — AGoT experiment runner ### Graph Analysis - src/graph_analysis/reasoning_graph.py — Graph property computation (cyclicity, diameter, small-world) - src/graph_analysis/visualize.py — Visualization charts ### Documentation - docs/graph_reasoning_principles.md — Comprehensive principles document - docs/next_steps.md — Roadmap for reliable reasoning tool ### Experiment Results - GSM8K/MATH baselines on qwen2.5:32b and qwq:latest - AGoT validation (100% on small sample) - Chinese dataset benchmarks (C-GSM8K, CMATH) - Graph property analysis confirming Topology of Reasoning findings
2026-07-03 22:23:42 +08:00
parent aa8ba6a6d7
commit d84f98799b
37 changed files with 4880 additions and 0 deletions
@@ -0,0 +1,103 @@
+{
+  "summary": {
+    "method": "IO",
+    "model": "qwen2.5:32b",
+    "task": "gsm8k",
+    "samples": 10,
+    "correct": 7,
+    "accuracy": 0.7,
+    "avg_time_s": 0.89
+  },
+  "details": [
+    {
+      "question": "Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins with four every day. She sells the remaining eggs at the farmers' market daily for $2 per dozen. How much money does she make per day at the farmers' market?",
+      "expected": "1.50",
+      "predicted": "2",
+      "full_response": "$2",
+      "correct": false,
+      "time_s": 0.51,
+      "tokens": 1
+    },
+    {
+      "question": "A robe takes 2 1/2 yards of fabric and 1/8 yard of trim. If you have 10 yards of fabric and 1 yard of trim, how many robes can you make?",
+      "expected": "4",
+      "predicted": "4",
+      "full_response": "4",
+      "correct": true,
+      "time_s": 0.37,
+      "tokens": 1
+    },
+    {
+      "question": "Kyle is a collector of rare coins. One day he found a rare coin online and bought it for $50. The next month, he sold it at a 20% profit. Six months later, he bought the same coin back, but had to pay 10% more than he sold it for. How much did Kyle pay to buy the coin back?",
+      "expected": "66",
+      "predicted": "66",
+      "full_response": "66 dollars",
+      "correct": true,
+      "time_s": 0.59,
+      "tokens": 4
+    },
+    {
+      "question": "A pet shelter has 9 kittens available for adoption. The shelter normally has 2 cats available. If each cat treats 2 kittens as their own, how many animals are there in all?",
+      "expected": "7",
+      "predicted": "18",
+      "full_response": "18",
+      "correct": false,
+      "time_s": 0.46,
+      "tokens": 1
+    },
+    {
+      "question": "Faye is a flower farmer. She plants 12 rows of daisies with 100 daisies in each row. She also plants 7 rows of roses with 12 roses in each row. How many more daisies than roses does she plant?",
+      "expected": "1116",
+      "predicted": "1056",
+      "full_response": "1056",
+      "correct": false,
+      "time_s": 0.67,
+      "tokens": 2
+    },
+    {
+      "question": "A farmer had 27 watermelons. After selling some watermelons, he had 13 left. Then he bought 28 more watermelons. Later, he sold 15 watermelons. How many watermelons does he have now?",
+      "expected": "26",
+      "predicted": "26",
+      "full_response": "26",
+      "correct": true,
+      "time_s": 0.48,
+      "tokens": 1
+    },
+    {
+      "question": "James buys a bag of 40 cookies. He eats 5 cookies and then divides the rest evenly among his 7 friends. How many cookies does each friend get?",
+      "expected": "5",
+      "predicted": "5",
+      "full_response": "5",
+      "correct": true,
+      "time_s": 0.36,
+      "tokens": 1
+    },
+    {
+      "question": "Tom's mother had 15 apples. She gave Tom 3 apples and gave Sarah twice as many as Tom. How many apples does she have left?",
+      "expected": "6",
+      "predicted": "6",
+      "full_response": "6",
+      "correct": true,
+      "time_s": 0.35,
+      "tokens": 1
+    },
+    {
+      "question": "A school bus can hold 48 students. If 3 buses are used for a field trip and each bus is 75% full, how many students are on the field trip?",
+      "expected": "108",
+      "predicted": "108",
+      "full_response": "108",
+      "correct": true,
+      "time_s": 0.56,
+      "tokens": 2
+    },
+    {
+      "question": "Maria bought 6 notebooks for $2 each and 4 pens for $1.50 each. She paid with a $20 bill. How much change did she receive?",
+      "expected": "2",
+      "predicted": "2",
+      "full_response": "$20 - (6 * $2 + 4 * $1.50) = $20 - ($12 + $6) = $20 - $18 = $2",
+      "correct": true,
+      "time_s": 4.55,
+      "tokens": 21
+    }
+  ]
+}