Files
s-LLM-project/data/results/gsm8k_IO_qwq_latest_20260703_152126.json
cat d84f98799b Phase 1 & 2: Baseline experiments + AGoT implementation + reasoning graph analysis
## Added

### Core framework
- src/llm_client.py — Ollama API wrapper for local Docker inference
- src/samples.py — Embedded benchmark datasets (GSM8K/MATH/AIME + Chinese)
- src/run_all.py — Unified experiment runner

### Phase 1: Baselines
- src/baseline/benchmark.py — IO, CoT, CoT-SC, ToT evaluation with Chinese support

### Phase 2: AGoT
- src/agot/agot.py — AGoT core algorithm (6 agent types, recursive decomposition)
- src/agot/graph_utils.py — DAG graph data structure with cycle detection
- src/agot/prompts.py — 6 agent prompt templates (EN + ZH)
- src/agot/run.py — AGoT experiment runner

### Graph Analysis
- src/graph_analysis/reasoning_graph.py — Graph property computation (cyclicity, diameter, small-world)
- src/graph_analysis/visualize.py — Visualization charts

### Documentation
- docs/graph_reasoning_principles.md — Comprehensive principles document
- docs/next_steps.md — Roadmap for reliable reasoning tool

### Experiment Results
- GSM8K/MATH baselines on qwen2.5:32b and qwq:latest
- AGoT validation (100% on small sample)
- Chinese dataset benchmarks (C-GSM8K, CMATH)
- Graph property analysis confirming Topology of Reasoning findings
2026-07-03 22:23:42 +08:00

103 lines
12 KiB
JSON

{
"summary": {
"method": "IO",
"model": "qwq:latest",
"task": "gsm8k",
"samples": 10,
"correct": 9,
"accuracy": 0.9,
"avg_time_s": 25.54
},
"details": [
{
"question": "Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins with four every day. She sells the remaining eggs at the farmers' market daily for $2 per dozen. How much money does she make per day at the farmers' market?",
"expected": "1.50",
"predicted": "1.50",
"full_response": "First, we need to find out how many eggs Janet has left after eating some and baking with others.\n\nJanet's ducks lay 16 eggs per day.\n\nShe eats 3 eggs for breakfast every morning and uses 4 eggs for baking muffins every day.\n\nSo, the total eggs used daily are 3 (for breakfast) + 4 (for baking) = 7 eggs.\n\nTherefore, the remaining eggs are 16 (total laid) - 7 (used) = 9 eggs.\n\nNow, she sells these remaining eggs at the farmers' market for $2 per dozen.\n\nFirst, we need to find out how many dozens are there in 9 eggs.\n\nSince 1 dozen equals 12 eggs, then 9 eggs is equal to 9/12 = 0.75 dozen.\n\nNow, if she sells each dozen for $2, then for 0.75 dozen, she would make 0.75 * $2 = $1.50.\n\nTherefore, Janet makes $1.50 per day at the farmers' market from selling eggs.\n\n**Final Answer**\n\n\\[ \\boxed{1.50} \\]",
"correct": true,
"time_s": 35.64,
"tokens": 269
},
{
"question": "A robe takes 2 1/2 yards of fabric and 1/8 yard of trim. If you have 10 yards of fabric and 1 yard of trim, how many robes can you make?",
"expected": "4",
"predicted": "4",
"full_response": "To determine how many robes you can make, we need to calculate how many robes can be made from the available fabric and trim separately, and then take the smaller number since both materials are required.\n\nFirst, let's find out how much fabric is needed for one robe. The problem states that each robe takes 2 1/2 yards of fabric. We can convert this mixed number to an improper fraction for easier calculation:\n\n\\[ 2 \\frac{1}{2} = \\frac{5}{2} \\text{ yards per robe} \\]\n\nNow, we have 10 yards of fabric available. To find out how many robes can be made from the fabric, we divide the total fabric by the fabric required per robe:\n\n\\[ \\text{Number of robes from fabric} = \\frac{10}{\\frac{5}{2}} = 10 \\times \\frac{2}{5} = 4 \\]\n\nSo, with 10 yards of fabric, you can make 4 robes.\n\nNext, let's look at the trim. Each robe requires 1/8 yard of trim. We have 1 yard of trim available. To find out how many robes can be made from the trim, we divide the total trim by the trim required per robe:\n\n\\[ \\text{Number of robes from trim} = \\frac{1}{\\frac{1}{8}} = 1 \\times 8 = 8 \\]\n\nSo, with 1 yard of trim, you can make 8 robes.\n\nHowever, since both fabric and trim are required to make a robe, the limiting factor is the fabric, which allows only 4 robes. Therefore, the number of robes you can make is:\n\n\\[ \\boxed{4} \\]",
"correct": true,
"time_s": 37.39,
"tokens": 437
},
{
"question": "Kyle is a collector of rare coins. One day he found a rare coin online and bought it for $50. The next month, he sold it at a 20% profit. Six months later, he bought the same coin back, but had to pay 10% more than he sold it for. How much did Kyle pay to buy the coin back?",
"expected": "66",
"predicted": "66",
"full_response": "Let's see. Kyle bought the coin for $50 and sold it with a 20% profit. So, first, I need to calculate what price he sold it for.\n\nA 20% profit on $50 would be:\n\n\\[ \\text{Profit} = 50 \\times \\frac{20}{100} = 50 \\times 0.20 = 10 \\]\n\nSo, he sold it for:\n\n\\[ \\text{Selling Price} = 50 + 10 = 60 \\]\n\nThen, six months later, he bought it back at 10% more than he sold it for. So, I need to find 10% of $60 and add it to the selling price.\n\n\\[ \\text{Additional Amount} = 60 \\times \\frac{10}{100} = 60 \\times 0.10 = 6 \\]\n\nTherefore, the buyback price is:\n\n\\[ \\text{Buyback Price} = 60 + 6 = 66 \\]\n\nSo, Kyle paid $66 to buy the coin back.\n\n**Final Answer**\n\n\\[ \\boxed{66} \\]",
"correct": true,
"time_s": 26.78,
"tokens": 222
},
{
"question": "A pet shelter has 9 kittens available for adoption. The shelter normally has 2 cats available. If each cat treats 2 kittens as their own, how many animals are there in all?",
"expected": "7",
"predicted": "11",
"full_response": "So, the shelter has 9 kittens and normally has 2 cats. Each cat treats 2 kittens as their own. I need to find out the total number of animals in the shelter.\n\nFirst, I should figure out what \"each cat treats 2 kittens as their own\" means. Does that mean that these kittens are being taken care of by the cats, or something like that?\n\nWait, maybe it's just about counting how many kittens are being cared for by the cats. But the question is about the total number of animals, which includes both cats and kittens.\n\nLet me see. If there are 2 cats and each treats 2 kittens as their own, that means there are 2 x 2 = 4 kittens being treated by the cats.\n\nBut there are 9 kittens in total. So, out of these 9 kittens, 4 are being treated by the cats, and the remaining 5 are not being treated by any cat.\n\nWait, but the question doesn't specify that only some kittens are being treated by the cats. It says each cat treats 2 kittens as their own, but it doesn't say that's all the kittens there are.\n\nMaybe I'm overcomplicating this. The shelter has 9 kittens and normally has 2 cats. So, the total number of animals would just be the sum of the kittens and the cats.\n\nSo, 9 kittens + 2 cats = 11 animals in total.\n\nBut the question mentions that each cat treats 2 kittens as their own. Does that mean that these kittens are considered to be part of the cats' groups, but still separate animals?\n\nI think the way to look at it is that the kittens are distinct animals from the cats, even if they are being treated by the cats.\n\nSo, the total number of animals is still 9 kittens plus 2 cats, which equals 11 animals.\n\nWait, but maybe there's something else I'm missing. Let's read the question again: \"If each cat treats 2 kittens as their own, how many animals are there in all?\"\n\nIt seems like the question is simply asking for the total number of animals, considering both cats and kittens, and the fact that cats are treating some kittens as their own doesn't change the count of individual animals.\n\nSo, the answer should be 11 animals in total.\n\n**Final Answer**\n\n\\[ \\boxed{11} \\]",
"correct": false,
"time_s": 51.37,
"tokens": 696
},
{
"question": "Faye is a flower farmer. She plants 12 rows of daisies with 100 daisies in each row. She also plants 7 rows of roses with 12 roses in each row. How many more daisies than roses does she plant?",
"expected": "1116",
"predicted": "1116",
"full_response": "First, I need to find out how many daisies Faye planted. She has 12 rows of daisies with 100 daisies in each row. So, the total number of daisies is 12 times 100, which is 1200 daisies.\n\nNext, I need to find out how many roses she planted. She has 7 rows of roses with 12 roses in each row. So, the total number of roses is 7 times 12, which is 84 roses.\n\nNow, to find out how many more daisies than roses she planted, I subtract the number of roses from the number of daisies. That's 1200 minus 84, which equals 1116.\n\nSo, Faye planted 1116 more daisies than roses.",
"correct": true,
"time_s": 19.41,
"tokens": 189
},
{
"question": "A farmer had 27 watermelons. After selling some watermelons, he had 13 left. Then he bought 28 more watermelons. Later, he sold 15 watermelons. How many watermelons does he have now?",
"expected": "26",
"predicted": "26",
"full_response": "So the farmer started with 27 watermelons. He sold some and had 13 left. Then he bought 28 more, and after that, he sold 15. I need to find out how many he has now.\n\nFirst, he had 27 and sold some to have 13 left. So, the number he sold first is 27 minus 13, which is 14 watermelons.\n\nBut I don't really need to know how many he sold; I just need to know he has 13 after selling some.\n\nThen, he bought 28 more, so now he has 13 plus 28, which is 41 watermelons.\n\nAfter that, he sold 15, so he has 41 minus 15, which is 26 watermelons.\n\nSo, the farmer has 26 watermelons now.\n\n**Final Answer**\n\n\\[ \\boxed{26} \\]",
"correct": true,
"time_s": 21.35,
"tokens": 204
},
{
"question": "James buys a bag of 40 cookies. He eats 5 cookies and then divides the rest evenly among his 7 friends. How many cookies does each friend get?",
"expected": "5",
"predicted": "5",
"full_response": "So James starts with 40 cookies. He eats 5, so that leaves him with 35 cookies. Then he divides these 35 cookies evenly among his 7 friends. To find out how many cookies each friend gets, you divide 35 by 7, which is 5. So each friend gets 5 cookies.",
"correct": true,
"time_s": 7.53,
"tokens": 84
},
{
"question": "Tom's mother had 15 apples. She gave Tom 3 apples and gave Sarah twice as many as Tom. How many apples does she have left?",
"expected": "6",
"predicted": "6",
"full_response": "Let's see. Tom's mother started with 15 apples. She gave Tom 3 apples. So, she has 15 - 3 = 12 apples left. Then, she gave Sarah twice as many as she gave Tom. Since she gave Tom 3 apples, twice that amount would be 2 * 3 = 6 apples to Sarah. So, she gave away an additional 6 apples to Sarah. Therefore, the number of apples she has left is 12 - 6 = 6 apples.\n\n**Final Answer**\n\n\\[ \\boxed{6} \\]",
"correct": true,
"time_s": 12.79,
"tokens": 132
},
{
"question": "A school bus can hold 48 students. If 3 buses are used for a field trip and each bus is 75% full, how many students are on the field trip?",
"expected": "108",
"predicted": "108",
"full_response": "So, each bus can hold 48 students, and there are 3 buses being used. Each bus is 75% full. To find out how many students are on the field trip, I need to calculate 75% of 48 for one bus and then multiply that by 3 because there are three buses.\n\nFirst, I'll find 75% of 48. Percentage means per hundred, so 75% is the same as 75/100 or 0.75 in decimal form. So, 0.75 times 48 should give me the number of students per bus.\n\n0.75 * 48 = 36\n\nSo, each bus has 36 students. Since there are three buses, I need to multiply 36 by 3.\n\n36 * 3 = 108\n\nTherefore, there are 108 students on the field trip.\n\nBut just to make sure, maybe I should check my calculations again. First, 75% of 48:\n\n75% = 0.75\n\n0.75 * 48 = 36, that seems correct.\n\nThen, 36 students per bus times 3 buses:\n\n36 * 3 = 108\n\nYes, that's correct. So, the final answer is 108 students.\n\n**Final Answer**\n\n\\[ \\boxed{108} \\]",
"correct": true,
"time_s": 32.03,
"tokens": 295
},
{
"question": "Maria bought 6 notebooks for $2 each and 4 pens for $1.50 each. She paid with a $20 bill. How much change did she receive?",
"expected": "2",
"predicted": "2",
"full_response": "First, calculate the total cost of the notebooks: 6 notebooks * $2/notebook = $12.\n\nNext, calculate the total cost of the pens: 4 pens * $1.50/pen = $6.\n\nNow, add the costs together to find the total amount spent: $12 + $6 = $18.\n\nFinally, subtract the total amount spent from the amount Maria paid: $20 - $18 = $2.\n\nSo, Maria received $2 in change.",
"correct": true,
"time_s": 11.13,
"tokens": 117
}
]
}