Files
s-LLM-project/data/results/gsm8k_CoT_qwen2.5_32b_20260703_150510.json
cat d84f98799b Phase 1 & 2: Baseline experiments + AGoT implementation + reasoning graph analysis
## Added

### Core framework
- src/llm_client.py — Ollama API wrapper for local Docker inference
- src/samples.py — Embedded benchmark datasets (GSM8K/MATH/AIME + Chinese)
- src/run_all.py — Unified experiment runner

### Phase 1: Baselines
- src/baseline/benchmark.py — IO, CoT, CoT-SC, ToT evaluation with Chinese support

### Phase 2: AGoT
- src/agot/agot.py — AGoT core algorithm (6 agent types, recursive decomposition)
- src/agot/graph_utils.py — DAG graph data structure with cycle detection
- src/agot/prompts.py — 6 agent prompt templates (EN + ZH)
- src/agot/run.py — AGoT experiment runner

### Graph Analysis
- src/graph_analysis/reasoning_graph.py — Graph property computation (cyclicity, diameter, small-world)
- src/graph_analysis/visualize.py — Visualization charts

### Documentation
- docs/graph_reasoning_principles.md — Comprehensive principles document
- docs/next_steps.md — Roadmap for reliable reasoning tool

### Experiment Results
- GSM8K/MATH baselines on qwen2.5:32b and qwq:latest
- AGoT validation (100% on small sample)
- Chinese dataset benchmarks (C-GSM8K, CMATH)
- Graph property analysis confirming Topology of Reasoning findings
2026-07-03 22:23:42 +08:00

103 lines
12 KiB
JSON

{
"summary": {
"method": "CoT",
"model": "qwen2.5:32b",
"task": "gsm8k",
"samples": 10,
"correct": 8,
"accuracy": 0.8,
"avg_time_s": 25.93
},
"details": [
{
"question": "Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins with four every day. She sells the remaining eggs at the farmers' market daily for $2 per dozen. How much money does she make per day at the farmers' market?",
"expected": "1.50",
"predicted": "1.50",
"full_response": "Step 1: Calculate the total number of eggs laid by Janet's ducks each day.\n- The ducks lay 16 eggs per day.\n\nStep 2: Determine how many eggs Janet uses for breakfast and baking daily.\n- She eats three eggs for breakfast every morning.\n- She bakes muffins with four eggs every day.\n- Total eggs used daily = 3 (for breakfast) + 4 (for baking) = 7 eggs\n\nStep 3: Calculate the number of eggs remaining after Janet uses some for breakfast and baking.\n- Remaining eggs per day = total laid - used\n- Remaining eggs per day = 16 - 7 = 9 eggs\n\nStep 4: Determine how many dozens of eggs are available to sell at the farmers' market.\n- There are 12 eggs in a dozen.\n- Number of dozens Janet can sell = remaining eggs / 12\n- Number of dozens Janet can sell = 9 / 12 = 0.75 dozens\n\nStep 5: Calculate how much money Janet makes per day from selling the eggs at $2 per dozen.\n- Money made per day = number of dozens sold * price per dozen\n- Money made per day = 0.75 dozens * $2/dozen = $1.50\n\nTherefore, Janet makes \\boxed{1.50} dollars per day at the farmers' market from selling her ducks' eggs.",
"correct": true,
"time_s": 30.82,
"tokens": 362
},
{
"question": "A robe takes 2 1/2 yards of fabric and 1/8 yard of trim. If you have 10 yards of fabric and 1 yard of trim, how many robes can you make?",
"expected": "4",
"predicted": "4",
"full_response": "To solve this problem, we need to determine the number of robes that can be made given the constraints on both fabric and trim.\n\nStep 1: Calculate the amount of fabric needed for one robe.\n- A single robe requires \\(2 \\frac{1}{2}\\) yards of fabric. Converting this mixed number to an improper fraction:\n\\[ 2 \\frac{1}{2} = \\frac{5}{2} \\text{ yards} \\]\n\nStep 2: Calculate the amount of trim needed for one robe.\n- A single robe requires \\( \\frac{1}{8} \\) yard of trim.\n\nStep 3: Determine how many robes can be made with the available fabric.\n- We have 10 yards of fabric. To find out how many robes we can make, divide the total amount of fabric by the amount needed per robe:\n\\[ \\text{Number of robes from fabric} = \\frac{10}{\\frac{5}{2}} = 10 \\times \\frac{2}{5} = 4 \\]\n\nStep 4: Determine how many robes can be made with the available trim.\n- We have 1 yard of trim. To find out how many robes we can make, divide the total amount of trim by the amount needed per robe:\n\\[ \\text{Number of robes from trim} = \\frac{1}{\\frac{1}{8}} = 1 \\times 8 = 8 \\]\n\nStep 5: Determine the limiting factor.\n- The number of robes we can make is limited by the smaller of the two quantities calculated in Steps 3 and 4. Here, fabric limits us to 4 robes.\n\nTherefore, the final answer is:\n\\boxed{4}",
"correct": true,
"time_s": 37.42,
"tokens": 426
},
{
"question": "Kyle is a collector of rare coins. One day he found a rare coin online and bought it for $50. The next month, he sold it at a 20% profit. Six months later, he bought the same coin back, but had to pay 10% more than he sold it for. How much did Kyle pay to buy the coin back?",
"expected": "66",
"predicted": "66",
"full_response": "Step 1: Calculate the selling price of the coin when Kyle sold it at a 20% profit.\n\nKyle bought the coin for $50 and then sold it with a 20% profit. To find out how much he sold it for, we calculate:\n\n\\[ \\text{Selling Price} = \\text{Cost Price} + (\\text{Profit Percentage} \\times \\text{Cost Price}) \\]\n\\[ \\text{Selling Price} = \\$50 + (0.20 \\times \\$50) \\]\n\\[ \\text{Selling Price} = \\$50 + \\$10 \\]\n\\[ \\text{Selling Price} = \\$60 \\]\n\nStep 2: Calculate the price Kyle paid to buy back the coin, which was 10% more than what he sold it for.\n\nKyle bought the coin back at a price that is 10% higher than $60. To find this amount:\n\n\\[ \\text{New Buying Price} = \\text{Selling Price} + (\\text{Increase Percentage} \\times \\text{Selling Price}) \\]\n\\[ \\text{New Buying Price} = \\$60 + (0.10 \\times \\$60) \\]\n\\[ \\text{New Buying Price} = \\$60 + \\$6 \\]\n\\[ \\text{New Buying Price} = \\$66 \\]\n\nTherefore, Kyle paid $66 to buy the coin back.\n\n\\boxed{66}",
"correct": true,
"time_s": 32.49,
"tokens": 313
},
{
"question": "A pet shelter has 9 kittens available for adoption. The shelter normally has 2 cats available. If each cat treats 2 kittens as their own, how many animals are there in all?",
"expected": "7",
"predicted": "11",
"full_response": "To solve this problem, we need to determine the total number of animals at the pet shelter after considering both the kittens and the cats.\n\nStep 1: Identify the number of kittens available for adoption.\n- There are 9 kittens available for adoption.\n\nStep 2: Determine the normal number of cats at the shelter.\n- The shelter normally has 2 cats available.\n\nStep 3: Understand how each cat treats the kittens.\n- Each cat treats 2 kittens as their own. However, this information does not change the actual count of animals but might be a way to describe the relationship between the cats and kittens.\n\nStep 4: Calculate the total number of animals by adding the number of kittens and the number of cats.\n- Total number of animals = Number of kittens + Number of cats\n- Total number of animals = 9 (kittens) + 2 (cats)\n- Total number of animals = 11\n\nTherefore, the total number of animals at the pet shelter is \\boxed{11}.",
"correct": false,
"time_s": 21.72,
"tokens": 307
},
{
"question": "Faye is a flower farmer. She plants 12 rows of daisies with 100 daisies in each row. She also plants 7 rows of roses with 12 roses in each row. How many more daisies than roses does she plant?",
"expected": "1116",
"predicted": "1116",
"full_response": "To solve this problem, we need to calculate the total number of daisies and the total number of roses Faye plants, then find the difference between these two quantities.\n\nStep 1: Calculate the total number of daisies.\nFaye plants 12 rows with 100 daisies in each row. Therefore, the total number of daisies is:\n\\[ \\text{Total Daisies} = 12 \\times 100 = 1200 \\]\n\nStep 2: Calculate the total number of roses.\nFaye plants 7 rows with 12 roses in each row. Therefore, the total number of roses is:\n\\[ \\text{Total Roses} = 7 \\times 12 = 84 \\]\n\nStep 3: Find how many more daisies than roses Faye plants by subtracting the total number of roses from the total number of daisies.\n\\[ \\text{Difference} = \\text{Total Daisies} - \\text{Total Roses} = 1200 - 84 = 1116 \\]\n\nTherefore, Faye plants 1116 more daisies than roses.\n\n\\boxed{1116}",
"correct": true,
"time_s": 27.07,
"tokens": 276
},
{
"question": "A farmer had 27 watermelons. After selling some watermelons, he had 13 left. Then he bought 28 more watermelons. Later, he sold 15 watermelons. How many watermelons does he have now?",
"expected": "26",
"predicted": "26",
"full_response": "Step 1: Determine the number of watermelons sold initially.\nThe farmer started with 27 watermelons and had 13 left after selling some, so we can calculate how many were sold:\n\\[ \\text{Watermelons sold} = 27 - 13 = 14 \\]\n\nStep 2: Calculate the number of watermelons after buying more.\nThe farmer then bought 28 more watermelons. So, we add these to the remaining 13:\n\\[ \\text{Total watermelons after purchase} = 13 + 28 = 41 \\]\n\nStep 3: Determine the final number of watermelons after selling again.\nThe farmer sold another 15 watermelons from his new total. So, we subtract these from the current total:\n\\[ \\text{Final number of watermelons} = 41 - 15 = 26 \\]\n\nTherefore, the final answer is: \\boxed{26}",
"correct": true,
"time_s": 21.92,
"tokens": 235
},
{
"question": "James buys a bag of 40 cookies. He eats 5 cookies and then divides the rest evenly among his 7 friends. How many cookies does each friend get?",
"expected": "5",
"predicted": "5",
"full_response": "Step 1: Determine how many cookies are left after James eats some.\nJames starts with 40 cookies and eats 5 of them, so we subtract:\n\\[ 40 - 5 = 35 \\]\nSo there are 35 cookies remaining.\n\nStep 2: Divide the remaining cookies among his friends.\nJames has 7 friends and he divides the 35 remaining cookies evenly among them. We need to divide 35 by 7:\n\\[ 35 \\div 7 = 5 \\]\n\nEach friend gets 5 cookies.\n\n\\boxed{5}",
"correct": true,
"time_s": 12.73,
"tokens": 136
},
{
"question": "Tom's mother had 15 apples. She gave Tom 3 apples and gave Sarah twice as many as Tom. How many apples does she have left?",
"expected": "6",
"predicted": "6",
"full_response": "Step 1: Determine how many apples Tom received.\nTom received 3 apples from his mother.\n\nStep 2: Calculate the number of apples given to Sarah.\nSarah received twice as many apples as Tom, so:\n\\[ \\text{Apples for Sarah} = 2 \\times \\text{Apples for Tom} = 2 \\times 3 = 6 \\]\n\nStep 3: Find out how many apples were given away in total.\nTotal apples given to both Tom and Sarah is the sum of what each received:\n\\[ \\text{Total apples given away} = \\text{Apples for Tom} + \\text{Apples for Sarah} = 3 + 6 = 9 \\]\n\nStep 4: Calculate how many apples are left with Tom's mother.\nTom's mother originally had 15 apples. Subtract the total number of apples given away from this amount:\n\\[ \\text{Remaining apples} = \\text{Original apples} - \\text{Total apples given away} = 15 - 9 = 6 \\]\n\nTherefore, the final answer is: \\boxed{6}",
"correct": true,
"time_s": 24.18,
"tokens": 272
},
{
"question": "A school bus can hold 48 students. If 3 buses are used for a field trip and each bus is 75% full, how many students are on the field trip?",
"expected": "108",
"predicted": "108",
"full_response": "Step 1: Determine the capacity of one bus.\nThe problem states that one school bus can hold 48 students.\n\nStep 2: Calculate the number of students in one bus when it is 75% full.\nTo find out how many students are on a single bus at 75% capacity, we multiply the total capacity by 0.75 (which represents 75%).\n\n\\[ \\text{Number of students per bus} = 48 \\times 0.75 \\]\n\nStep 3: Perform the multiplication.\n\\[ 48 \\times 0.75 = 36 \\]\nSo, each bus has 36 students when it is 75% full.\n\nStep 4: Calculate the total number of students on all three buses.\nSince there are 3 buses and each carries 36 students at 75% capacity:\n\n\\[ \\text{Total number of students} = 3 \\times 36 \\]\n\nStep 5: Perform the multiplication to find the total number of students.\n\\[ 3 \\times 36 = 108 \\]\nTherefore, there are a total of 108 students on the field trip.\n\n\\boxed{108}",
"correct": true,
"time_s": 26.76,
"tokens": 282
},
{
"question": "Maria bought 6 notebooks for $2 each and 4 pens for $1.50 each. She paid with a $20 bill. How much change did she receive?",
"expected": "2",
"predicted": "\\$2",
"full_response": "Step 1: Calculate the total cost of the notebooks.\nMaria bought 6 notebooks at $2 each, so the total cost is:\n\\[ \\text{Total cost of notebooks} = 6 \\times \\$2 = \\$12 \\]\n\nStep 2: Calculate the total cost of the pens.\nShe also bought 4 pens at $1.50 each, so the total cost for the pens is:\n\\[ \\text{Total cost of pens} = 4 \\times \\$1.50 = \\$6 \\]\n\nStep 3: Find the overall total cost.\nAdd the costs from Step 1 and Step 2 to get the overall total cost:\n\\[ \\text{Overall total cost} = \\$12 + \\$6 = \\$18 \\]\n\nStep 4: Calculate the change Maria received.\nMaria paid with a $20 bill, so subtract the overall total cost from $20 to find out how much change she received:\n\\[ \\text{Change received} = \\$20 - \\$18 = \\$2 \\]\n\nTherefore, the amount of change Maria received is \\boxed{\\$2}.",
"correct": false,
"time_s": 24.18,
"tokens": 259
}
]
}