cat
d84f98799b
## Added ### Core framework - src/llm_client.py — Ollama API wrapper for local Docker inference - src/samples.py — Embedded benchmark datasets (GSM8K/MATH/AIME + Chinese) - src/run_all.py — Unified experiment runner ### Phase 1: Baselines - src/baseline/benchmark.py — IO, CoT, CoT-SC, ToT evaluation with Chinese support ### Phase 2: AGoT - src/agot/agot.py — AGoT core algorithm (6 agent types, recursive decomposition) - src/agot/graph_utils.py — DAG graph data structure with cycle detection - src/agot/prompts.py — 6 agent prompt templates (EN + ZH) - src/agot/run.py — AGoT experiment runner ### Graph Analysis - src/graph_analysis/reasoning_graph.py — Graph property computation (cyclicity, diameter, small-world) - src/graph_analysis/visualize.py — Visualization charts ### Documentation - docs/graph_reasoning_principles.md — Comprehensive principles document - docs/next_steps.md — Roadmap for reliable reasoning tool ### Experiment Results - GSM8K/MATH baselines on qwen2.5:32b and qwq:latest - AGoT validation (100% on small sample) - Chinese dataset benchmarks (C-GSM8K, CMATH) - Graph property analysis confirming Topology of Reasoning findings
40 lines
4.7 KiB
JSON
40 lines
4.7 KiB
JSON
{
|
|
"summary": {
|
|
"method": "CoT",
|
|
"model": "qwen2.5:32b",
|
|
"task": "gsm8k",
|
|
"samples": 3,
|
|
"correct": 3,
|
|
"accuracy": 1.0,
|
|
"avg_time_s": 33.54
|
|
},
|
|
"details": [
|
|
{
|
|
"question": "Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins with four every day. She sells the remaining eggs at the farmers' market daily for $2 per dozen. How much money does she make per day at the farmers' market?",
|
|
"expected": "1.50",
|
|
"predicted": "1.50",
|
|
"full_response": "Step 1: Calculate the total number of eggs laid by Janet's ducks each day.\n- The ducks lay 16 eggs per day.\n\nStep 2: Determine how many eggs Janet uses for breakfast and baking daily.\n- She eats three eggs for breakfast every morning.\n- She bakes muffins with four eggs every day.\n- Total eggs used daily = 3 (for breakfast) + 4 (for baking) = 7 eggs\n\nStep 3: Calculate the number of eggs remaining after Janet uses some for breakfast and baking.\n- Remaining eggs per day = total laid - used\n- Remaining eggs per day = 16 - 7 = 9 eggs\n\nStep 4: Determine how many dozens of eggs are available to sell at the farmers' market.\n- There are 12 eggs in a dozen.\n- Number of dozens Janet can sell = remaining eggs / 12\n- Number of dozens Janet can sell = 9 / 12 = 0.75 dozens\n\nStep 5: Calculate how much money Janet makes per day from selling the eggs at $2 per dozen.\n- Money made per day = number of dozens sold * price per dozen\n- Money made per day = 0.75 dozens * $2/dozen = $1.50\n\nTherefore, Janet makes \\boxed{1.50} dollars per day at the farmers' market from selling her ducks' eggs.",
|
|
"correct": true,
|
|
"time_s": 30.7,
|
|
"tokens": 362
|
|
},
|
|
{
|
|
"question": "A robe takes 2 1/2 yards of fabric and 1/8 yard of trim. If you have 10 yards of fabric and 1 yard of trim, how many robes can you make?",
|
|
"expected": "4",
|
|
"predicted": "4",
|
|
"full_response": "To solve this problem, we need to determine the number of robes that can be made given the constraints on both fabric and trim.\n\nStep 1: Calculate the amount of fabric needed for one robe.\n- A single robe requires \\(2 \\frac{1}{2}\\) yards of fabric. Converting this mixed number to an improper fraction:\n\\[ 2 \\frac{1}{2} = \\frac{5}{2} \\text{ yards} \\]\n\nStep 2: Calculate the amount of trim needed for one robe.\n- A single robe requires \\( \\frac{1}{8} \\) yard of trim.\n\nStep 3: Determine how many robes can be made with the available fabric.\n- We have 10 yards of fabric. To find out how many robes we can make, divide the total amount of fabric by the amount needed per robe:\n\\[ \\text{Number of robes from fabric} = \\frac{10}{\\frac{5}{2}} = 10 \\times \\frac{2}{5} = 4 \\]\n\nStep 4: Determine how many robes can be made with the available trim.\n- We have 1 yard of trim. To find out how many robes we can make, divide the total amount of trim by the amount needed per robe:\n\\[ \\text{Number of robes from trim} = \\frac{1}{\\frac{1}{8}} = 1 \\times 8 = 8 \\]\n\nStep 5: Determine the limiting factor.\n- The number of robes we can make is limited by the smaller of the two quantities calculated in Steps 3 and 4. Here, fabric limits us to 4 robes.\n\nTherefore, the final answer is:\n\\boxed{4}",
|
|
"correct": true,
|
|
"time_s": 37.48,
|
|
"tokens": 426
|
|
},
|
|
{
|
|
"question": "Kyle is a collector of rare coins. One day he found a rare coin online and bought it for $50. The next month, he sold it at a 20% profit. Six months later, he bought the same coin back, but had to pay 10% more than he sold it for. How much did Kyle pay to buy the coin back?",
|
|
"expected": "66",
|
|
"predicted": "66",
|
|
"full_response": "Step 1: Calculate the selling price of the coin when Kyle sold it at a 20% profit.\n\nKyle bought the coin for $50 and then sold it with a 20% profit. To find out how much he sold it for, we calculate:\n\n\\[ \\text{Selling Price} = \\text{Cost Price} + (\\text{Profit Percentage} \\times \\text{Cost Price}) \\]\n\\[ \\text{Selling Price} = \\$50 + (0.20 \\times \\$50) \\]\n\\[ \\text{Selling Price} = \\$50 + \\$10 \\]\n\\[ \\text{Selling Price} = \\$60 \\]\n\nStep 2: Calculate the price Kyle paid to buy back the coin, which was 10% more than what he sold it for.\n\nKyle bought the coin back at a price that is 10% higher than $60. To find this amount:\n\n\\[ \\text{New Buying Price} = \\text{Selling Price} + (\\text{Increase Percentage} \\times \\text{Selling Price}) \\]\n\\[ \\text{New Buying Price} = \\$60 + (0.10 \\times \\$60) \\]\n\\[ \\text{New Buying Price} = \\$60 + \\$6 \\]\n\\[ \\text{New Buying Price} = \\$66 \\]\n\nTherefore, Kyle paid $66 to buy the coin back.\n\n\\boxed{66}",
|
|
"correct": true,
|
|
"time_s": 32.43,
|
|
"tokens": 313
|
|
}
|
|
]
|
|
} |