Chain-of-Thought Program vs Direct LLM Calls Across 8 Independent Configurations

Context and Methodology

The objective is to evaluate whether a structured chain-of-thought program can amplify analytical performance beyond native direct-call behavior, especially for the smaller 20B model. Each direct call attempted to return a parseable numeric value under low, medium, and high reasoning settings. When all parse attempts failed, the output was recorded as ERR. Correctness is analyzed with the benchmark rule used in these logs and then stress-tested under stricter tolerance thresholds (absolute and percent). Because Level 8 tasks blend branching logic, numerical methods, and heavy distractors, this is intentionally a high-friction comparison designed to measure robustness, not only raw arithmetic precision.