​Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., West, P., Bhagavatula, C., Le Bras, R., Hwang, J. D., Sanyal, S., Welleck, S., Ren, X., Ettinger, A., Harchaoui, Z., & Choi, Y. (2023). Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems (NeurIPS2023).https://arxiv.org/abs/2305.18654  

Hong, P., Majumder, N., Ghosal, D., Aditya, S., Mihalcea, R., & Poria, S. (2024). Evaluating LLMs’ mathematical and coding competency through ontology-guided interventions. arXiv. https://arxiv.org/abs/2401.09395

Jiang, B., Xie, Y., Hao, Z., Wang, X., Mallick, T., Su, W. J., Taylor, C. J., & Roth, D. (2024). A peek into token bias: Large language models are not yet genuine reasoners. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 4722–4756). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.272 

Kambhampati, S. (2024). Can large language models reason and plan? arXiv. https://arxiv.org/abs/2403.04121

Lewis, M., & Mitchell, M. (2024). Using counterfactual tasks to evaluate the generality of analogical reasoning in large language models. arXiv. https://doi.org/10.48550/arXiv.2402.08955 

McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024). Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 121(41), e2322420121. https://doi.org/10.1073/pnas.2322420121 

Mirzadeh, S. I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., & Farajtabar, M. (2025). GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models. In International Conference on Learning Representations (ICLR 2025). https://openreview.net/forum?id=AjXkRZIvjB 

Mondorf, P., & Plank, B. (2024). Beyond accuracy: Evaluating the reasoning behavior of large language models—A survey. arXiv. https://arxiv.org/abs/2404.01869 

Nezhurina, M., Cipolina-Kun, A., Cherti, M., Borovykh, A., Oseledets, I., & Jitsev, A. (2024). Alice in Wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models. arXiv.https://arxiv.org/abs/2406.02061 

Prabhakar, A., Griffiths, T. L., & McCoy, R. T. (2024). Deciphering the factors influencing the efficacy of chain-of-thought: Probability, memorization, and noisy reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 3710–3724). Association for Computational Linguistics. https://aclanthology.org/2024.findings-emnlp.212

Qiu, L., Jiang, L., Lu, X., Sclar, M., Pyatkin, V., Bhagavatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., & Ren, X. (2024). Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In International Conference on Learning Representations (ICLR 2024). LINK  

Srivastava, S., Annarose, M. B., Anto, P. V., Menon, S., Sukumar, A., Samod, A., Philipose, A., Prince, S., & Thomas, S. (2024). Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv. https://arxiv.org/abs/2402.19450 

Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y. (2024). Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies (pp. 1819–1862). Association for Computational Linguistics. LINK

Yan, J., Wang, C., Huang, J., & Zhang, W. (2024). Do large language models understand logic or just mimick context? arXiv. LINK