I‑AIMS Institute for Artificial Intelligence
and Mathematical Sciences
/

Hierarchical Grading in Large Language Models

Tanush Shaska

I-AIMS paper sh-109 · 2026

MSC classes: Primary 68T07; Secondary 68T50, 16W50

References

  1. T. Shaska. Artificial Neural Networks on Graded Vector Spaces. To appear in Contemporary Mathematics, 2026. Also available as arXiv:2407.19031, 2024.
  2. T. Shaska. Graded Neural Networks. Submitted to International Journal of Data Science in the Mathematical Sciences arXiv:2502.17751, 2025.
  3. T. Shaska. Graded Transformers: A Symbolic-Geometric Approach to Structured Learning. Submitted to IEEE Transactions on Neural Networks and Learning Systems arXiv:2507.20108 (2025).
  4. E. Shaska and T. Shaska. Machine learning for moduli space of genus two curves and an application to isogeny-based cryptography. Journal of Algebraic Combinatorics, 2025. DOI: 10.1007/s10801-025-01393-8.
  5. T. Shaska. Graded Quantum Codes: From Weighted Algebraic Geometry to Homological Chain Complexes. arXiv:2508.07542, 2025.
  6. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All You Need. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017), pp. 5998–6008, 2017.
  7. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling Laws for Neural Language Models. arXiv:2001.08361, 2020.
  8. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 2019.
  9. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019, pp. 4171–4186, 2019. Also available as arXiv:1810.04805 (2018).
  10. M. Belkin, D. Hsu, and P. Ma. Two models of double descent and their four kinds of double descent. arXiv:2012.06566, 2020.
  11. M. El, Y. Zhang, and L. Qin. Robustness in Sequence Modeling: A Survey. arXiv:2109.13978, 2021.
  12. R. Child, S. Gray, A. Radford, and I. Sutskever. Generating Long Sequences with Sparse Transformers. arXiv:1904.10509, 2019.
  13. N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The Efficient Transformer. In Proceedings of ICLR 2020, 2020.
  14. A. Jacot, F. Gabriel, and C. Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018), pp. 8571–8580, 2018.
  15. P. L. Bartlett and S. Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. Journal of Machine Learning Research, 3:463–482, 2002.
  16. M. Anthony and P. L. Bartlett. Neural Network Learning: Theoretical Foundations. Cambridge University Press, 1999.
  17. I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and Harnessing Adversarial Examples. arXiv:1412.6572, 2015. (ICLR 2015).
  18. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. In Proceedings of ICLR 2018, 2018. Also available as arXiv:1706.06083.
  19. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2018.
  20. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
  21. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Nash, M. Dehghani, M. Devlin, M. Chang, X. Lee, and K. Toutanova. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.
  22. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
  23. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
  24. T. Dettmers, A. Pagnoni, A. Holzinger, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.
  25. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  26. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023.
  27. A. Sergeev and M. Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv:1802.05799, 2018.
  28. Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In NeurIPS, 2019.
  29. S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16, 2020.
  30. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In AISTATS, 2017.
  31. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
  32. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
  33. X. Xie, P. Wang, X. Wang, H. Yang, J. Bai, Q. Hou, M. Li, X. Gao, W. Liu, J. Guo, et al. The ademamix optimizer: Better, faster, older. arXiv preprint arXiv:2409.03137, 2024.
  34. S. Liu, X. Zeng, S. Guo, and J. Tang. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793, 2024.
  35. X. Chen and C. Du. Symbolic discovery of optimization algorithms. In NeurIPS, 2023.
  36. C. E. Shannon. Prediction and entropy of printed english. The Bell System Technical Journal, 30(1):50–64, 1951.
  37. S. F. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. In Proceedings of the 34th annual meeting on Association for Computational Linguistics, pages 310–318, 1996.
  38. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155, 2003.
  39. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
  40. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
  41. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, 2018.
  42. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pages 3104–3112, 2014.
  43. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, 2017.
  44. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2018.
  45. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. 2018.
  46. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Nash, M. Dehghani, M. Devlin, M. Chang, X. Lee, and K. Toutanova. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.
  47. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  48. I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
  49. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Ramesh, D. Dziedzic, I. Sidor, S. Black, G. Millican, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  50. N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), 2021.
  51. C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2017.
  52. Z. Ji, T. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, Z. Dai, and D. Lewis. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023.
  53. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Nature, 323(6088):533–536, 1986.
  54. Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166, 1994.
  55. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
  56. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pages 3104–3112, 2014.
  57. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
  58. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, 2018.
  59. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, 2017.
  60. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2018.
  61. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  62. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. 2018.
  63. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019.
  64. K. Clark, M. Luong, Q. V. Le, and C. D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020.
  65. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  66. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
  67. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Nash, M. Dehghani, M. Devlin, M. Chang, X. Lee, and K. Toutanova. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.
  68. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohammad, R. Levy, V. Stoyanov, and L. Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, 2020.
  69. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  70. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  71. A. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, I. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, W. Larcher, L. Sauvageau, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
  72. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishra, J. Wu, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763, 2021.
  73. J.-B. Alayrac, J. Donahue, P. Lucic, A. Meyerson, J. Tulinsky, E. Barnes, D. Hertzmann, O. Saito, K. Sato, A. Sax, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
  74. H. Liu, C. Cao, J. Yang, J. Li, L. Li, Y. Du, and S. Yang. Llava: Large language and vision assistant. arXiv preprint arXiv:2304.08485, 2023.
  75. V. Dubois, A. Fan, C. Fournier, T. Wolf, and P.-A. Crook. Llama 3.1. Meta AI Report, 2024.
  76. OpenAI. Introducing gpt-4o. OpenAI Blog, May 2024.
  77. xAI Team. Grok-2: Towards multimodal understanding. xAI Blog, August 2024.
  78. A. Gu, A. C. Peters, M. Dao, J. Ba, and T. Dao. Mamba-2: Algorithms and systems. arXiv preprint arXiv:2405.21060, 2024.
  79. Mistral AI Team. Mistral large 2. Mistral AI Blog, July 2024.
  80. E. Labuschagne, J. Manyika, and J. E. Blumenstock. The 2025 ai index report. Stanford HAI, 2025.
  81. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, LKaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, 2017.
  82. K. T. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, L. Sarlós, P. Hawkins, J. Davis, A. Mohiuddin, L. Kung, et al. Rethinking attention with performers. In ICLR, 2021.
  83. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Re. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS, 2022.
  84. J. Ainslie, J. Ling, Y. Xiang, M. Kovaleva, and S. Kuratov. Few-shot learning with grouped query attention. In ICLR, 2023.
  85. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  86. R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Jin, H. Zhou, and Z. Liu. On layer normalization in the transformer architecture. In ICML, pages 10424–10433, 2020.
  87. B. Zhang and R. Sennrich. Why does pretraining help in nlp? On the role of generalization error. In EMNLP, 2022.
  88. R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In ACL, 2016.
  89. T. Kudo and J. Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In EMNLP, 2018.
  90. V. Dubois, A. Fan, C. Fournier, T. Wolf, and P.-A. Crook. Llama 3.1. Meta AI Report, 2024.
  91. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Nash, M. Dehghani, M. Devlin, M. Chang, X. Lee, and K. Toutanova. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.
  92. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. 2018.
  93. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohammad, R. Levy, V. Stoyanov, and L. Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, 2020.
  94. I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
  95. N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
  96. M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. B. Thoppilan, R. Timalsina, Q. V. Le, and A. Ahmed. Big bird: Transformers for longer sequences. In NeurIPS, 2020.
  97. M. Liu, V. K. Ithapu, K. Ramchandran, and A. G. Dimakis. Infini-transformer: Scaling the context window of large language models with compressive memory. arXiv preprint arXiv:2404.01458, 2024.
  98. N. Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.

All preprints