I‑AIMS Institute for Artificial Intelligence
and Mathematical Sciences
/

Graded Transformers: A Symbolic-Geometric Approach to Structured Learning

Tanush Shaska

I-AIMS paper sh-95 · Published in AI in Mathematics and Theoretical Physics (2026) · 2026

We introduce the Graded Transformer framework, embedding algebraic inductive biases via grading transformations on vector spaces. Extending Graded Neural Networks (GNNs), we propose the Linearly Graded Transformer (LGT) and Exponentially Graded Transformer (EGT), which apply parameterized scaling—via fixed or learnable grading tuples, with exponential factors for EGT—to encode hierarchies in attention and representation layers. We establish rigorous guarantees, including universal approximation, reduced sample complexity via VC dimension bounds, Lipschitz continuity, and robustness to perturbations. A graded loss ensures stable optimization, with learnable grades enabling adaptive feature scaling. The framework excels in hierarchical and neuro-symbolic tasks across algebraic geometry, physics, NLP, biological sequence analysis, and emerging domains like graphs and finance, offering efficiency and interpretability over data-driven models. By integrating geometric and algebraic principles, Graded Transformers provide a principled approach to structured deep learning, with potential to reshape modeling of complex systems.

MSC classes: 68T07, 14Q20, 68T05

PDF

References

  1. Tony Shaska, Artificial Neural Networks on Graded Vector Spaces, Contemporary Math., vol. 781, Amer. Math. Soc.,
  2. [Providence], RI, American Math. Soc., 2026.
  3. ———, Graded Neural Networks, International Journal of Data Science in the Mathematical Sciences (2025), available at 2502.17751. submitted.
  4. Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar, Are transformers universal approximators of sequence-to-sequence functions?, arXiv preprint arXiv:1912.10077 (2019).
  5. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention is all you need, arXiv preprint arXiv:1706.03762 (2017).
  6. Tony Shaska, Finsler Metric Clustering in Weighted Projective Spaces, 2025. submitted.
  7. Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun, Transformers in time series: A survey, Proceedings of the thirty-second international joint conference on artificial intelligence, 2023, pp. 6778–6786.
  8. Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov, Transformer dissection: A unified understanding of transformer’s attention via the lens of kernel, Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), 2019, pp. 4344–4353.
  9. Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet, A mathematical perspective on transformers, arXiv preprint arXiv:2312.10794 (2023December).
  10. Elira Shaska, Jorge Mello, Sajad Salami, and Tony Shaska, Rational points and zeta functions of Humbert surfaces with square discriminant, NATO Science for Peace and Security Series - D: Information and Communication Security, vol. 66, IOS Press, 2025.
  11. Vladimir N. Vapnik, Statistical learning theory, Wiley-Interscience, 1998.
  12. Bahaa E. A. Saleh and Malvin Carl Teich, Fundamentals of photonics, 2nd ed., Wiley, 2007.
  13. Diederik P. Kingma and Jimmy Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014December).
  14. Hyunjik Kim, George Papamakarios, and Andriy Mnih, The lipschitz constant of self-attention, Proceedings of the 38th international conference on machine learning, ICML 2021, 18-24 july 2021, virtual event, 2021, pp. 5562–5571.
  15. Yurii Nesterov, Lectures on convex optimization, 2nd ed., Springer, 2018.
  16. Adrian Clingher, Andreas Malmendier, and Tony Shaska, Kummer Surfaces, Isogenies and Theta Functions, NATO Science for Peace and Security Series - D: Information and Communication Security, vol. 66, IOS Press, 2025.
  17. Elira Shaska and Tanush Shaska, Machine learning for moduli space of genus two curves and an application to isogeny-based cryptography, J. Algebraic Combin. 61 (2025), no. 2, Paper No. 23, 35.
  18. Parv Kapoor, Logically constrained robotics transformers for enhanced perception-action planning, 2024.
  19. Songming Liu et al., Rdt-1b: a diffusion foundation model for bimanual manipulation, 2024.
  20. Istiaq Ahmed Fahad, Automatic vehicle detection using detr: A transformer-based approach for navigating treacherous roads, 2025.
  21. Xichang Liu, Helai Huang, Jiang Bian, Rui Zhou, Zhiyuan Wei, and Hanchu Zhou, Generating intersection pre-crash trajectories for autonomous driving safety evaluation using transformer time-series generative adversarial networks, Engineering Applications of Artificial Intelligence (2025).
  22. Chen Wang, Imperative learning: A self-supervised neuro-symbolic learning framework for robot autonomy, The International Journal of Robotics Research (IJRR) (2025).
  23. Sunny Katyara, Toward general physical intelligence for resilient agile manufacturing automation, 2025.
  24. Kamal Acharya, Integrating neurosymbolic ai in advanced air mobility: A comprehensive survey, 2025.

All preprints