I‑AIMS Institute for Artificial Intelligence
and Mathematical Sciences
/

Internalizing Tools as Morphisms in Graded Transformers

Tanush Shaska

I-AIMS preprint sh-111 · 2025

We introduce a graded formulation of internal symbolic computation for transformers. The hidden space is endowed with a grading V = ⨁_(g ∈ G) V_g, and symbolic operations are realized as typed block maps (morphisms) ϕ_(h ← g): V_g → Vₕ that are activated selectively by a differentiable routing policy. A self-supervised graded utility functional—defined as the loss reduction induced by a candidate morphism—governs activation and yields sparse, interpretable behavior. We develop the algebraic and geometric foundations: an internal model category whose objects are homogeneous components and whose morphisms are admissible grade transitions; adjoint pairs encoding typed round trips; and information-geometric interpretations in terms of KL gain, mirror descent with Bregman divergences, and Fisher natural gradients. Methodologically, we specify a utility-aware routing mechanism and objective that remain fully end-to-end differentiable. Analytic case studies and lightweight sanity checks illustrate selective morphic activation on hybrid symbolic–linguistic tasks. The framework unifies symbolic computation, geometry, and self-supervised learning within the graded transformer formalism, while subsuming prior external-tool paradigms (e.g., Toolformer) as a special case via functorial internalization.

MSC classes: 18C10, 68T07, 62F12, 68Q32, 18D05

References

  1. T. Shaska, Artificial neural networks on graded vector spaces, Contemporary Mathematics, vol. 835, American Mathematical Society, 2026.
  2. S.-i. Amari, Information geometry and its applications, Applied Mathematical Sciences, vol. 194, Springer Japan, 2016.
  3. S.-i. Amari and H. Nagaoka, Methods of information geometry, Translations of Mathematical Monographs, vol. 191, American Mathematical Society, 2000.
  4. F. Bach, R. Jenatton, J. Mairal, and G. Obozinski, Optimization with sparsity-inducing penalties, Foundations and Trends in Machine Learning 4 (2012), no. 1, 1–106.
  5. P. Billingsley, Convergence of probability measures, 2nd ed., John Wiley & Sons, Inc., New York, 1999.
  6. S. Lang, Real and functional analysis, 3rd ed., Graduate Texts in Mathematics, vol. 142, Springer, New York, NY, 1993.
  7. F. Liang, Y. Kim, and Q. Sun, Extended fiducial inference: Accelerated sampling and inference for modern statistical models, 2024.
  8. S. MacLane, Categories for the working mathematician, 2nd ed., Graduate Texts in Mathematics, vol. 5, Springer, New York, NY, 1998.
  9. S. Mohamed, M. Rosca, M. Figurnov, and A. Mnih, Monte carlo gradient estimation in machine learning, Journal of Machine Learning Research 21 (2020), no. 132, 1–50. arXiv:1906.10652 [stat.ML].
  10. Y. Nesterov, Introductory lectures on convex optimization: A basic course, Applied Optimization, vol. 87, Springer, Boston, MA, 2004.
  11. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, Toolformer: Language models can teach themselves to use tools, 2023.
  12. T. Shaska, Graded transformers: A symbolic-geometric approach to structured learning, Preprint (2025), available at 2507.20108.
  13. ———, Finsler metric clustering in weighted projective spaces, Advances in Pure and Applied Mathematics to appear (2026), available at 2507.00001.
  14. ———, Graded neural networks, International Journal of Data Science in the Mathematical Sciences (2026), available at 2502.17751.
  15. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 5th international conference on learning representations, ICLR 2017, toulon, france, april 24-26, 2017, conference track proceedings, 2017.
  16. M. J. Wainwright, High-dimensional statistics: A non-asymptotic viewpoint, Cambridge Series in Statistical and Probabilistic Mathematics, Cambridge University Press, 2019.

All preprints