Research

I am interested in building scalable and efficient sequence modeling architectures as an alternative to standard Transformers. On the architecture side, I have joined the development of linear attention and recurrent models that achieve Transformer-level quality at a fraction of the cost:

• Linear attention with expressive recurrent states (I joined RWKV and GSA)
• Matmul-free and neuromorphic language modeling (Matmul-free LM)

What I care about most is touching scaling with my own hands. My personal scaling trajectory covers three orders of magnitude in compute:

• 1×10¹⁹ FLOPs: SpikeGPT (216M params, 10B tokens, 2023)
• 2×10²¹ FLOPs: Matmul-free LM (2.7B params, 100B tokens, 2024)
• 4.6×10²³ FLOPs: Ouro (2.6B×4, 7.7T tokens, 2025). Ouro is a looped language model where the same 2.6B parameters are reused across multiple passes, enabling a model trained from scratch to reach the quality of an 8B-scale industrial model at a fraction of the parameter count. I led the entire process, from infrastructure and pre-training to mid-training and post-training.

For each of these runs, I watched every checkpoint from the very first to the last, witnessing a model go from random to intelligence. That is what I am really enjoying. The journey is the reward.

Please refer to publications for the full list.

Talks & Media

• Scaling Latent Reasoning via Looped Language Models, ASAP Seminar #48
• LLMs Don't Need More Parameters. They Need Loops., YouTube
• 利用Loop语言模型扩展Latent Reasoning, FAI × UCSC（Bilibili）

Experience

• Jul. 2023 – Present: Ph.D. Student, UC Santa Cruz
• Sep. 2019 – Jul. 2023: B.Management, University of Electronic Science and Technology of China

This website is adapted from Tianyu Gao's design, which is in turn adapted from Gregory Gunderson.