Huanrui Yang, Univ. of AZ, From Static to Dynamic: A Journey of Accurate and Flexible Low-Precision Quantization

Abstract: Quantization has been the key of the efficient deployment of large AI models. In this talk, I will discuss my recent work in the field of foundation model quantization. I will cover highly accurate post-training quantization technique for foundation models, effective mixed-precision quantization scheme for mixture-of-experts, and dynamically changing the quantization precision on the fly to fit hardware resource constraints. I will end my talk with a vision on the application of quantization, together with other efficiency techniques, in the era of agentic inference of foundation models.