Hao Zhao (赵皓)

I received the M.S. degree (with distinction in research) from the School of Engineering at EPFL 🇨🇭 in February 2024, specializing in Automatic and Systems. Previously, I completed my master thesis at TML lab, EPFL, supervised by Nicolas Flammarion, and will continue my research at TML lab as a research assistant. I am interested in developing effective methods that enable AI systems to adapt efficiently to new environments and tasks.

During my master studies, I was also fortunate to spend time as a semester project student at VITA lab, EPFL, supervised by Alexandre Alahi. I got my bachelor's degree from Zhejiang University 🇨🇳.

I am actively looking for PhD positions starting from 2025 Fall and research assistant positions starting from 2024 Fall. If you think my background is a good fit, please don't hesitate to get in touch!

For pronunciation, my full name /how-jow/, but just my first name would be lot easier :)

Google Scholar  /  Email  /  Twitter  /  Github  /  LinkedIn

[Oct 10, 2024] Our work Is In-Context Learning Sufficient for Instruction Following in LLMs? was accepted to Workshop AFM@NeurIPS'24 🎉. Many thanks to anonymous reviewers for their constructive and helpful reviews!

[Oct 07, 2024] Our work on many-shot ICL is featured by MIT Technology Review China 🎉!

[Oct 07, 2024] A new version of our work Is In-Context Learning Sufficient for Instruction Following in LLMs? is available online! See a Twitter/X thread for summary. Great thanks to Maksym for spreading the word.

[May 02, 2024] One work on data selection for Instruction Fine-Tuning LLMs was accepted to ICML 2024.

[Feb 29, 2024] I successfully defended my master thesis with a perfect grade of 6.0/6.0 and received nominations for EPFL Master Thesis Awards 🎉. Great thanks to my committee: Nicolas Flammarion, Giancarlo Ferrari Trecate, Tao Lin, and my supervisors: Maksym Andriushchenko, Francesco Croce.

[Jan 16, 2024] One work on Parameter-efficient Fine-tuning was accepted to ICLR 2024.

[Apr 25, 2023] One work on Test-time Adaptation was accepted to ICML 2023.

(* denotes equal contribution)

In this work, we show that, while effective, ICL alignment with URIAL (Lin et al., 2024) still underperforms compared to instruction fine-tuning on the established benchmark MT-Bench, especially with more capable base LLMs, such as GPT-4-Base. We then uncover the most relevant elements for successful in-context alignment, finding the crucial role of the decoding parameters. Based on these insights, we show that the approach of URIAL can indeed be improved by adding high-quality, possibly carefully selected via greedy search, demonstrations in context, getting closer to the performance of instruct models. Finally, we provide the first, to our knowledge, systematic comparison of ICL and instruction fine-tuning (IFT) for instruction following in the low data regime, where ICL can be a viable alternative to IFT.

We show that filtering via length heuristics is an effective and efficient approach to select instruction tuning data for alignment. In addition, we propose a lightweight refinement of long instructions which can further improve the abilities of the fine-tuned LLMs. In particular, we obtain the 2nd highest-ranked Llama-2-7B-based model on AlpacaEval 2.0 while training on only 1,000 examples and no extra preference data.

We propose CapaBoost, a simple yet effective strategy that enhances model capacity by leveraging low-rank updates through parallel weight modules in target layers. By applying static random masks to the shared weight matrix, CapaBoost constructs a diverse set of weight matrices, effectively increasing the rank of incremental weights without adding parameters. Notably, our approach can be seamlessly integrated into various existing parameter-efficient fine-tuning methods.

We present TTAB, a test-time adaptation benchmark that encompasses ten state-of-the-art algorithms, a diverse array of distribution shifts, and two evaluation protocols. Through extensive experiments, our benchmark reveals three common pitfalls in prior efforts. Our findings underscore the need for future research in the field to conduct rigorous evaluations on a broader set of models and shifts, and to re-examine the assumptions behind the empirical success of TTA.

[2021.9 - 2024.2] M.S. in Automatic and Systems, EPFL, Switzerland

[2017.9 - 2021.6] B.Eng. in Mechanical Engineering, Zhejiang University, China

Conferences: ICLR'25

Workshops: FITML@NeurIPS'24, ICL@ICML'24, WANT@ICML'24, DMLR@ICLR'24

When I'm not working, I enjoy strength training 🏋, hiking 🏃, and swimming 🏊.

I borrowed this website layout from here!