Yang Xu

I’ve been working on a multi-agent framework for SQL generation with RL built on top of AgentLightning. After multiple rounds of hyperparameter tuning using Qwen3-8B as the base model with naive result-regulated GRPO, we consistently observe a reward collapse pattern after a certain number of training steps, which looks very similar to what was reported in "SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning". Interestingly, colleagues have reported very similar behavior when experimenting with Search-R1 using Qwen2.5-7B under naive GRPO settings, which suggests that this phenomenon might not be specific to our setup. Another strange observation during training is the emergence of cross-lingual token mixing inside the <think> traces. For example, reasoning that initially appears purely in English gradually turns into something like: “Okay, the current schema 看上去 not…”. In other words, English and Chinese tokens start blending together more frequently as training progresses. At this point, it’s still unclear what exactly is driving this behavior. One possibility is that the multi-turn environment introduces OOD trajectories as reported in "SimpleTIR". Another possibility is that RL is exposing or amplifying some intrinsic bias or instability in the Qwen series models. Curious if others working on RL for tool-using agents or reasoning models have observed something similar.

13

I just published Vol. 143 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Production-Scale LLM Ranking for Semantic Search, from LinkedIn 📚 The Role of Embedding Magnitude in Contrastive Learning, from NAIST 📚 Bidirectional Diffusion Models for Dense Text Embeddings, from Perplexity AI 📚 Exposing Reproducibility Failures and Conceptual Flaws in Diffusion Recommenders, from Benigni et al. 📚 Learning Sparse High-Dimensional Embeddings for Efficient Collaborative Filtering, from Vančura et al. 📚 LLM-Driven Autonomous Optimization for Industrial-Scale Recommendation Systems, from Google 📚 Compressing User Histories into Learnable Memory for Scalable Generative Recommendation, from Tencent 📚 Scaling Industrial Ranking Models to 15 Billion Parameters with TokenMixer-Large, from ByteDance 📚 A Reasoning-Enhanced LLM Framework for Recommendation Re-ranking, from Meta 📚 Establishing Scaling Laws for Massive-Scale Recommender Systems, from Meta #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

63

2 Comments