Toloka

"If you don't have a repeatable way to judge how a new model might affect your output, everything is a one-off." - Shopify's VP & Head of Engineering Farhan Thawar, in a video interview with VentureBeat on how Shopify evaluates AI in production, and where Toloka fits in. The pattern: a golden dataset, expert-corrected data to cover the gaps, and a fine-tuning loop that turns "is this new model better?" into a question you can answer in days, not months. If your team is trying to figure out what "evals" should actually mean for your product, happy to talk through what that looks like for you. https://lnkd.in/eqkWurX9

Our team is back from a week at ICML in Seoul. Grateful to everyone who stopped by our booth to talk evals, fine-tuning, robotics, RL gyms and more, and to everyone who joined Renaud de la Guéronnière's talk at the GenSR Workshop, where he spoke on human judgment at scale for evaluating generative search and recommendation systems. If you missed us at the conference, and you're building or evaluating models, we can help. Connect with our team to learn more about our RL gyms, benchmarks, OTS datasets, and our custom services.

We added Sonnet 5 to Toloka Arena. Here is what the data shows about the gap between one-shot performance and actual reliability. At first glance, single-attempt performance improved almost across every domain compared to Sonnet 4.6. The only exception was Bank HR, which went backward by -3.6pp. However, when we evaluate pass^5—solving a task 5/5 times—a very different story emerges: - Sonnet 5 proved more consistent than 4.6 in 5 of 7 domains, with 12-18pp gains in Travel and Restaurant Operations. - Conversely, it is less consistent in Logistics and Manufacturing, despite scoring higher on a single try. - Bank HR remains the hardest domain for every model we've tested. It's the one place this model didn't improve at all. Looking at the overall composite score, Sonnet 5 ranks 10th out of 33 models. It sits ahead of Sonnet 4.6, but remains well behind Fable 5, the GPT-5.5 family, and Opus 4.7 and 4.8. What this means for AI development is clear: a single-attempt score and a reliability score can move in opposite directions within the exact same domain. Model developers optimizing purely for one-shot benchmarks may be flying blind when it comes to real-world consistency. Contact our team if you want the full domain-by-domain breakdown.

Anyone at #ICML ? Let’s meet and chat about data for Frontier Physical AI Enterprise Finetuning Toloka team is here the whole week 💪

We're here at #ICML2026! Come find us at Booth 104 to talk RL gyms, private benchmarks, and red-teaming of agents. Explore Toloka Arena, our agentic capabilities leaderboard, and learn more about our off-the-shelf catalog, spanning across STEM, coding, robotics, and more. Olga Megorskaya Sergey Polyashov Alexander Borodetskiy Renaud de la Guéronnière Roman Arkhipov Ty Layton Mimi Zheng Rosmiyana Shekhovtsova Konstantin Chernyshev Ilianna Danengirsa Ksenia Peresvetova Jillian B.

Today’s the day. If you’re at #ICML2026 and thinking about how generative search and recommendation systems should be evaluated, join Renaud de la Guéronnière this afternoon. He'll be speaking at 5:20 PM at the GenSR Workshop at COEX. Starting tomorrow, the rest of the Toloka team will be at Booth 104 for the rest of the week, so stop by to talk evals, benchmarks, RL gyms and more.

What counts as a good search result when the result is generated by a model? That's the question Renaud de la Guéronnière, our VP of Agent Eval R&D, will be talking about at the GenSR Workshop at #ICML2026, hosted by Shopify. The workshop runs from 4:00 PM KST on July 6, covering the shift from multi-stage discriminative pipelines to unified generative foundation models in search and recommendation. Renaud's session, "Ground Truth for the Generative Turn: Human Judgment at Scale for Search and Recommendation," starts at 5:20 PM KST. He'll dive into why click data still has a place, but only gets you so far. When models are shaping the results themselves, evaluation needs to evolve from past behavior. In retail, that means judgment from people who match the user the system is serving, real buyers with real purchase histories, not generic raters. If you're working on search, recommendations, or model evaluation, join the full workshop to hear from Renaud and Alexander Borodetskiy (Toloka), as well as Ruiming Tang (Kuaishou Technology), Xi Liu and Heng Liu (Meta), and Shuying Sun (Shopify).

The Toloka self-service platform just shipped a public pipeline API (beta), which means everything you build in the visual editor can now be operated programmatically. Design your pipeline once, then run it entirely via HTTP: feed data, launch runs, monitor progress, pull results, without ever touching the visual editor again. We also added Agent Plan Mode: instead of building your pipeline, the agent now studies your brief, produces a structured plan, and waits for your approval before touching a single node. Review it, leave comments on anything you want changed, and the agent updates the plan before it builds. A few more things shipped alongside it: self-check to test individual nodes before a full run, pay-as-you-go billing with upfront cost estimates, and now you can pause and resume without losing progress. Test out the latest features at https://platform.toloka.ai. Full breakdown in the comments.

Our team is in Austin this week for the Enterprise AI Maturity & Transformation Assembly, hosted by The Millennium Alliance. It's a room full of enterprise CIOs working through the same question: how do you take AI from promising pilots to something you can actually trust in production? That's exactly where Toloka lives: the data quality, evaluation, and human judgment that decide whether an AI system is ready for the real world. Vaibhav Srivastava and Mimi Zheng are on the ground comparing notes with leaders solving this every day. Two conversations we especially valued: Manoj Tiwary at Subaru Canada Inc and James Lin at Experian. Thanks to The Millennium Alliance team for a well run event. We are already looking forward to the next one!