Matt Stephenson, PhD

AI capability timelines matter a lot - especially in the whistleblowing context. Shorter timelines likely mean - more information asymmetry between lab insiders vs public/ regulators (less time for effective regulation + regulator capacity build up; more resources required to keep track of rising complexity + change; possibly incentives for more internal deployment instead of publicly (scrutinizable) releases) - more safety risks as harmful capability uplift occurs more quickly, race dynamics are more intense leading to safety skipping. An example: Very concerning might be risks that don’t require wide diffusion of new capabilities. Think: lone-wolf actors surrounding bio threats or loss of control scenarios. This combination makes concern disclosure by insiders (internally in company, to regulators, or the public) more important AND more challenging. We at The AI Whistleblower Initiative therefore try to keep an eye on overall timeline predictions and indicators pointing to step changes - but coming to decent confidence assessments is not easy. One of the most popular (and we still think best) benchmarks for capability acceleration is METR‘s time horizon benchmark. There’s been some discussion in the past days around reliability and what it actually measures: Read one of the main authors notes if timelines matter a lot in your models (I’d be surprised if they didn’t). (Apologies for formatting or potential typos - Friday morning thoughts written on monile)

9

An important question I get asked is, why is RL important and is supervised finetuning enough for LLMs? I spent the weekend putting together a substack and a youtube video explaining the difference between RL and SFT, and the theoretical foundations for why RL works in challenging domains. I highly recommend this if you are working with LLMs or just curious to understand how they are trained. You can also find the substack version here: https://lnkd.in/eKewpccT https://lnkd.in/eNjYbWzh

7

2 Comments

This looks really cool. Using agents to evaluate and optimize DECISIONS, and not just code. Being in a highly regulated industry, I'm noticing the built-in audit trail for how those decisions were arrived at as an important feature. The PyMC Labs team continues to put out great work.

4

When generating data with Markov Chains, two practical issues can get overlooked: 1. How long until the chain’s state distribution converges to the stationary distribution? 2. How sensitive are runs to the choice of random seed or initial state? In this piece, I explore convergence empirically (with multi-seed, multi–initial state simulations) and analytically (via spectral gap, relaxation rate, and Jensen–Shannon Distance). The results line up nicely: a predicted number of steps were enough for a model to “forget” its starting conditions, reducing both seed variance and initial-state bias. 👉 Full breakdown with code: Markov Chain Convergence in Python #Mathematics #Python

2

2 Comments