Anshuman Mishra

Ahead-of-Time Inference Everyone in ML inference is optimizing the same thing, making the response faster *after* a query. What if the answer was already computed *before* anyone asked? LayerScale is a stateful architecture and an highly efficient inference engine that operates by computing deltas between queries instead of processing each query in full. LayerScale treats the KV cache as a programmable, mutable in-VRAM store. We're rethinking WHEN the computation happens. Transformers can stream, the bottleneck isn't attention complexity, the bottleneck was the assumption that context must be reprocessed from scratch! https://layerscale.ai

61

3 Comments

Just published a new post on AI Realist Substack – this time on synthetic datasets. Part 1 is a recap of a great ACL tutorial on synthetic data and how it's used in NLP. Part 2 covers real-world applications in enterprise use cases we see often. Synthetic datasets are not a panacea. They are tricky to get right – they need careful evaluation, can be overused easily, and there's always the risk of model collapse. But when used correctly, they can: – Verbalize non-linguistic data – Add diversity – Speed up human annotations All of which can improve performance. Often, they are the easiest way to boost your system. Read the post and check out the tutorial.

18

6 Comments

A non-ML engineer ran 37 overnight experiments and woke up to a better model. Shopify's CEO used Karpathy's autoresearch framework on an internal model. By morning, a 0.8B-parameter version outperformed his previous 1.6B model by 19%. No ML team. No research sprint. One overnight compute run. The constraint on experimentation has never been intelligence. It has been the cost of running structured tests at scale. When that cost approaches zero and the loop runs autonomously, the teams treating this as a productivity gain will be outrun by the ones treating it as a fundamentally different epistemology.

77

6 Comments

**The Trillion-Dollar Encoder We Keep Throwing Away** Every time you call GPT-4, your document gets encoded into a dense semantic representation — a vector that captures what it actually *means*. Then you get your summary. And that embedding? Discarded. Tomorrow you’ll pay again to re-encode the same document. And throw it away again. We’re renting understanding by the millisecond and walking away from the asset. ----- The *generative* part of GenAI — chat, summaries, drafts — that’s the demo. The *encoder* — the part that learned to represent meaning across text, images, audio, structured data — that’s the revolution. ----- We spent a decade building enterprise data plumbing: ETL, MDM, semantic layers, data products. Hundreds of billions in integration. Unified multimodal embeddings make much of that optional — for analytical workloads. Operational systems (transactions, audit, regulatory) still need structure. But for insight? The integration layer becomes geometry. ----- Channel embeddings ≠ unified embeddings. Embedding calls separately from emails separately from documents? Vectorized silos. Still need MDM. Still need joins. One encoder, one space, all modalities — that’s where “frustrated customer with water damage claim” becomes a queryable region, not a 6-table join. ----- And ontologies? They existed because language was too wild for machines. Embeddings tame it. No committee required. ----- GenAI’s gift wasn’t chatbots. It was representation learning. Stop renting understanding. Start owning it. https://lnkd.in/gPQj-7Uh #tPower #ml #ai #GenAI4FS #inc81starch

36

4 Comments

Andrew Ng is right: vertical AI is where the value is. Andrew Ng said something on last week's 20VC podcast that captures what we've learned building Diffusely: the real value in AI isn't in foundation models - it's in vertical applications that transform specific industries. As Ng put it: "Moats tend to be a function of the industry rather than a function of the technology." This isn't about having better models than OpenAI or Google. It's about industry-specific data, workflows and integrations that horizontal tools can't replicate. CarCutter integrates with DMS systems and understands dealership workflows. AutoRetouch handles brand consistency across thousands of SKUs. Both leverage our 200M image library of professional photography. Horizontal AI tools can't do this - not because their models aren't good enough, but because they're not built for it. Ng emphasizes that AI's value isn't saving 20% on one task. It's redesigning workflows for transformation. One automotive client reduced photography time from 28 to 5 minutes per vehicle. €600k saved annually. Fashion brands achieve 90% faster time-to-market. This is 10x gains - fundamental changes to operations, not incremental improvements. The current narrative is that foundation model companies will dominate. Ng's thesis - and our experience - suggests otherwise. Foundation models are infrastructure. Valuable, but not where economic value gets captured. The value is in vertical applications built on proprietary data that integrate deeply with industry workflows. Vertical AI requires understanding the industry as deeply as you understand AI. Professional services, not just software. Patience to integrate with legacy systems. Most AI companies optimize for horizontal scale because it's faster. Building vertical AI is slower, requires more expertise. 𝐓𝐡𝐚𝐭'𝐬 𝐰𝐡𝐲 𝐢𝐭'𝐬 𝐝𝐞𝐟𝐞𝐧𝐬𝐢𝐛𝐥𝐞. Three years building vertical AI convinced us Ng is right. The companies that win won't just have the best models. They'll have the deepest vertical expertise, specialized data, and strongest integration into industry workflows. #verticalai #ai #moat #entrepreneurship #enterprisesaas

35

1 Comment

For many of you who expressed interest in my Large Language Models from Large Causal Models Arxiv paper, here's a nice video that summarizes the main points of the paper, coupled with a slick narration. https://lnkd.in/gwHBaaHp The video gets a lot of the high-level ideas correct. It focuses on the six main modules, and correctly zooms in on the "magic sauce" of the Geometric Transformer (stay tuned for future papers!). I thought the video, no doubt done with modern AI tools, got the main zeitgeist of the paper: how to build massively large causal models by exploiting the knowledge latent in today's hundred billion parameter LLM. It is not traditional causal inference. It is different. It is new. It is weird and wild, but in a nice way. Of course, what makes it all work, besides the great work that has already gone into building giant LLMs (and kudos to everyone who has done that arduous task, truly one of the most significant advances in AI in the last few decades), is a new way to think of Transformer models and Backpropagation. For Democritus -- the system that converts LLM queries into LCMs -- does not just "work", it works because there is some deep categorical ML underneath the hood. So, how does this work? Well, you will have to be patient because the more detailed papers are in the works, but at a high level, you want a new type of backpropagation method that enforces constraints, and does more than associative learning. To get Democritus to work, you need to coerce deep learning to respect the underlying causal structure. This idea can be generalized so it applies to many domains -- from NLP to puzzle solving. There's a new loss function that is used to construct constrained solutions, and that is part of the reason why Democritus works at all. Otherwise, as the video shows nicely, you get a giant "hairball" with no structure. Happy holidays everyone. In my Category Theory for AGI spring seminar at U.Mass, Amherst, I will go over all the underlying theory that makes systems like Democritus work, and introduce the students to the wonderful world of Geometric Transformers. We will see how the deepest ideas in pure mathematics of the mid-to-late 20th century will help us design the next generation of AI systems.

109

2 Comments

AI governance will remain incomplete until memory is treated as a first-class risk object. Tools like MemoryBench show how governance can move from abstract principles to enforceable controls. The real shift here is not technical, it is cultural: from trusting AI memory to governing it. The question leaders should now be asking is not whether AI can remember, but whether organizations are equipped to govern what AI remembers. #AIGovernance #ResponsibleAI #AIRiskManagement #AIControls #ISO42001 #TrustworthyAI FAIrMind | Institute of Risk Management | Institute of Directors (IOD), India

1