Unsloth AI

Gemma 4 is now faster and much more accurate! 🚀 Google DeepMind made huge improvements to tool-calling and chat accuracy, reliability + speed. To get fixes, re-download our updated GGUF, MLX, NVFP4 quants! Unsloth quants: https://lnkd.in/dwSt3VZv Google Gemma 4 Guide: https://lnkd.in/deygCJdB

We collaborated with Amazon Web Services (AWS) on a complete guide to LLM Quantization and Deployment. 🔥 Learn about: • Model formats + Unsloth AI dynamic quants + making your own • Choosing GGUF, NVFP4 or FP8 • The right tools to deploy Amazon SageMaker AI • Benchmark quality, latency, cost & more Read: https://lnkd.in/gCUucUcc GitHub: https://lnkd.in/gyaDBTxK Thank you Dylan S., Michaelangelo B. and Zoish Pithawala for the collab!

DeepSeek-V4 can now run locally with Unsloth AI GGUFs! 🐳 Run lossless DeepSeek-V4-Flash on 168GB RAM. 3-bit works on 110GB Mac, RAM, VRAM setups. We also improved the chat template. Run with Thinking toggles via Unsloth Studio. Guide: https://lnkd.in/gUXeNX6f GGUF: https://lnkd.in/guZjA66M We fixed DeepSeek-V4 issues in llama.cpp that caused gibberish after the 2nd turn. The cause was broken prompt caching. To run correctly, you must use PR #25402. We also improved the DeepSeek-V4 chat jinja template, and tested over 4000 conversations to be equivalent with the official baseline.

What’s your go-to local model right now?

1-bit GLM-5.2 GGUF vs. Claude 4.8 Opus vs. GPT-5.5 We gave 3 models the same prompt and compared one-shot outputs. 1-bit GLM-5.2 GGUF ran locally on a Mac Studio M3 Ultra with 256GB RAM at ~21.6 tok/s. Which output do you like best? GGUF: https://lnkd.in/gGTazQW5 Guide: https://lnkd.in/grTPKWeY

You can now run Kimi K2.7 Code locally! 🌘 We shrank the 1T model to 325GB (-48%) via Dynamic 2-bit where important layers are upcasted. Run at >40 tok/s on 330GB RAM/VRAM setups. Run full precision on 610 GB. Guide: https://lnkd.in/gWURBXEn GGUF: https://lnkd.in/gHxy9x8u

MiniMax M3 can now be run locally! 🔥 The new 428B (23B active) open model has 1M context and performs on par with Gemini 3.1 Pro. Run the Dynamic 2-bit GGUF on 138GB RAM/VRAM or 3-bit on 165GB. GGUFs on Hugging Face: https://lnkd.in/gfA3Z36r Guide: https://lnkd.in/ggkJkXat The MiniMax GGUFs and implementation are experimental.

2-bit Google Gemma 4 12B GGUF, only 4.66 GB on disk, managed to cite 15 sites from a single prompt. 🔥 Try this locally on >6GB RAM via Unsloth Studio GitHub: https://lnkd.in/dcqhW9Vv

You can now train 120B+ parameter models locally on a laptop! 🔥 We collabed with NVIDIA and Microsoft to bring LLM training on the 128GB unified memory RTX Spark laptop! GitHub: https://lnkd.in/dcqhW9Vv