Coiled

Excited to see another trillion rows of data processed. GizmoData crushed the Coiled Trillion Row Challenge with a 1,000-worker cluster on Azure. Read more about Coiled's One Trillion Row Challenge and how GizmoData completed the run in under 5 seconds. One Trillion Row Challenge: https://lnkd.in/gDqApvmq GizmoData Blog: https://lnkd.in/egweh6pZ

Here's what's new from Coiled 👇 📁 Filestores: Persistent cloud storage for Coiled Batch 🔒 Private notebooks 🎯 Sidecar containers for Coiled Batch 📊 Run MLflow on Coiled with sidecars and filestores 📈 How Nelson Griffiths uses Coiled to scale international equity trading simulations at Double River. Their stack: Polars for data processing, Prefect for orchestration, Snowflake for storage, and Coiled for scalable compute. 🎤 Events: We'll be at STAC Summit in NYC on October 28 and QuantMinds International in London from November 18-20. Stop by and say hi! 👋

Data scientist Matthew Rocklin argues Kubernetes isn't the best way to run large-scale Python workloads in the cloud. His company, Coiled, uses VMs instead. By David Cassel

Double River is a quantitative hedge fund that uses Coiled to scale their international equity trading simulations. With Coiled, simulations that previously took days now complete in hours and the engineering team reclaimed valuable time previously spent on infrastructure management. Their stack includes: - Polars for data processing - Prefect for workflow orchestration - Snowflake for scalable data storage - Coiled for scalable compute "We can always use the fastest machines. We don't have to worry about infrastructure... We can make sure our time's focused on doing the things that add value and make money." - Nelson Griffiths, Head of Engineering Read the full case study: https://lnkd.in/gEHsfTPK

Join us in Boston for a casual happy hour! 🍻 The Coiled team will be in town and we’re hosting a casual happy hour. Come join us for food, drinks, and great conversation with others in the Python + data community. 📅 Tuesday, Sept 16th, 6-9pm 📍Night Shift Brewing, Inc. - Lovejoy Wharf 🔗 RSVP: https://lu.ma/bveuradz Hope to see you there!

Here's what's new from Coiled 👇 🐳 Lightweight container support `coiled run --container` now works with minimal containers like astral/uv:debian-slim that don't include Dask/Distributed. ⚡ Auto-timeout clusters + custom disk configs Set automatic cluster shutdowns with `cluster_timeout`, and launch with high-performance EBS volumes for demanding workloads. 🍃 Marimo on Coiled with uv Run interactive marimo notebooks on cloud VMs with `coiled run --interactive`. Sync local files, scale to GPUs or larger machines. 🥑 Learn how Guac uses Coiled to scale demand forecasting and reduce food waste with end-to-end ETL + ML orchestration using Dagster Labs, PyTorch, and Dask. 📦 Package Sync improvements Smarter dependency handling that deprioritizes indirect dependencies, reducing conda build issues.

Easily run parallel tasks on the cloud, even if your code doesn’t use Dask. In this demo, we: - Spin up 100 VMs - Reproject 3,000 satellite images - Finish in 5 minutes for less than $1 Coiled Batch is ideal for embarrassingly parallel jobs, and is especially useful when you want a fast, low-friction way to run Python (or any other language) on multiple machines. But Batch isn’t just for satellite imagery. Teams use it for: - Multi-node GPU LLM fine-tuning - Stress-testing S3 concurrency - Running an arbitrary bash script 100 times - … or any other embarrassingly parallel job With Batch, you can: - Map over inputs with `--map-over-file` - Run on any hardware (CPUs, GPUs, ARM) in any region - Handle dependencies with package sync or Docker - Monitor jobs with hardware metrics and logs No Kubernetes YAML. No Terraform scripts. Check out this short walkthrough from Alexandre Chabot-Leclerc to learn more.

Happy to be part of the stack making geospatial workloads easier (and faster) to run in the cloud.

Here’s what’s new from Coiled 👇 ⚙️ uv support Use uv, Astral's fast Rust-based Python package manager, to define dependencies right in your script and run it with Coiled Batch. 📦 Coiled Batch upgrades: easier parallelism + task coordination `--map-over-values` makes it easy to parallelize jobs over inputs. Built-in task metadata like task ID, cluster ID, scheduler IP addresses are great for tracking with tools like MLflow or W&B. 🚀 Distributed training with Hugging Face Accelerate Train large PyTorch models across multiple GPU-enabled VMs with just a few lines of config. Our new example shows how to scale fine-tuning in the cloud. 🪨 Learn how KoBold Metals uses Coiled to process massive geospatial datasets for battery metal discovery. ☁️ Coiled vs AWS Lambda for long-running jobs Lambda’s great, until your job runs longer than 15 minutes or needs a GPU. We wrote about how Coiled compares to Lambda, Fargate, and Batch.

The Python packaging world is… vast. One of the most delightful recent developments is uv, a blazing-fast package and environment manager from Astral. It’s fast. It’s simple. And when you pair it with Coiled, running Python scripts on the cloud becomes just as seamless. With uv + Coiled, you can: - Declare script-specific dependencies directly in your Python file (`uv add --script`) - Specify runtime config (container, region, hardware) with inline # COILED comments `uvx coiled batch run uv run process.py` Prefer a CLI-only approach? You can also do all this in a single command from your terminal: uvx coiled batch run \ --region us-east-2 \ --container ghcr.io/astral-sh/uv:debian-slim \ uv run \ --with "pandas pyarrow s3fs" \ process.py Compare that to something like AWS Lambda or AWS Batch, where you’d typically need to: - Package your script and dependencies into a ZIP file or build a Docker image - Configure IAM roles, triggers, and permissions - Handle versioning, logging, or hardware constraints With Coiled, there’s: - No YAML jungle - No clicking around in the AWS console - No K8s Just Python on the cloud without the overhead. Check out this demo from James Bourbeau to learn more.