Spice AI

New in 2.0: GA of the Cluster-Sidecar Deployment The cluster-sidecar architecture combines a Spice multi-node cluster for scale and lightweight sidecars for locality, so agentic workloads get both fast, low-latency query of hot data and fast, distributed query across petabyte-scale data lakes. In this pattern, Single-node Spice instances run co-located with applications, materializing only the working set that application or AI agent requires. When a sidecar needs to reach beyond its local working set or run heavy long-running queries, it delegates to the cluster. The tiers collaborate rather than just stack. The multi-node cluster handles ingestion, replication, shared accelerations, while each sidecar stays light and specific to its agent or dashboard, serving and caching the working set it needs. The sidecar acts as a physically isolated sandbox of working data, while Spice manages refreshing data and delegating queries to the cluster replicas instead of the production database, so analytical queries and agents never contend with the operational workload.

20s -> 13s -> 3s The execution times for the same query in Spice ran with single-node Apache DataFusion, multi-node on DataFusion Ballista, and then with Spice Distributed Accelerations on Ballista. The query identifies which product category grew the most between 2012 and 2015 in an Amazon Reviews dataset, aggregates review counts per product category separately for each year, and joins the two years together to calculate the growth between them. Distributing the query cut execution time by about a third. Adding local acceleration on top cut it by another 75%. Full demo from Spice CTO Phillip LeBlanc here: https://hubs.ly/Q04pP57v0

Luke Kim and Spice AI with another big release! They are becoming cornerstone data infrastructure in this AI Agent era helping some of the most demanding enterprises with distributed query, search, and inference. Think of it this way - most enterprise have data distributed in a myriad of systems, Spice seamlessly brings this all together for Agents to reason over in a highly performant and low-cost manner -- including rapidly changing operational data.

Lenovo 🤝 Spice AI Excited to be on the journey with Linda Yao and the Lenovo AI Solutions team. More info in the 2.0 blog: https://hubs.ly/Q04pGTH40

New in 2.0: Use Spice as an analytics replica for live operational data Point Spice at an operational database and it replicates changes without ETL or adding query load on production. It's incrementally adoptable and composable; start with one data store or table, replicate changes to Spice datasets, and even execute joins across them. These same sandboxed, real-time replicas give agents and applications fast, governed access to production data, without pointing an agent at prod and without feeding it a stale copy.

Registration for the second Apache DataFusion Community Showcase is live! 🗓️ Thursday, July 23rd at 4 pm PST/7 pm EST on Zoom This session will feature two speakers: Jordan Epstein will present accelerating Apache Iceberg writes and table compaction using DataFusion Comet, and Spice CTO Phillip LeBlanc will cover how Spice built a highly available, distributed query engine on Apache DataFusion Ballista, including stateless schedulers, partition-aware routing, and how distribution composes with local acceleration on Vortex. cc Andrew Lamb, InfluxData Sign up here: https://hubs.ly/Q04pmKlx0

Our first edition of the Apache DataFusion community included both geospatial orthodoxy as well as scientific computing and neural network training via SQL. First, Dewey Dunnington spoke about building Apache Sedona DB ' s geospatial capabilities (ST_* type functions) and spatial joins. Then, Alex Merose demonstrated leveraging DataFusion to view multi-dimensional scientific data as relational tables and process them with SQL, inclduing a 🤯 demonstration of neural network training using SQL (hint not a just a call to a UDF). Thanks to Wyatt Wenzel and SpiceAI for co hosting and as always to InfluxData, Evan Kaplan and Paul Dix for sponsoring my own work with DataFusion. https://lnkd.in/gZWFdwXJ

Spice AI's Spice 2.0 release is a pretty amazing example of the type of analytics stack you can build with the open data ecosystem, including Apache DataFusion and Spiral's Vortex. Building something like this previously required the DataBricks strategy: Billions 💰 to build it from scratch or paying handsomely to acquire other companies. I can't wait to see how their customers adopt it.

It was an honour to talk alongside Alex Merose at today's inagural Apache DataFusion showcase hosted by Spice AI! It was fun to reflect on Apache SedonaDB's progress since the last time I gave a talk: last November we had just released SedonaDB 0.2.0 with 88 scalar functions...as of today we support over 175 of them including Geography support for many of them! Link to Alex's benchmarks and my slides in the comments!

Thanks to Cerebral Valley for covering the Spice 2.0 launch, the company's origin story, and where we're headed next. "The vision for Spice has always been to provide that full data platform across all your data - what you used to get from your data team, all in a single box. Naturally, you have to grow beyond single-node to do large-scale distributed compute - the things you would do in systems like Spark. The single-node engine was getting us to about the single-terabyte range, and we started doing projects with larger customers in the five-terabytes-and-beyond range. That’s when we decided to build out the rest of the vision: full multi-node distributed compute." Interview here: https://hubs.ly/Q04pcw9l0