Spark Queries That Take Hours.Now Done in Minutes.
TabbyDB is a drop-in fork of Apache Spark that eliminates compile-time blowup, OutOfMemory failures, and nested join underperformance — at the optimizer level. Zero code changes. Zero cluster changes.
The Real Bottleneck Is in Apache Spark
Production workloads hit walls that benchmarks never reveal. Analyzer and optimizer inefficiencies silently balloon compilation times, pushing complex queries into OOM failures.
Root Cause: Query Planning
Compile time can take minutes to hours — but Spark's UI only registers a query after plan submission. The bottleneck is in planning, not execution, and it's invisible to your metrics.
Why Tuning Fails
Compile-time problems are routinely misdiagnosed as runtime issues. Runtime tuning doesn't reduce planning time. More compute doesn't mean faster planning. The fixes you try don't touch the root cause.
Workarounds Backfire
Disabling optimizer rules can reduce compile time — but at the cost of runtime performance. Every workaround forces a tradeoff: faster planning vs. slower execution. TabbyDB eliminates this tradeoff.
Minutes to hours of delay, wasted compute, missed SLAs — with no clear path to fix it in stock Spark.
TabbyDB — Turbocharged Apache Spark
Fully compatible with Apache Spark 4.1.3. Same APIs. Same clusters. Better engine. Drop in the jars and get back hours.
Intelligent Compile-Time Optimizations
Fundamental improvements to critical optimizer rules: optimized constraint propagation, early project collapsing during analysis, reduced Hive Metastore calls, and targeted rule application to avoid expensive tree traversals. Complex queries that took 8 hours now compile in minutes.
Scalable Query Tree Management
Safely collapses project nodes early in the query lifecycle, preventing unbounded query plan growth and reducing memory pressure during compilation. Faster compilation and dramatically reduced risk of out-of-memory failures on deeply nested workloads.
Advanced Broadcast Hash Join Handling
Dynamic file pruning using Broadcast Hash Join data on non-partitioned columns — the fix Spark never shipped. Extended to enable dynamic file pruning for non-partitioned joins, reducing data scan time. 21% improvement on 1TB TPC-DS using Hive.
Patent-Pending Runtime Optimization
New in 4.1.3-tabbydb.2: a patent-pending runtime optimization technique that cuts query execution time further, on top of TabbyDB's existing runtime enhancements. Included in the 1 TB TPC-DS results on GCP: TabbyDB finishes in 6,253.7s vs 7,884.7s on Hive Parquet and 7,066.7s vs 9,034.4s on Iceberg.
Improved Cache Lookup Efficiency
Enhances how cached in-memory query plans are matched and reused, increasing the likelihood of successful cache hits. Higher cache reuse and lower execution overhead — especially for repeated or structurally similar queries.
Hive Parquet Tables — 21% Faster
On 1TB TPC-DS with non-partitioned Hive Parquet tables, TabbyDB cuts total runtime by 21% (1.49x geo-mean speedup per query), with Broadcast Hash Join keys pruning files on non-partitioned columns. Existing Hive tables get faster as they are — no repartitioning, no data rewrite.
Apache Iceberg — 22% Faster
On 1TB TPC-DS with non-partitioned Iceberg tables, TabbyDB cuts total runtime by 22% (1.40x geo-mean speedup per query), with Broadcast Hash Join keys pruning files at the scan layer. The Iceberg performance layer the ecosystem has been waiting for.
Seamless Spark Compatibility. No Lock-In.
Maintains full compatibility with Apache Spark APIs, features, and tooling while delivering every performance improvement above. Replace Spark jars with TabbyDB jars. To revert: swap back. No code rewrites. No cluster changes. No disabling of optimizer rules.
Performance That Redefines Complex Query Execution
1TB TPC-DS on GCP — 7,884.7s → 6,253.7s total, 1.49x geo-mean speedup per query
1TB TPC-DS on GCP — 9,034.4s → 7,066.7s total, 1.40x geo-mean speedup per query
Complex DataFrame API queries — hours to compile, now minutes
Measured on 1 TB TPC-DS with non-partitioned, date-sorted tables — TabbyDB 4.1.3-tabbydb.2 vs Apache Spark 4.1.3 on identical hardware and data. Standard TPC-DS (partitioned) shows equivalent performance to stock Spark by design.
See the performance difference for yourself. Run the same query on stock Spark and TabbyDB side by side in our live Zeppelin notebooks — two demos, each targeting a different root cause. Note: running the Stock Spark paragraph may take 5–12 minutes.
TPC-DS Benchmark Results
Benchmark setup
- Cluster
- Google Cloud us-west4 — 1 driver + 2 workers, Spark standalone
- Driver
- e2-highmem-4 (4 vCPU, 32 GB)
- Workers
- 2 × n2-custom (14 vCPU, 112 GB), 2 × 375 GB local NVMe SSD each
- Executors
- 4 × (7 cores, 40 GB) — 28 cores, 160 GB total
- Engines
- Apache Spark 4.1.3 vs TabbyDB 4.1.3-tabbydb.2, Java 21
- Data
- TPC-DS SF 1000 (1 TB) Parquet on GCS, non-partitioned, date-sorted
- Workload
- 103 TPC-DS v2.4 queries (q84 excluded), Hive and Iceberg tables
- Correctness
- Row counts match on all 103 queries; full result sets compared
Compare Performance in Real Time
Run the same query on stock Spark vs TabbyDB in our hosted Zeppelin notebooks. No setup. No install. See the difference in seconds.
Run the TPC-DS Benchmark on Your Own Machine
Don't take our word for it. Reproduce the exact benchmark used in our published results — on your own cluster, using both stock Spark 4.1.3 and TabbyDB 4.1.3, and compare the query timings yourself.
What you get
Single-node standalone cluster — one machine acting as both master and worker
Same generated data, both engines — stock Spark 4.1.3 and TabbyDB 4.1.3
Non-partitioned, date-sorted tables — the configuration that exposes TabbyDB's gains
Side-by-side query timings for all TPC-DS queries
Spark SQL Modules Optimized in TabbyDB
Side by side: the modules TabbyDB touches versus the stock Spark pipeline.
28+ Apache Spark JIRA Issues — Resolved
Filed, triaged, and fixed in TabbyDB. Many are still open upstream.
More Than a Faster Engine
Built for teams running complex, production-critical Apache Spark workloads, where query compilation time and optimizer behavior matter as much as execution speed.
Engine-Level Enhancements
Improvements are made at the optimizer and execution engine level — not as application-layer patches. Every workload benefits, without tuning or code changes.
Built for Production Stability
Beyond raw performance, TabbyDB fixes 28+ Spark performance and functionality bugs — self-join inconsistencies, data loss in POJO conversions, broken idempotency in streaming joins.
Compatible With Everything You Have
Full compatibility with existing Spark APIs, tooling, and workflows. No migration. No vendor lock-in. If you ever need to roll back, swap the jars.
No Shortcuts. Ever.
TabbyDB never disables optimizer rules as a workaround. Doing so trades away runtime execution speed — the optimizer can no longer do its job. Every fix is implemented at the root cause, in the analyzer, optimizer, or execution engine, so you get optimal compilation and runtime performance.
Partner With the Team Behind TabbyDB
If you're encountering functional or performance issues in Apache Spark — particularly in the SQL or optimizer layer — we're open to collaborating on solutions tailored to your workload or codebase.
Whether it's diagnosing a bottleneck, validating a fix, or contributing targeted improvements, we're happy to engage.
See the Difference on a Sample Query
Run the same query on stock Spark and TabbyDB side by side in our live Zeppelin notebooks — two demos, each targeting a different root cause. No setup required.
Read the Algorithms
Every optimization in TabbyDB is documented. Read the white paper before you decide. Then run the benchmark.
Co-presented at the Databricks Spark Summit, 2021
Constraint Propagation Rule Optimization
The new algorithm that solves Constraint blow-up from permutational logic in stock Spark.
Capping the Query Plan Size
Collapsing project nodes in the analysis phase to prevent tree size explosion in complex DataFrame API queries.
Broadcast Hash Join Key Pushdown
Dynamic file pruning for non-partitioned columns — the runtime performance fix Spark never shipped.
TPC-DS Benchmark Details
Setup, configuration, and query-by-query results for the 1 TB TPC-DS benchmark on Google Cloud.
Common Subexpression Extraction
Applying expensive optimizer rules only once to complex repeated sub-expressions — avoiding redundant tree traversal.
Live in 15 Minutes. Revert in 30 Seconds.
Download TabbyDB jars
Download the Spark 4.1.3 build
Replace existing Spark jars
Swap them in — no configuration changes
Run your pipelines unchanged
Same APIs. Same code. Faster engine.
Our commitment
Performance guarantee
KwikQuery will never regress you relative to stock Apache Spark. Full stop.
Extended test coverage
Our testing has already uncovered and fixed real Spark bugs that may still be present in the upstream release.
Maximum-speed fixes
Any functional issue — whether from our changes or lurking in stock Spark — is our highest priority to fix.
Three Commands. One Faster Spark.
Drop in the TabbyDB jars and run your existing pipelines. Roll back in seconds.
# Download the complete TabbyDB Spark distribution
$ wget https://github.com/perf-apt/tabbydb-trial-releases/releases/download/licensed-release/tabbydb-4.1.3-tabbydb.2-bin-tabbydb.tgz
$ tar -xzf tabbydb-4.1.3-tabbydb.2-bin-tabbydb.tgz
$ export SPARK_HOME=$PWD/spark-4.1.3-tabbydb.2-bin-tabbydb
# Run your pipelines
$ $SPARK_HOME/bin/spark-submit your_app.pyQuestions Engineers Ask
The honest answers we give when a data platform team is evaluating TabbyDB.
Stop Waiting for Spark to Compile.
Download the trial. Run it on your actual queries. See the difference on your own cluster.
100% Apache Spark API compatible. No code changes. Full rollback in seconds.