How Query Engines Work: An Introductory Guide

SQL is powerful and widely understood but has limitations in the world of so-called “Big Data,” where data scientists often need to mix in custom code with their queries. Platforms and tools such as Apache Hadoop, Apache Hive, and Apache Spark are now widely used to query and manipulate vast data volumes.

A query engine is a piece of software that can execute queries against data to produce answers to questions, such as: • What were my average sales by month so far this year? • What were the five most popular web pages on my site in the past day? • How does web traffic compare month-by-month with the previous year?

Although Rust is a great choice for a high-performance query engine, it is not ideal for teaching the concepts around query engines, so I recently built a new query engine, implemented in Kotlin, as I was writing this book. Kotlin is a very concise language and easy to read, making it possible to include source code examples in this book.