Hydronitrogen Tech Blog

Hamel Ajay Kothari writes about computers and stuff.



Articles in the optimizer tag

Spark 2.2.0 - Cost Based Optimizer Explained

With the release of Spark 2.2.0, they've touted the initial release of the cost based optimizer. This article explains what is included and how it's likely to affect you.

Overview

The release notes contain the following highlights for the cost based optimizer:

  • SPARK-17075 SPARK-17076 SPARK-19020 SPARK-17077 SPARK-19350: Cardinality estimation for filter, join, aggregate, project and limit/sample …

Continue reading →

Posted in Spark on

In the Code: Spark SQL Query Planning and Execution

If you've dug around into the Spark SQL code, whether it's in catalyst, or the core code, you've probably seen tons of references to Logical and Physical plans. These are the core units of query planning, a common concept in database programming. In this post we're going to go over what these are at a high level and then how they're represented and used in Spark. The query planning code is especially important because it …


Continue reading →

Posted in Spark on


Powered by Pelican, Python, Markdown and tons of other helpful stuff.