Hive is probably the most popular SQL engine for Hadoop but not because it is fast nor does it support SQL used by most organizations. It is just the oldest. Or is it?
Pivotal HDB, powered by Apache HAWQ (incubating), is much faster and has much better SQL language support. Pivotal HDB gets its roots from Greenplum database which has been around longer so the SQL language support is much more mature.
Performance
In all of the performance tests I’ve done comparing Pivotal HDB with Apache Hive, the results are dramatic. Pivotal HDB loads data significantly faster and executes queries significantly faster too. And yes, I’ve tested it with Tez and even with LLAP. Hive doesn’t come close to the performance of Pivotal HDB. More will be published on this subject soon.
SQL Language Support
The real reason for this post is the lack of SQL language support that Hive has. This makes it more difficult to write SQL in this environment and it makes it more difficult to migrate from a legacy RDBMS to Hadoop. Here are some examples.
Example 1: Subqueries in the WHERE Clause
This is query 6 from the TPC-DS benchmark:
select a.ca_state state, count(*) cnt from customer_address a ,customer c ,store_sales s ,date_dim d ,item i where a.ca_address_sk = c.c_current_addr_sk and c.c_customer_sk = s.ss_customer_sk and s.ss_sold_date_sk = d.d_date_sk and s.ss_item_sk = i.i_item_sk and d.d_month_seq = (select distinct (d_month_seq) from date_dim where d_year = 1998 and d_moy = 2 ) and i.i_current_price > 1.2 * (select avg(j.i_current_price) from item j where j.i_category = i.i_category) group by a.ca_state having count(*) >= 10 order by cnt limit 100;
This query won’t work in Hive because of the two subqueries in the WHERE clause. You will have to rewrite this query like this:
select a.ca_state state, count(*) cnt from customer_address a ,customer c ,store_sales s ,date_dim d ,item i , (select distinct (d_month_seq) as d_month_seq from date_dim where d_year = 1998 and d_moy = 2 ) as sq1 , (select j.i_category, avg(j.i_current_price) as avg_i_current_price from item j group by j.i_category) as sq2 where a.ca_address_sk = c.c_current_addr_sk and c.c_customer_sk = s.ss_customer_sk and s.ss_sold_date_sk = d.d_date_sk and s.ss_item_sk = i.i_item_sk and d.d_month_seq = sq1.d_month_seq and sq2.i_category = i.i_category and i.i_current_price > 1.2 * sq2.avg_i_current_price group by a.ca_state having count(*) >= 10 order by cnt limit 100;
It took me a few minutes to rewrite this and I’m still not 100% sure it is right. In a conversion effort, I would have to execute it in the legacy RDBMS and see if the results are the same as in Hive. If the legacy RDBMS is changing, then I would have a very difficult time with just a single query.
If I was doing this for benchmarking purposes, I would have to execute TPC-DS in another database that does support this query with the same data to validate the results. Clearly, the lack of SQL language support is a problem.
Example 2: Subqueries in the SELECT Clause
This is query 9 from the TPC-DS benchmark:
select case when (select count(*) from store_sales where ss_quantity between 1 and 20) > 31003 then (select avg(ss_ext_list_price) from store_sales where ss_quantity between 1 and 20) else (select avg(ss_net_paid_inc_tax) from store_sales where ss_quantity between 1 and 20) end bucket1 , case when (select count(*) from store_sales where ss_quantity between 21 and 40) > 24212 then (select avg(ss_ext_list_price) from store_sales where ss_quantity between 21 and 40) else (select avg(ss_net_paid_inc_tax) from store_sales where ss_quantity between 21 and 40) end bucket2, case when (select count(*) from store_sales where ss_quantity between 41 and 60) > 28398 then (select avg(ss_ext_list_price) from store_sales where ss_quantity between 41 and 60) else (select avg(ss_net_paid_inc_tax) from store_sales where ss_quantity between 41 and 60) end bucket3, case when (select count(*) from store_sales where ss_quantity between 61 and 80) > 21646 then (select avg(ss_ext_list_price) from store_sales where ss_quantity between 61 and 80) else (select avg(ss_net_paid_inc_tax) from store_sales where ss_quantity between 61 and 80) end bucket4, case when (select count(*) from store_sales where ss_quantity between 81 and 100) > 4078 then (select avg(ss_ext_list_price) from store_sales where ss_quantity between 81 and 100) else (select avg(ss_net_paid_inc_tax) from store_sales where ss_quantity between 81 and 100) end bucket5 from reason where r_reason_sk = 1
You get this error:
FAILED: ParseException line 2:18 cannot recognize input near '(' 'select' 'count' in expression specification (state=42000,code=40000)
The fix is to move all of those subqueries from the SELECT to the FROM section of the query. Or you can write an elaborate CASE statement. Again, this takes time to rewrite and even more time to validate.
Example 3: Correlated Subqueries
This is query 10 from the TPC-DS benchmark:
select cd_gender, cd_marital_status, cd_education_status, count(*) cnt1, cd_purchase_estimate, count(*) cnt2, cd_credit_rating, count(*) cnt3, cd_dep_count, count(*) cnt4, cd_dep_employed_count, count(*) cnt5, cd_dep_college_count, count(*) cnt6 from customer c,customer_address ca,customer_demographics where c.c_current_addr_sk = ca.ca_address_sk and ca_county in ('Clinton County','Platte County','Franklin County','Louisa County','Harmon County') and cd_demo_sk = c.c_current_cdemo_sk and exists (select * from store_sales,date_dim where c.c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk and d_year = 2002 and d_moy between 3 and 3+3) and (exists (select * from web_sales,date_dim where c.c_customer_sk = ws_bill_customer_sk and ws_sold_date_sk = d_date_sk and d_year = 2002 and d_moy between 3 ANd 3+3) or exists (select * from catalog_sales,date_dim where c.c_customer_sk = cs_ship_customer_sk and cs_sold_date_sk = d_date_sk and d_year = 2002 and d_moy between 3 and 3+3)) group by cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating, cd_dep_count, cd_dep_employed_count, cd_dep_college_count order by cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating, cd_dep_count, cd_dep_employed_count, cd_dep_college_count limit 100;
This query correlates data between customer and the tables in each subquery. Rewriting this can be tricky because you need to make sure you don’t duplicate data when joining to the subqueries.
Example 4: INTERSECT
INTERSECT is a feature used in SQL that is similar to UNION or EXCEPT. It gets a distinct list of values from two queries that are contained in both queries. More information on this feature: https://www.postgresql.org/docs/8.2/static/sql-select.html#SQL-INTERSECT
This is query 8 from the TPC-DS benchmark:
select s_store_name ,sum(ss_net_profit) from store_sales ,date_dim ,store, (select ca_zip from ( (SELECT substr(ca_zip,1,5) ca_zip FROM customer_address WHERE substr(ca_zip,1,5) IN ( '89436','30868','65085','22977','83927','77557', '58429','40697','80614','10502','32779', '91137','61265','98294','17921','18427', '21203','59362','87291','84093','21505', '17184','10866','67898','25797','28055', '18377','80332','74535','21757','29742', '90885','29898','17819','40811','25990', '47513','89531','91068','10391','18846', '99223','82637','41368','83658','86199', '81625','26696','89338','88425','32200', '81427','19053','77471','36610','99823', '43276','41249','48584','83550','82276', '18842','78890','14090','38123','40936', '34425','19850','43286','80072','79188', '54191','11395','50497','84861','90733', '21068','57666','37119','25004','57835', '70067','62878','95806','19303','18840', '19124','29785','16737','16022','49613', '89977','68310','60069','98360','48649', '39050','41793','25002','27413','39736', '47208','16515','94808','57648','15009', '80015','42961','63982','21744','71853', '81087','67468','34175','64008','20261', '11201','51799','48043','45645','61163', '48375','36447','57042','21218','41100', '89951','22745','35851','83326','61125', '78298','80752','49858','52940','96976', '63792','11376','53582','18717','90226', '50530','94203','99447','27670','96577', '57856','56372','16165','23427','54561', '28806','44439','22926','30123','61451', '92397','56979','92309','70873','13355', '21801','46346','37562','56458','28286', '47306','99555','69399','26234','47546', '49661','88601','35943','39936','25632', '24611','44166','56648','30379','59785', '11110','14329','93815','52226','71381', '13842','25612','63294','14664','21077', '82626','18799','60915','81020','56447', '76619','11433','13414','42548','92713', '70467','30884','47484','16072','38936', '13036','88376','45539','35901','19506', '65690','73957','71850','49231','14276', '20005','18384','76615','11635','38177', '55607','41369','95447','58581','58149', '91946','33790','76232','75692','95464', '22246','51061','56692','53121','77209', '15482','10688','14868','45907','73520', '72666','25734','17959','24677','66446', '94627','53535','15560','41967','69297', '11929','59403','33283','52232','57350', '43933','40921','36635','10827','71286', '19736','80619','25251','95042','15526', '36496','55854','49124','81980','35375', '49157','63512','28944','14946','36503', '54010','18767','23969','43905','66979', '33113','21286','58471','59080','13395', '79144','70373','67031','38360','26705', '50906','52406','26066','73146','15884', '31897','30045','61068','45550','92454', '13376','14354','19770','22928','97790', '50723','46081','30202','14410','20223', '88500','67298','13261','14172','81410', '93578','83583','46047','94167','82564', '21156','15799','86709','37931','74703', '83103','23054','70470','72008','49247', '91911','69998','20961','70070','63197', '54853','88191','91830','49521','19454', '81450','89091','62378','25683','61869', '51744','36580','85778','36871','48121', '28810','83712','45486','67393','26935', '42393','20132','55349','86057','21309', '80218','10094','11357','48819','39734', '40758','30432','21204','29467','30214', '61024','55307','74621','11622','68908', '33032','52868','99194','99900','84936', '69036','99149','45013','32895','59004', '32322','14933','32936','33562','72550', '27385','58049','58200','16808','21360', '32961','18586','79307','15492')) intersect (select ca_zip from (SELECT substr(ca_zip,1,5) ca_zip,count(*) cnt FROM customer_address, customer WHERE ca_address_sk = c_current_addr_sk and c_preferred_cust_flag='Y' group by ca_zip having count(*) > 10)A1))A2) V1 where ss_store_sk = s_store_sk and ss_sold_date_sk = d_date_sk and d_qoy = 1 and d_year = 2002 and (substr(s_zip,1,2) = substr(V1.ca_zip,1,2)) group by s_store_name order by s_store_name limit 100;
The fix for this is to change the INTERSECT to UNION and make this another subquery that also does a GROUP BY so you have a distinct list of zip codes.
Summary
Of the 99 TPC-DS queries, 19 contains SQL that is not supported by Hive. Pivotal HDB can execute all 99 TPC-DS queries without modification. So not only it is much faster, it is easier to migrate from a legacy RDBMS to Hadoop and easier to start using because the SQL language support is so robust.