Spark error – Parquet does not support decimal. See HIVE-6384
I was creating a Hive table in Databricks Notebook from a Parquet file located in Azure Data Lake store by following command:
val df = spark.read.parquet( "abfss://adlsstore@MyStorageAccount.dfs.core.windows.net/x/y/z/*.parquet") df.write.mode("overwrite").saveAsTable("tblOrderDetail")
But I was getting following error:
warning: there was one feature warning; re-run with -feature for details
java.lang.UnsupportedOperationException: Parquet does not support decimal. See HIVE-6384
As per the above error it relates to some Hive version conflict, so I tried checking the Hive version by running below command and found that it is pointing to an old version (0.13.0). This version of Hive metastore did not support the BINARY datatypes for parquet formatted files.
spark.conf.get("spark.sql.hive.metastore.version")
Also as per this Jira Task on HIVE-6384 the support for multiple datatypes was implemented for Parquet SerDe in Hive 1.2.0 version.
So to update the Hive metastore to the current version you just need to add below commands in the configuration of the cluster you are using.
Click on “Clusters” –> click “Edit” on the top –> expand “Advanced Options” –> under “Spark” tab and “Spark Config” box add the below two commands:
spark.sql.hive.metastore.version 1.2.1
spark.sql.hive.metastore.jars builtin
You just need to restart the cluster so that the new settings are in use.
Some similar errors:
– Parquet does not support date
– Parquet does not support timestamp
-
August 6, 2020 at 5:45 pmHIVE-6384 Errors with Spark and Parquet – Curated SQL
- Spark: The Definitive Guide
- Learning Spark 2Ed
SQL Server - Certification exam books:
- TSQL: 70-761 , 70-762
- DBA: 70-764 , 70-765
- DW/BI: 70-767
SQL Tags
M | T | W | T | F | S | S |
---|---|---|---|---|---|---|
1 | 2 | |||||
3 | 4 | 5 | 6 | 7 | 8 | 9 |
10 | 11 | 12 | 13 | 14 | 15 | 16 |
17 | 18 | 19 | 20 | 21 | 22 | 23 |
24 | 25 | 26 | 27 | 28 | 29 | 30 |
31 |
Categories
- Big Data (14)
- Apache Spark (5)
- Databricks (3)
- Hadoop (5)
- DBA Stuff (34)
- Interview Q (8)
- Microsoft Azure (25)
- MS BI (9)
- Analysis Services (1)
- DW and BI (1)
- Integration Services (1)
- Power BI (4)
- Reporting Services (2)
- Tabular Model (1)
- Others (196)
- Blockchain (1)
- Certifications (9)
- Cloud Computing (1)
- Cpp (32)
- Cpp Graphics (21)
- Excel (10)
- Informatica (5)
- Java (56)
- Linux (8)
- Microsoft (14)
- MS.net (2)
- Oracle (1)
- Powershell (3)
- Python (1)
- Reviews (5)
- Security (1)
- Spark SQL (2)
- SQL Server (7)
- SQLwithManoj (11)
- Uncategorized (7)
- VBA Macro (1)
- Visual Studio (1)
- Windows (6)
- SQL Server Conferences (1)
- SQL Server Internals (85)
- Datatypes (5)
- DB Concepts (28)
- Indexes (11)
- JOINS (4)
- ML Python (2)
- SQL DB Engine (8)
- SQL Server Migration (1)
- SQL Server Upgrade (1)
- Stored Procedures (11)
- tempdb (3)
- XML (16)
- SQL Server Questions (41)
- SQL Server Versions (135)
- SQL Azure (4)
- SQL Server 2005 (7)
- SQL Server 2008 (5)
- SQL Server 2012 (33)
- SQL Server 2014 (18)
- SQL Server 2016 (49)
- SQL Server 2017 (20)
- SQL Server 2019 (2)
- SQL Server Express (1)
- T SQL (182)
- Differences (31)
- JSON (6)
- Misconception (7)
- Optimization Performance (13)
- SQL Basics (15)
- SQL Built-in Functions (6)
- SQL Errors (44)
- SQL Tips (67)
- SQL Trivia (9)
Archives
Top Posts
- SQL Error - The server may be running out of resources, or the assembly may not be trusted with PERMISSION_SET = EXTERNAL_ACCESS or UNSAFE
- Getting started with SQL Server 2014 | Download and Install Free (Express) or Full version
- Using IDENTITY function with SELECT statement in SQL Server
- Reading JSON string with Nested array of elements | SQL Server 2016 - Part 3
- SQL DBA - Windows could not start the SQL Server... refer to service-specific error code 17051 - SQL Server Evaluation period has expired
- SQL Server 2016 RTM full and final version available - Download it now !!!
- DACPAC - Could not deploy package, the database platform service with type Sql130DatabaseSchemaProvider is not valid
- SQL Error - SQL Server blocked access to procedure 'dbo.sp_send_dbmail' of component 'Database Mail XPs'
- Querying Excel 2010 from SQL Server in 64-bit environment
- Download & Install SQL Server Reporting Services (SSRS) 2017 (decoupled from SQL Server engine setup)
Blog Stats, since Aug 2010
- 4,450,962 hits
StatCounter …since April 2012

Leisure blog: Creek & Trails
- I got full refund of my flight tickets during COVID lockdown (AirIndia via MakeMyTrip)
- YouTube – Your Google Ads account was cancelled due to no spend
- YouTube latest update on its YPP (YouTube Partner Program) which may affect your channel
- Starting your own blog !!!
- How to file ITR (Income Tax Return) online AY 2017-18 (for simple salaried)
Disclaimer
This is my personal blog site.
The opinions expressed here represent my own and not those of my employer. For accuracy and official reference refer to MS Books On Line and/or MSDN/TechNet.
My employer do not endorse any tools, applications, books, or concepts mentioned on the blog. I have documented my personal experience on this blog.