Can't connect to Dataproc Metastore from Dataproc Batch Jobs

dbhateja · January 19, 2024, 11:30pm

Hi,
While I’m trying to connect to hive-metastore(Dataproc Metastore) using thrift url within Spark Configuration, I’m getting various Metastore exceptions as included below.

Spark config:
`spark = ( SparkSession .builder .appName(“IcebergSparkSrvrlesss”) .config(“spark.sql.catalog.iceberg_catalog”,“org.apache.iceberg.spark.SparkCatalog”) .config(“spark.sql.extensions”,“org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions”) .config(“spark.sql.catalog.iceberg_catalog.type”,“hive”) .config(‘spark.sql.catalog.spark_catalog.uri’, ‘thrift://...:9083’)

.config(“spark.sql.catalog.iceberg_catalog.warehouse”,“gs://usmedp-devstg-icebergpoc/iceberg-catalog”).enableHiveSupport().getOrCreate()

Read the data and create the table df_account_partition.writeTo(f"{iceberg_catalog}.{iceberg_warehouse}.icbg_account_tbl").tableProperty(“format-version”, “2”).createOrReplace()`

Exception:
Query for candidates of org.apache.hadoop.hive.metastore.model.MVersionTable and subclasses resulted in no possible candidates
Required table missing : “VERSION” in Catalog “” Schema “”. DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable “datanucleus.schema.autoCreateTables”
org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table missing : “VERSION” in Catalog “” Schema “”. DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable “datanucleus.schema.autoCreateTables”
at org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:606)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

Traceback (most recent call last):
File “/tmp/srvls-batch-461fc9f5-cf14-4b80-b879-a2f8f369d268/iceberg_hive.py”, line 21, in
spark.sql(“CREATE NAMESPACE IF NOT EXISTS iceberg_catalog.test;”)
File “/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py”, line 723, in sql
File “/usr/lib/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py”, line 1321, in call
File “/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py”, line 111, in deco
File “/usr/lib/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py”, line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o83.sql.
: org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore
org.apache.spark.sql.connector.catalog.SupportsNamespaces.namespaceExists(SupportsNamespaces.java:97)
at org.apache.spark.sql.execution.datasources.v2.CreateNamespaceExec.run(CreateNamespaceExec.scala:43)

… 51 more
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

… 63 more
Caused by: MetaException(message:Version information not found in metastore. )
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:83)
… 68 more
Caused by: MetaException(message:Version information not found in metastore. )

Poala_Tenorio · January 26, 2024, 1:45pm

The exception you’re encountering seems to be related to the Hive Metastore and its version table. The error message suggests that the required table “VERSION” is missing in the specified catalog and schema.

Here are a few suggestions to troubleshoot and resolve the issue:

Ensure that the Hive Metastore schema is properly initialized. The “VERSION” table is a part of the Hive Metastore schema. You can check whether the necessary tables, including “VERSION,” exist in your Hive Metastore schema. Verify the version of Hive Metastore you are using and ensure it is compatible with your Spark and Iceberg versions. Sometimes, compatibility issues can lead to problems like missing tables.

The error suggests enabling “datanucleus.schema.autoCreateTables.” You can configure this property in your Spark configuration to allow DataNucleus to automatically create missing tables.

Add the following configuration to enable auto-creation:

.config(“spark.hadoop.datanucleus.schema.autoCreateAll”, “true”)

Make sure you are using compatible versions of Spark, Hive, and Iceberg. Check for any updates or patches that might address the issue.

Here’s an example of how you can update your Spark configuration with the suggested changes:

spark = (

SparkSession.builder

.appName(“IcebergSparkSrvrlesss”)

.config(“spark.sql.catalog.iceberg_catalog”, “org.apache.iceberg.spark.SparkCatalog”)

.config(“spark.sql.extensions”, “org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions”)

.config(“spark.sql.catalog.iceberg_catalog.type”, “hive”)

.config(‘spark.sql.catalog.spark_catalog.uri’, ‘thrift://...:9083’)

.config(“spark.sql.catalog.iceberg_catalog.warehouse”, “gs://usmedp-devstg-icebergpoc/iceberg-catalog”)

Topic		Replies	Views
Issue with properly setting up a Spark Session to my Apache Iceberg BigLake tables Data Analytics bigquery , dataproc	0	79	June 28, 2024
Issues with dataproc serverless connection to iceberg biglake tables Data Analytics bigquery , dataproc	3	65	December 15, 2024
Dataproc Metastore couldn't start Data Analytics dataproc	5	44	January 18, 2024

Can't connect to Dataproc Metastore from Dataproc Batch Jobs

AI Suggested topics