Databricks-> Snowflake:SQL编译错误:阶段:“ XYZ”不能是管道定义中的临时阶段

我尝试将Databricks的流具体化为snowflake表:

parsedStream   .writeStream     .outputMode(“ append”)     .options(选项)     .option(“ dbtable”,“ test_table”)     .option(“ streaming_stage”,“ test_stage”)     .option(“ checkpointLocation”,“ / demo-checkpoints”)     .format(“雪花”)     .start()

options包含对雪花进行身份验证所需的所有详细信息,并且此部分有效。 我检查了预先创建的阶段和不存在的阶段,以便Databricks可以创建临时阶段(这就是它应该如何工作的方式)。

我收到以下错误: net.snowflake.client.jdbc.snowflakeSQLException:SQL编译错误:Stage:'test_stage'不能是管道定义中的临时阶段。

at net.snowflake.client.jdbc.snowflakeUtil.checkErrorAndThrowExceptionSub(snowflakeUtil.java:135)
at net.snowflake.client.jdbc.snowflakeUtil.checkErrorAndThrowException(snowflakeUtil.java:60)
at net.snowflake.client.core.StmtUtil.pollForOutput(StmtUtil.java:503)
at net.snowflake.client.core.StmtUtil.execute(StmtUtil.java:370)
at net.snowflake.client.core.SFStatement.executeHelper(SFStatement.java:474)
at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:230)
at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:172)
at net.snowflake.client.core.SFStatement.execute(SFStatement.java:663)
at net.snowflake.client.jdbc.snowflakeStatementV1.executeQueryInternal(snowflakeStatementV1.java:161)
at net.snowflake.client.jdbc.snowflakePreparedStatementV1.executeQuery(snowflakePreparedStatementV1.java:153)
at net.snowflake.spark.snowflake.JDBCWrapper$$anonfun$executePreparedQueryInterruptibly$1.apply(snowflakeJDBCWrapper.scala:257)
at net.snowflake.spark.snowflake.JDBCWrapper$$anonfun$executePreparedQueryInterruptibly$1.apply(snowflakeJDBCWrapper.scala:255)
at net.snowflake.spark.snowflake.JDBCWrapper$$anonfun$2.apply(snowflakeJDBCWrapper.scala:292)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

导入net.snowflake.spark.snowflake.Utils.snOWflaKE_SOURCE_NAME res17:org.apache.spark.sql.streaming.StreamingQuery = org.apache.spark.sql.execution.streaming.StreamingQueryWrapper@1f36b6a9

有什么想法吗?

wanderer0 回答:Databricks-> Snowflake:SQL编译错误:阶段:“ XYZ”不能是管道定义中的临时阶段

目前,通常不提供带有Snowflake的

火花流(使用功能 writeStream )。您可能想要重写您的应用程序,以使用当前可用的Spark连接器(不支持结构化流)。

https://docs.snowflake.net/manuals/user-guide/spark-connector.html

本文链接:https://www.f2er.com/3144255.html

大家都在问