我尝试将Databricks的流具体化为snowflake表:
parsedStream .writeStream .outputMode(“ append”) .options(选项) .option(“ dbtable”,“ test_table”) .option(“ streaming_stage”,“ test_stage”) .option(“ checkpointLocation”,“ / demo-checkpoints”) .format(“雪花”) .start()
options包含对雪花进行身份验证所需的所有详细信息,并且此部分有效。 我检查了预先创建的阶段和不存在的阶段,以便Databricks可以创建临时阶段(这就是它应该如何工作的方式)。
我收到以下错误: net.snowflake.client.jdbc.snowflakeSQLException:SQL编译错误:Stage:'test_stage'不能是管道定义中的临时阶段。
at net.snowflake.client.jdbc.snowflakeUtil.checkErrorAndThrowExceptionSub(snowflakeUtil.java:135)
at net.snowflake.client.jdbc.snowflakeUtil.checkErrorAndThrowException(snowflakeUtil.java:60)
at net.snowflake.client.core.StmtUtil.pollForOutput(StmtUtil.java:503)
at net.snowflake.client.core.StmtUtil.execute(StmtUtil.java:370)
at net.snowflake.client.core.SFStatement.executeHelper(SFStatement.java:474)
at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:230)
at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:172)
at net.snowflake.client.core.SFStatement.execute(SFStatement.java:663)
at net.snowflake.client.jdbc.snowflakeStatementV1.executeQueryInternal(snowflakeStatementV1.java:161)
at net.snowflake.client.jdbc.snowflakePreparedStatementV1.executeQuery(snowflakePreparedStatementV1.java:153)
at net.snowflake.spark.snowflake.JDBCWrapper$$anonfun$executePreparedQueryInterruptibly$1.apply(snowflakeJDBCWrapper.scala:257)
at net.snowflake.spark.snowflake.JDBCWrapper$$anonfun$executePreparedQueryInterruptibly$1.apply(snowflakeJDBCWrapper.scala:255)
at net.snowflake.spark.snowflake.JDBCWrapper$$anonfun$2.apply(snowflakeJDBCWrapper.scala:292)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
导入net.snowflake.spark.snowflake.Utils.snOWflaKE_SOURCE_NAME res17:org.apache.spark.sql.streaming.StreamingQuery = org.apache.spark.sql.execution.streaming.StreamingQueryWrapper@1f36b6a9
有什么想法吗?