我正在运行带有transform() using
子句和python脚本的Hive-on-Tez(Hortonworks HDP-3.1)查询。当我的脚本失败时,我得到的只是java stacktrace:
TaskAttempt 3 failed,info=[Error: Error while running task ( failure ) : attempt_1572037513902_1984_1_00_000006_3:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
...
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:576)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
... 19 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20001]: An error occurred while reading or writing to your custom script. It may have crashed with an error.
at org.apache.hadoop.hive.ql.exec.ScriptOperator.process(ScriptOperator.java:464)
...
... 20 more
Caused by: java.io.IOException: Stream closed
...
... 33 more
我在任何地方都找不到python stacktrace,也没有在Yarn容器日志中找到。在正常的map-reduce流工作中,我总是得到python stacktrace。
我已经设置了tez.am.log.level = DEBUG,并获得了大量的DEBUG信息,但仍然没有python跟踪。
是否可以通过UDF脚本获取HIVE-Tez查询的python堆栈跟踪?
谢谢!