Hive-Tez UDF-如何获取python stacktrace?

我正在运行带有transform() using子句和python脚本的Hive-on-Tez(Hortonworks HDP-3.1)查询。当我的脚本失败时,我得到的只是java stacktrace:

TaskAttempt 3 failed,info=[Error: Error while running task ( failure ) : attempt_1572037513902_1984_1_00_000006_3:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)

...
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:576)
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
    ... 19 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20001]: An error occurred while reading or writing to your custom script. It may have crashed with an error.
    at org.apache.hadoop.hive.ql.exec.ScriptOperator.process(ScriptOperator.java:464)
...
    ... 20 more
Caused by: java.io.IOException: Stream closed
...
    ... 33 more

我在任何地方都找不到python stacktrace,也没有在Yarn容器日志中找到。在正常的map-reduce流工作中,我总是得到python stacktrace。

我已经设置了tez.am.log.level = DEBUG,并获得了大量的DEBUG信息,但仍然没有python跟踪。

是否可以通过UDF脚本获取HIVE-Tez查询的python堆栈跟踪?

谢谢!

sbziqian 回答:Hive-Tez UDF-如何获取python stacktrace?

暂时没有好的解决方案,如果你有好的解决方案,请发邮件至:iooj@foxmail.com
本文链接:https://www.f2er.com/3138554.html

大家都在问