我正在AWS EC2上运行从头开始的集群。我有一个用S3上的数据定义的外部表(已分区)。我可以查询此表,并通过简单的select *语句将结果接收到控制台:
hive> set hive.execution.engine=tez;
hive> select * from external_table where partition_1='1' and partition_2='2';
<correct results returned>
运行需要Tez的查询不会将结果返回到控制台:
hive> set hive.execution.engine=tez;
hive> select count(*) from external_table where partition_1='1' and partition_2='2';
Status: Running (Executing on YARN cluster with App id application_1572972524483_0012)
OK
+------+
| _c0 |
+------+
+------+
No rows selected (8.902 seconds)
但是,如果我在日志和文件系统中进行挖掘,则可以从该查询中找到结果:
(yarn.resourcemanager.log) org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1572972524483_0022 CONTAINERID=container_1572972524483_0022_01_000002 RESOURCE=<memory:1024,vCores:1> QUEUENAME=default
(container_folder/syslog_attempt) [TezChild] |exec.FileSinkOperator|: New Final Path: FS file:/tmp/<REALLY LONG FILE PATH>/000000_0
[root #] cat /tmp/<REALLY LONG FILE PATH>/000000_0
SEQ"org.apache.hadoop.io.BytesWritableorg.apache.hadoop.io.Textl▒ꩇ1som}▒▒j¹▒ 2060
2060是该分区的正确计数。
现在,奇怪的是,如果我在HDFS上插入覆盖目录,我就能从应用程序中获取结果:
hive> set hive.execution.engine=tez;
hive> INSERT OVERWRITE DIRECTORY '/tmp/local_out' select count(*) from external_table where partition_1='1' and partition_2='2';
[root #] hdfs dfs -cat /tmp/local_out/000000_0
2060
但是,尝试插入覆盖本地目录失败:
hive> set hive.execution.engine=tez;
hive> INSERT OVERWRITE LOCAL DIRECTORY '/tmp/local_out' select count(*) from external_table where partition_1='1' and partition_2='2';
[root #] cat /tmp/local_out/000000_0
cat: /tmp/local_out/000000_0: No such file or directory
如果我为该查询提供容器结果文件,则仅是数字,没有类名或特殊字符:
[root #] cat /tmp/<REALLY LONG FILE PATH>/000000_0
2060
我只能找到的异常日志消息来自YARN ResourceManager日志:
(yarn.resourcemanager.log) INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1572972524483_0023 CONTAINERID=container_1572972524483_0023_01_000004 RESOURCE=<memory:1024,vCores:1> QUEUENAME=default
(yarn.resourcemanager.log) WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root IP=nmIP OPERATION=AM Released Container TARGET=Scheduler RESULT=FAILURE DESCRIPTION=Trying to release container not owned by app or with invalid id. PERMISSIONS=Unauthorized access or invalid container APPID=application_1572972524483_0023 CONTAINERID=container_1572972524483_0023_01_000004
基于印象的模糊性,我的猜测是,当将结果写入本地文件系统时会出现字符编码问题(因此,容器结果文件中的特殊字符),但实际上只是猜测,我不知道如何验证/解决该问题。任何帮助将不胜感激!