在Oozie工作流程中运行Pig操作时,与HiveMetaStoreClient的连接丢失

我正在Azure Hortonworks cloudbreak上的Oozie工作流中运行Pig操作,但是Pig操作以以下错误结束,因此我尝试将yarn.timeline-service.version从2.0f更改为1.5f,但仍然低于错误,


2019-12-06 13:54:21,074 [main] WARN  org.apache.pig.PigServer  - Error posting to ATS:
org.apache.hadoop.service.ServiceStateException: java.io.IOException: Timeline V1 client is not properly configured. Either timeline service is not enabled or version is not set to 1.x
        at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
        at org.apache.pig.backend.hadoop.PigATSClient.<init>(PigATSClient.java:68)
        at org.apache.pig.backend.hadoop.PigATSClient.getInstance(PigATSClient.java:57)
        at org.apache.pig.PigServer.<init>(PigServer.java:260)
        at org.apache.pig.PigServer.<init>(PigServer.java:219)
        at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:46)
        at org.apache.pig.Main.run(Main.java:495)
        at org.apache.pig.PigRunner.run(PigRunner.java:49)
        at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:273)
        at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:216)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:78)
        at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:67)
        at sun.reflect.NativeMethodaccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodaccessorImpl.invoke(NativeMethodaccessorImpl.java:62)
        at sun.reflect.DelegatingMethodaccessorImpl.invoke(DelegatingMethodaccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:231)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.security.accessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.io.IOException: Timeline V1 client is not properly configured. Either timeline service is not enabled or version is not set to 1.x
        at org.apache.hadoop.yarn.client.api.impl.TimelineclientImpl.serviceInit(TimelineclientImpl.java:100)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        ... 24 more

然后,在错误上方设置此属性后,我设置了yarn.timeline-service.enabled = false,但Pig动作停留在运行状态。 以下是养猪作业日志:


2019-12-06 14:15:40,534 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat  - Total input files to process : 1
2019-12-06 14:15:40,534 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil  - Total input paths to process : 1
2019-12-06 14:15:40,552 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil  - Total input paths (combined) to process : 1
2019-12-06 14:15:40,661 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter  - number of splits:1
2019-12-06 14:15:40,719 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter  - Submitting tokens for job: job_1575641509651_0005
2019-12-06 14:15:40,720 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter  - Executing with tokens: [Kind: mapreduce.job,Service: job_1575641509651_0004,Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@61b2418b),Kind: RM_DELEGATION_TOKEN,Service: 10.0.0.7:8050,Ident: (RM_DELEGATION_TOKEN owner=hdfs,renewer=oozie mr token,realUser=oozie,issueDate=1575641721915,maxDate=1576246521915,sequenceNumber=164,masterKeyId=50)]
2019-12-06 14:15:40,842 [JobControl] INFO  org.apache.hadoop.mapred.YARNRunner  - Job jar is not present. Not adding any jar to the list of resources.
2019-12-06 14:15:41,955 [JobControl] INFO  org.apache.hadoop.conf.Configuration  - resource-types.xml not found
2019-12-06 14:15:41,956 [JobControl] INFO  org.apache.hadoop.yarn.util.resource.ResourceUtils  - Unable to find 'resource-types.xml'.
2019-12-06 14:15:42,216 [JobControl] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl  - Submitted application application_1575641509651_0005
2019-12-06 14:15:42,247 [JobControl] INFO  org.apache.hadoop.mapreduce.Job  - The url to track the job: http://bigdata-m0.b3qiao2a5uce3lh13l1kvcoueg.gx.internal.cloudapp.net:8088/proxy/application_1575641509651_0005/
2019-12-06 14:15:42,247 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher  - HadoopJobId: job_1575641509651_0005
2019-12-06 14:15:42,247 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher  - Processing aliases filterlevel,raw
2019-12-06 14:15:42,248 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher  - detailed locations: M: raw[3,6],filterlevel[-1,-1] C:  R:
2019-12-06 14:15:42,253 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher  - 0% complete
2019-12-06 14:15:42,253 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher  - Running jobs are [job_1575641509651_0005]
Heart beat
Heart beat
Heart beat
Heart beat
2019-12-06 14:17:43,613 [HiveclientCache-cleaner-0] INFO  org.apache.hadoop.hive.metastore.HiveMetaStoreclient  - Closed a connection to metastore,current connections: 1
2019-12-06 14:17:43,current connections: 0
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
End of LogType:stdout.This log file belongs to a running container (container_e24_1575641509651_0004_01_000002) and so may not be complete.

Pig作业似乎失去了与HiveMetaStoreclient的连接,但是所有Hive服务都已启动并且仍在运行,并且仍停留在运行状态。

谢谢

sacklise 回答:在Oozie工作流程中运行Pig操作时,与HiveMetaStoreClient的连接丢失

暂时没有好的解决方案,如果你有好的解决方案,请发邮件至:iooj@foxmail.com
本文链接:https://www.f2er.com/2962140.html

大家都在问