Monday, February 1, 2016

[BigData] About error saying "Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=XXXXX"

Hi,

I will talk about an error when people run query on HUE and get following error. Interesting part is that when a developer gets this error, another developer in same team works succesfully. In this post, i will investigate the error from HIVE server logs and the solution.

"Fetching results ran into the following error(s):
Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=b1d336e8-3c7e-42f3-9f2a-3f3b9cb4fa21]
Bad status for request TFetchResultsReq"


First of all, you must know that HUE shows generic error logs and you must go deep with hive error logs.

This post is written for Oracle BigData Appliance v4.2 with CDH 5.4.0.

When the developer called me, i searched hive logs and see following:

...
2016-01-14 14:06:13,139 WARN org.apache.hive.service.cli.operation.Operation: Unable to create operation log file: /tmp/hive/operation_logs/d6c316f4-476e-497a-8dbb-06384e61d6e8/c6dc6899-fab2-40d0-abe2-5715dde0c84b 
 java.io.IOException: No such file or directory 
 at java.io.UnixFileSystem.createFileExclusively(Native Method) 
 at java.io.File.createNewFile(File.java:1012) 
 at org.apache.hive.service.cli.operation.Operation.createOperationLog(Operation.java:195) 
 at org.apache.hive.service.cli.operation.Operation.beforeRun(Operation.java:237 
...

I checked for permissions, but i think permissions are correct. 

$ hadoop fs -ls /tmp/hive/operation_logs 

 drwxr-xr-x 11 hive hive 4096 Jan 14 15:09 operation_logs 

As you see hive user can read/write the path. 

Remember, in my previous posts i configured EdgeNode to connect BDA targets in order to run several hadoop tools. With trying on edgenode, same developer does not get this error. 

Because of hive operation logs are created on local tmp files and user who runs query does not get error as well. So what is the problem?

After some search and Cloudera suggestions, there is a parameter you need to set on Cloudera Manager.
hive.server2.logging.operation.enabled , you need to check it on your CMs.

I checked my CM and didn't see an entry for it, you can set this parameter with following way:

Go to CM ->Hive->Configuration and find Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml

After adding following lines and restarting CM, the error disappears.

 <property> 
 <name>hive.server2.logging.operation.enabled</name> 
 <value>true</value> 
 </property>

I think it is a bug, because in hive documentation it is said that parameter is TRUE.

I wish this helps you, too.

Ok, That's all.

Thanks for reading.

Enjoy & share.

Source:









2 comments :

  1. teşekkürler kamudba'i.

    paylaşımlarını takip ediyoruz.

    ReplyDelete
  2. really Good blog post.provided a helpful information.I hope that you will post more updates like thisHadoop Administration Online Course Bangalore

    ReplyDelete