Wednesday, November 4, 2015

[NoSQL] Solving "No RMI service for adminX" error on a storage node

Hi,
I will talk about an error that i see from NoSQL Administrative Console page, says that one of the admin services is not running. The error says "Verification violation: admin[3] No RMI service for admin3: service name=commandService". In this post, i will talk about how to search logs of storage nodes and how to bring up the admin service.

We can monitor our NoSQL database (12.1.3.2.5) via Administrative Console page from port 5001, or command line and also Oracle Enterprise Manager. I will talk about later how we can do monitoring with OEM.

When i jump into topology browser, i see following error


It says "Verification violation: admin[3] No RMI service for admin3: service name=commandService".  You can also logon NoSQL database and verify topology with this:

$ java -jar $KVHOME/lib/kvstore.jar runadmin -port 5000 -host localhost -security adminlogin.txt
Logged in admin as root
kv-> 
kv-> verify
The command:
        verify [-silent]
is deprecated and has been replaced by: 
        verify configuration [-silent]
...

With verify command you will see the following output , you can see that admin service on node 3 is not functioning


After that we should look for logs which is located under $KVROOT/$KVSTORE/log directory

Here we should look for admin3_0.log and it says following:

2015-11-04 02:43:43.533 UTC SEVERE [admin3] Exception creating service admin3: Problem creating log file for logger $KVSTORE(12.1.3.2.5): Problem creating log file for logger $KVSTORE(12.1.3.2.5) java.lang.IllegalStateException: Problem creating log file for logger $KVSTORE
        at oracle.kv.impl.util.server.LoggerUtils.addFileHandler(LoggerUtils.java:697)
        at oracle.kv.impl.util.server.LoggerUtils.getStatFileLogger(LoggerUtils.java:416)
        at oracle.kv.impl.monitor.views.PerfView.<init>(PerfView.java:85)
        at oracle.kv.impl.monitor.Monitor.createPredefinedViews(Monitor.java:269)
        at oracle.kv.impl.monitor.Monitor.<init>(Monitor.java:187)
        at oracle.kv.impl.admin.Admin.<init>(Admin.java:351)
        at oracle.kv.impl.admin.AdminService.startInternal(AdminService.java:224)
        at oracle.kv.impl.admin.AdminService.access$000(AdminService.java:95)
        at oracle.kv.impl.admin.AdminService$1.execute(AdminService.java:200)
        at oracle.kv.impl.fault.ProcessFaultHandler.execute(ProcessFaultHandler.java:179)
        at oracle.kv.impl.admin.AdminService.start(AdminService.java:195)
        at oracle.kv.impl.sna.ManagedAdmin.start(ManagedAdmin.java:210)
        at oracle.kv.impl.sna.ManagedService.main(ManagedService.java:655)
Caused by: java.io.FileNotFoundException: /opt/oracle/kvroot/$KVSTORE/log/$KVSTORE.stat (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
        at java.util.logging.FileHandler.open(FileHandler.java:210)
        at java.util.logging.FileHandler.openFiles(FileHandler.java:536)
        at java.util.logging.FileHandler.<init>(FileHandler.java:402)
        at oracle.kv.util.FileHandler.<init>(FileHandler.java:73)
        at oracle.kv.impl.util.server.LoggerUtils.addFileHandler(LoggerUtils.java:687)
        ... 12 more

        at oracle.kv.impl.admin.AdminServiceFaultHandler.getThrowException(AdminServiceFaultHandler.java:132)
        at oracle.kv.impl.fault.ProcessFaultHandler.rethrow(ProcessFaultHandler.java:244)
        at oracle.kv.impl.fault.ProcessFaultHandler.execute(ProcessFaultHandler.java:181)
        at oracle.kv.impl.admin.AdminService.start(AdminService.java:195)
        at oracle.kv.impl.sna.ManagedAdmin.start(ManagedAdmin.java:210)
        at oracle.kv.impl.sna.ManagedService.main(ManagedService.java:655)

What i understood from the log file is that it cannot read the file $KVSTORE.stat and so it cannot start admin service. When i look for permissions, i see that file has root access. After correcting it to oracle:oinstall, it succesfully started the admin service.

For stopping and starting a storage node

$ java  -Xmx256m -Xms256m -jar $KVHOME/lib/kvstore.jar stop -root $KVROOT  -config config.xml

$nohup java  -Xmx256m -Xms256m -jar $KVHOME/lib/kvstore.jar start -root $KVROOT -config config.xml &

After that you can see logs that admin service started. And again verify topology from Administrative Console or command line to see following:

Verify Configuration Results
Verify: starting verification of KVSTORE based upon topology sequence #772 
720 partitions and 3 storage nodes. Version: 12.1.3.2.5 Time: 2015-11-04 16:52:21 UTC 
Verification complete, no violations. 

That's all.

Thanks for reading.

Enjoy & share.






No comments :

Post a Comment