我已经在多个测试Kubernetes集群中部署了StatefulSet的Apache Ignite。
我已经通过当前配置通过了压力测试阶段。但是,我发现Apache Ignite中存在一些OutOfMemory错误,而某些新的测试集群中的错误却要低得多。
以下是我从1个Ignite实例中提取的日志快照:
INFO: TCP discovery spawning a new thread for connection [rmtAddr=/10.254.174.226,rmtPort=45453]
Nov 15,2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: GridWorker [name=tcp-disco-client-message-worker,igniteInstanceName=null,finished=false,heartbeatTs=1573779638619,hashCode=373238347,interrupted=true,runner=tcp-disco-client-message-worker-#109]
java.lang.OutOfMemoryError: Java heap space
Nov 15,2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: IgniteSpiThread [name=tcp-disco-client-message-worker-#109]
java.lang.OutOfMemoryError: Java heap space
Exception in thread "tcp-disco-client-message-worker-#109" java.lang.OutOfMemoryError: Java heap space
Nov 15,2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger info
INFO: TCP discovery accepted incoming connection [rmtAddr=/10.254.183.232,rmtPort=41313]
Nov 15,2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger info
INFO: TCP discovery spawning a new thread for connection [rmtAddr=/10.254.183.232,2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Started serving remote node connection [rmtAddr=/10.254.174.226:45453,2019 @ 09:01:26.612 Nov 15,2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger warning
Nov 15,2019 @ 09:01:26.612 WARNING: New next node has connection to it's previous,trying previous again. [next=TcpdiscoveryNode [id=5cbb5f1c-ca74-4b2f-ba70-314f621ab997,addrs=[10.254.168.12,127.0.0.1],sockAddrs=[/127.0.0.1:47500,ignite-sit-5.ignite-sit.sit.svc.cluster.local/10.254.168.12:47500],discPort=47500,order=3922,intOrder=2000,lastExchangeTime=1573779246139,loc=false,ver=2.7.5#20190603-sha1:be4f2a15,isClient=false]]
Nov 15,2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger info
Nov 15,2019 @ 09:01:26.612 INFO: New next node [newNext=TcpdiscoveryNode [id=6fcccf11-f903-4b4a-bbac-730ca0b80ce8,addrs=[10.254.169.217,ignite-sit-4.ignite-sit.sit.svc.cluster.local/10.254.169.217:47500],order=3912,intOrder=1993,lastExchangeTime=1573779190075,2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Finished serving remote node connection [rmtAddr=/10.254.174.226:45453,rmtPort=45453
Nov 15,2019 1:01:26 AM org.apache.ignite.logger.java.JavaLogger error
对不起,错误的日志格式。
我想知道是什么原因导致OutOfMemory错误,以及如何防止再次发生这种情况。
我们将不胜感激。
更新: 堆转储分析结果:
The thread org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader @ 0xd9fbe2d0 tcp-disco-sock-reader-#369 keeps local variables with total size 312,295,344 (48.95%) bytes.
TCP SocketReader似乎需要大量堆内存。