A java server project of mine use Hazelcast 3.8.1, and it’s almost online for public. before public, I always need to make sure it’s been well tested with
pressure, try to avoid crash down when we first launch so many player will online at the same time.
Our game is a HTML 5 game, it’s a real time MOBA fight game. client use web socket to connect with server, the player can act like different animals, like sheep, lion and rabbit, different animal can drop different bomb, and the
bomb will explode after a very shot time, players need to pick up coins in the map, and drop bomb will cost coin, players will kill each other by putting bomb near other players.
I spent a few days to create a robot pressure test tool, try to simulate about 500~2000 clients login the server and take actions like create a fighting room, join room, start fighting, A* path finding, find an pick coin in the map, drop boom to kill each other.
For Hazelcast, I use it not only as distributed memory, but also use it as message service. Almost every other modules in server rely on it.
after my google searching, I’ve concluded that maybe this is why we encounter above performance cases:
- Our java class which need to transfer between hazelcast, At the very beginning we just using the default serialize mechanism, we just impletment java.io.Serializabl
e, and java.io.Serializable is a slow and heavy serialization mechanism, we should switch to a more faster serialization solution like kryo or maybe there is better one.
- We send messages using hazelcast.IQueue too frequetly , if we send message at a very high frequency, It will invoke serialization when sending and deserialization when receiving. maybe that why we cost so many CPU time on IQueue.put() and IQueue.take(), we should reduce the IQueu.put() and take() invoke frequency.
- At the very beginning, we just use one thread to do IQueue.put() or IQueue.take(), but look like serialization is a CPU cost factor, maybe we need to use multiple thread to do IQueue.put() and IQueue.take.
- IQueue will do some action like backup or synchronize to other hazelcast node, in our use cacse for game server, we just want it to be a message queue between our servers, maybe we need to disable backup/synchronize to reduce network transfer, for transfer object to backup/synchronize to other hazelcast node will invoke serialization or deserialazation too, maybe that’s why com.hazeclast.internal.net
working.*.NoBlockingIOthread.* consumed so much CPU time.
- maybe Hazelcast.Ringbuffer is better solution in use case like transfer message between server node, we should replace IQueue, switch to Ringbuffer, for it have no sync and not too much allocate frequency, the object inside it can set a live time, it’s useful for us for the case if no consumer to consume message for timeout, it will be evicted. it’s a RingBuffer , like a round, just run to the end to restart from the beginning, just set the capability size. Hazelcast.Ringbuffer mus
t be the best solution for us, and we are using 3.8.1 version, already contained Ringbuffer.
- switch to kryo serialization, I follow this article found on hazelcast blog, https://blog.hazelcast.com/use
- I’ve modified our server hazelcast configuration, add configuration like this:
<serialization> <serializers> <global-serializer override-java-serialization="t
rue"> info.jerrinot.subzero.Serializ er </global-serializer> </serializers> </serialization>
I've modified my maven pom.xml, add configuration like this:
d> <artifactId>subzero-all</artif actId> <version>0.6</version> </dependency>
after done, I debug follow inside the serialization and deserialization when IQueue.put() or take(), I found that kryo already is working, the kryo method has been invoked.
- Reduce hazelcast.IQueue.put() and take() frequency, when we need to send message, don’t put it to hazelcast.IQueue immediatel
y, put it to a java.util.concurrent.Concurr entLinkedQueue like a producer role, and there is a thread pool act like a consumer role, thread pool will have multiple thread consumed the messages, composed many messages to a list, and put that list to hazelcast.IQueue once a time, the receiver side will IQueue.take() the list contain many message once a time. so it will decrease IQueue.put() and take() frequency. If put() and take() invoke reduce, serialize and deserialize reduce too.
- I built thread pool to do hazelcast.IQueue.put() and take(). my purpose is use multi thread to help me to make the serialize and deserialize operation more faster.
- I disable hazelcast.IQueue async backup and set a max size. I add code like this when I create hazelcast.IQueue:
QueueConfig queueConfig = config.getQueueConfig("default
"); queueConfig.setName(BattleServ er.getAppId()) .setBackupCount(0) .setMaxSize(1000000) .setAsyncBackupCount(0); config.addQueueConfig(queueCon fig);
- I’ve tried hazelcast.Ringbuffer, but I encountered many issues, because I’m using a thread pool to do Ringbuffer.addAsync() and Ringbuffer.readManyAsync(), when application start, I use a atomic long variable to get Ringbuffer.headSequence(), when I addAsync() one item to Ringbuffer, I let that atomic long increase one, and when I readManyAsync() item from Ringbuffer, I decrease the same count from that atomic long variable, that atomic long sequence variable will be used when invoke readManyAsync(), and I got exception look like the atomic long sequence I maintained is smaller than the headSequence or bigger than the tailSequence when my application run for a few minutes. If I don’t use multiple thread, just only on thread to write, and only one thread to read, then every thing works fine. I think maybe I miss-understanding the sequence mechanism inside Ringbuffer, I’ve tried to find more information about Ringbuffer in hazelcast website, try to find some example code about Ringbuffer how to maintain the sequence variable when invoke readManyAsync() in multiple thread, but it’s not too easy to found.