Flash Media Server: Core is no longer active (update)

In a past few weeks I have been working on a middle sized flash project. While its nature is in multi user communication and it was crucial to do that with low latency, our first choice was of course Flash Media Server. I spent few days by programming server side (fms) and few days on optimizing its performance and debugging and finally it was fast enough to get staged live. Server side was just about calling client methods (client -> fms -> client) and some SharedObjects, but no video nor audio streaming was used. My company dedicated super fast server (4 x quadcore Intel(R) Xeon(R) CPU E7440 @ 2.40GHz, 32 GB ram) just for Flash Media Server so we were able to run it on 16 cores. We installed fms v.3.0.1 r123 with valid unlimited licence on it. And went live.

First, we tried default fms configuration and soon after start master.log (and all other logs) started to grow up. The main content of the log was:

2009-10-07 00:03:31 24962 (i)2581223 Core (8684) is no longer active. -
2009-10-07 00:03:31 24962 (w)2581256 Core (8684) _defaultRoot_:_defaultVHost_:::_1 experienced 1 failure[s]! -
2009-10-07 00:03:31 24962 (i)2581221 Core (10772) started, arguments : -adaptor "_defaultRoot_" -vhost "_defaultVHost_" -app  -inst  -tag "_1" -conf "./conf/Server.xml" -name "_defaultRoot_:_defaultVHost_:::_1". -

This caused our client app to be unconnected for a few seconds but connected automaticaly back. So no big deal, right? … Well, after few days server started to refuse any connection into running instances (even with 1-2 clients in it). So there was no other possibility than to restart whole fms. We started thinking this is not good. So we taked action…

Well, what uncle google know about this? 3 000 000 results? Results taked me to adobe forums and I found there was dozens of topics with same problems. But no working solution for noone.

First, we tried to configure and reconfigure everything here and there, from default 3 cores to 1 than to 16 cores, disconnectiong idle clients etc. … but still the same. Master.log was growing lineary as more and more clients got connected at the same time (we are not talking about huge loads, just 10-20 simultaneous connections). Core crashes logged every 5 minutes, and after few days server refusing all client connections. :-(

Okay lets handle this by watch dog. We created java watchdog based on testing connection into instance, and fmscheck watchdog to check all running instances periodically every 10 minutes just to know when server is down (refusing connections). Soon we discovered, java watchdog reported correctly and based on its output we were able to restart fms in case of crash. But this was not the solution.

The next step was to contact support. I created and posted article to adobe forum, but noone has replied yet. As an Adobe client we sent e-mail to Adobe support but no reply. Finally our local solution manager for adobe replied that we should try to upgrade to v.3.0.4. We did, but nothing changed. We even tried to run it on v.3.5.x but still the same results.

Our administrators even tried to trace core crash on system level “strace -p PID” but when attached on process, core never crashed. Solution? Not at all, as I was told tracing is slowing down system. There is one more way to trace crash (Core crash on linux) but we never gave it a try based on all previous failures…

Few weeks after we launched our project we were still not able to make it run smoothly, core crashed here and there and our client was not happy of course…

We were forced to change socket server and as an alternative we have chosen Red5. While it is written in java, we were able to rebuild and debug server side in a few days. And now our project runs great. Our solution is open source flash server Red5.

I am really disappointed about fms because:

  • fms is crashy and broken, log files does not contain important informations
  • even if we paid big money for licence we did not got any support that would help us
  • server side scripting is based on some kind of extended version of JavaScript 1.5 what is definitely not a good language for server side now, Adobe please wake up
  • I am not mentioning browser crashes caused by corrupted SharedObject

It is great we have all these new features with every new version of Flash Media Server, but as long as it is absolutely unstable, I do not know whether I will ever count on Flash Media Server anymore.

update Mar 10, 2010:

We have seen similar issues and the best work around is to have your clients connect on alternate cores/servers. The way to accomplish this really depends on your type of application. (Adobe Forums)

4 comments so far

  1. Tim February 13, 2012 20:47

    Jozef, How did RED5 work out for you? Here we are (2012) and I have the same problem you have above.

  2. Jozef Chúťka February 14, 2012 11:09

    hi Tim, we were pretty satisfied with red5 even it was 0.x rc version a few years ago. I have not been working with any other rtmp server ever since, so there may be even better alternatives available now.

  3. QC August 13, 2012 01:35

    This is caused by in Application.xml (by default is 1MB), in FMS 3 there is a 50MB limit, no limit in FMS>4

    In the official Adobe forums no one (including staff) told me that I must modify this as it comes by default to 1MB, but FMS doesn’t throw the error it’s supposed to throw (javascript out of memory) but it just do what you’ve explained… core is no longer active stuff with no javascript warnings/errors.

    Seems that the upper limit for runtimesize calls the garbage collection while the server is busy and this is the cause of the instability, the refusing of connections and finally, the core being shut down.

    It’s been a hard way for me to discover this simple –and stupid– thing, and as I’ve said, no one including Adobe staff have told me that simple thing.

  4. Jozef Chúťka August 15, 2012 09:10

    QC, thank you for the info

Leave a comment

Please be polite and on topic. Your e-mail will never be published.