Monday, November 3, 2008

Zen and the Art of LocalConnection Maintenance

My colleague Jono Spiro hassled me to post this for my own benefit weeks ago, so here it is, fashionably late... This is culled from conversations with my pal, and ex-Player dev, Peter Grandmaison along with my own experiences working with LocalConnections

LocalConnection uses a block of shared memory to exchange messages between player processes. Outgoing messages targeting a LocalConnection are serialized and added to a queue within the sending player process, and these messages are moved into shared memory during idle time if there's room. Similarly, if messages sent to a LocalConnection owned by the player process are resident in shared memory, they're drained from shared memory during this idle time and processed by the player.

If the recipient process goes away without cleanly releasing its ownership of a LocalConnection, this can end up wedging the shared memory for the LocalConnection because messages aren't being removed for processing. Without any draining, there's no room for new messages.

The most common scenario leading to this is a player instance (or Breeze plugin, or AIR, or...) crashing or being force quit. Stopping a debug session in FlexBuilder behaves this way.

So, a fix was implemented where any player process polling shared memory will expunge anything stored there for more than 5 seconds regardless of which LocalConnection the messages are associated with. This reap cycle after 5 seconds prevents shared memory for a LocalConnection from wedging indefinitely - yay!

The implied caution here is to not send messages to a LocalConnection that will take the receiving process longer than a second or two to process. If you do, you run the risk of subsequent messages magically vanishing into the ether if they're reaped due to the 5 second rule.

In addition to the 5 second rule there's a size limit of 40K on the data sent in each message, and that's covered in the LiveDocs for LocalConnection.

So the take away is to keep your LocalConnection interactions short and sweet.

If you're trying to ship a large amount of data from one player process to another, do your best to limit the total amount of data you need to send, and if it's still substantial then slice it up to send in chunks. Make sure each chunk is guaranteed to be processed by the receiver(s) in well under 5 seconds.

The simplest way to handle this sort of chunking is to have the sending player process open its own LocalConnection that the receiving processes can 'ack' back over. This lets you chain the transfer like so, and protects against the sender potentially flooding the receiver:

Player 1 ---- sends data chunk over primary LC ---> Player 2
Player 1 <----- returns ack over secondary LC ------ Player 2
... rinse and repeat ...

Also, if the process that opened a LocalConnection doesn't exit cleanly, other processes will receive a "connection in use" error if they attempt to create a LocalConnection having the same name. You can work around this by implementing a retry on an interval slightly longer than 5 seconds, which gives the current process a chance to clean up the orphaned shared memory at which point it can successfully create the desired LocalConnection.


Jono said...

I love the title :) You probably could have come up with something about the "Five Second Rule" too. We should get Mythbusters to take this on.

BTW is there code you could reference in LCDS that roughly does this?

Seth Hodgson said...

Grant from Mythbusters lived at the same West Oakland lofts we used to be in a few years back :)

There's code in the LCDS client library that uses LocalConnections to broker access to local SharedObjects for offline data caching. That code's not public, and it should take these concerns into account.