For the past couple of months our team has been working on some very interesting M2M and IoT software projects. I thought we’d share a little challenge that we overcame this week. Here’s a summary of the discussion
Team Lead:
We have a multi-threaded environment. We had a software representation with multiple hardware components each with their own thread doing polling. As part of the polling we had to establish physical communications to the physical hardware.
When we were trying to to establish the communication we discovered that all ten threads, talking to the thread pool, were trying to initialize that thread pool all at the same time. But you only need one thread to initialize. Once it’s initialized that thread pool will only have a single object available in it, the connection object. Subsequent queries against the thread pool will then give that single initialized connection to whatever devices asks for it. The problem was that all ten threads were trying to initialize the thread pool all at the same time.
In order to prevent the problem a simple synchronized keyword on the initialize method allowed a single thread to initialize the pool while blocking all others until such time as the pool was initialized. And subsequently all other threads would get the single the single connection that was created as part of the initialization the communications went forward just fine. Before we had the synchronization keyword all ten threads were trying to initialize it and they were all trying to create that new connection causing the entire communication to freeze up.
Me
Sounds like, me walking in the door, and all 6 of my kids asking “Dad help me with my homework”. Who do I listen to first?
Team Lead:
Yes, the power of the key word, [or in your case with the kids holding up your hand to focus on one], changes the dynamic of the communication and how the system operates.
Me:
How did you figure it out?
Team Lead:
We did two things. The first thing we was to go and set up a simulator. We connected up our system to a PLC simulator representing the hardware we are communicating to. When we connected to the simulator the issue was timing related. If the timing changed and we didn’t have all ten threads going in at the same time it would communicate fine. So depending on the timing of the whole system, how speedy it was and what devices you were talking to it would not work so it was an intermittent issue. We connected it up to the PLC simulator and we could talk to it just fine. Then we pulled the simulator down to see, as a test, how our system operated when it wasn’t communicating. We were deliberately trying to break it.
We noticed that when we pulled the simulator down then tried to reconnect it by pulling the simulator back up we then ran into the issue. All of sudden we couldn’t communicate and we couldn’t reestablish communication. We found out that our in our pool that once you’ve created an error on that connection it wouldn’t talk to you any more. It was a dead connection. And so we made sure the connection could be reestablished. We recreated the connection pool when ever one of the clients said, “the connection pool you have, I tried to use it and it doesn’t work, so would you mind fixing that for me”.
We made it so that the connection pool goes and deletes the only one and creates a new one for the device. It could then reestablish connection. That’s what we did in the code so we could pull the hardware up and down with out causing an issue. But in that process of going to tell the connection pool to fix itself we noticed that it made us focus on that particular component.
We were still having some intermittent start up issues so I put some logging in the pool area to isolate the problem. The very first time I started it up with logging in place I noticed that there was a request on the connection pool to initialize itself ten times. That’s when we had proof that each one of the threads were trying to initialize it. We simply put a synchronize on the method then we had two problems solved for the price of one symptom. – Problem solved
Jody Sedrick, CEO
Follow me on Twitter: @jodysedrick and @zenwareinc
LinkedIn Profile: Jodysedrick
Zenware Inc. is a Boise Idaho based company that specializes In custom mobile software development, custom iOS development, custom Android development, iPad software development, iPhone software development, custom web development, web hosting and SaaS based work order management software. We also have a powerful line up of off-the-shelf mobile cloud solutions; RoadFS, and ZWarehouse
Jody Sedrick, CEO
Follow me on Twitter: @jodysedrick and @zenwareinc
LinkedIn Profile: Jodysedrick
Zenware Inc. is a Boise Idaho based company that specializes In custom mobile software development, custom iOS development, custom Android development, iPad software development, iPhone software development, custom web development, web hosting and SaaS based work order management software. We also have a powerful line up of off-the-shelf mobile cloud solutions; RoadFS, and ZWarehouse