SyncLock Starvation

We had an issue in production that was particularly hard to track down. One of our applications appeared to be hanging while not doing a whole lot of work. We figured out that it was running a bunch of queries and hanging there. Looking at SQL Server, we saw that the connection was always waiting on async network IO, but the wait time was always pretty short (less than 2 seconds). We saw our application hang for hours at a time.

The funny thing was that this only happened occasionally, and without a good pattern. I had debugged the application several times, but it never had this issue while debugging. Finally, I was able to debug the application while it was hanging. Pausing all the threads, I noticed that they were all waiting for a single SyncLock.

A little background for this SyncLock: we have a bunch of databases that change frequently, so we store the database locations in one central database. To avoid hitting this database all the time, we cache the result in the application's memory. Whenever the application needs a connection string, it would first try the cache. If it's not there, then it goes back to the database to reload the cache. While the cache is being refreshed, it can't serve other threads looking for a connection string, so there's a SyncLock guarding this section.

This works great when we expect all our requests to exist. However, we recently added a new algorithm that checks to see if a connection string exists or not. Most of the time, it does not. This means that the application is constantly refreshing its cache. While it's refreshing, no other threads can access any databases, because the code that gets a connection string is guarded by that SyncLock. Thus, it looks like our application is hanging. The application would've eventually finished its job, but it would've taken a long long time.

This made me curious. Does SyncLock not serve requests in FIFO order? Can a thread be starved while waiting for a SyncLock? The answer appears to be yes. Here's the code to reproduce this.

	Module Module1
	Sub Main()
	Dim lock As New Object
	Dim threadCount = 30
	Dim lastAcquiredTime = Enumerable.Range(0, threadCount).Select(Function(__) DateTime.Now).ToArray
	Enumerable.Range(0, threadCount) _
	.AsParallel.WithDegreeOfParallelism(threadCount) _
	.ForAll(Sub(i)
	While True
	SyncLock lock
	Threading.Thread.Sleep(100)
	Dim now = DateTime.Now
	Dim longestStarvation = lastAcquiredTime.Select(Function(l) now - l).Max
	Console.WriteLine(String.Format("thread {0,2} waited for {1,4}. maximum starvation time: {2,4}",
	i,
	(DateTime.Now - lastAcquiredTime(i)).TotalSeconds,
	longestStarvation.TotalSeconds))
	lastAcquiredTime(i) = DateTime.Now
	End SyncLock
	End While
	End Sub)
	End Sub
	End Module

view raw gistfile1.vb hosted with ❤ by GitHub

Unfortunately, this isn't a super reliable way of producing the situation. We do see, however, that the maximum starvation time occasionally is much larger than it should be. Most of the time the threads are served in near FIFO order, so our application didn't always have an issue. However, sometimes, the threads are served in some other ordering, so our application did appear to be waiting forever for that lock.

Starting Down the Journey to Extreme Programming

I'm trying to get my company to use XP.

A little bit of background first. We are a profitable SaaS shop with about 15 developers. We would like to call ourselves Agile, but we don't follow any practices very strictly. We have 6 week iterations with occasional shorter iterations for bug fixes and late code. We have customer-centric stories with short descriptions and nebulous scope.

So why do I want us to use XP? We regularly miss deadlines. If we are not promising a feature to a customer, there's a good chance it will miss the intended delivery date. We have customers that refuse to pay for our work.  We have last minute changes that set us back by weeks. We have developers who are so bogged down by maintenance that they can't work on their stories.

I think we are a good candidate for XP because we are making something that no one knows how it should work. We are an industry leader coming out with new ideas that change the way the industry thinks. Our customers can't really tell us what they want until they see it.

How are we going to know if it works? One measure is just to see how many missed deadlines we have. Another measure is the team's stress level. We can also measure bugs in new code.

How do we plan on making this work? Since few of our team members have done agile development before, we are going to study agile development together. We are going to discuss how practices can work in our organization, and how we they can benefit us.

There are some practices such as TDD or pair programming that take a while to learn, and will decrease our productivity for a while. We can't really afford to take a huge hit in productivity, so we will take these slowly. I've introduced the ideas to the team, but we won't really use it until a small team of us have proven it to be successful. So we have 4 volunteers who will try out TDD and pair programming on smaller projects that have value, but allow us to try it separately from the rest of the system without impacting deadlines. We will only spend a small percentage of our time experimenting. We will still be using existing methods the rest of the time.

There are some practices that can give us immediate value. We are planning to move to a new space, so we can implement Sit Together to increase collaboration and decrease the communication costs. We can start breaking down our stories into smaller more manageable stories. We can do team estimates.

So that's what we're doing, and we'll see how it goes. Maybe we'll adopt all XP practices in a few months. Maybe we'll pick and choose a few. Maybe we'll decide it doesn't work. We'll definitely gain some valuable experience from trying it out, though.

What Questions Should I Ask at Interviews?

"Do you have any questions for us?" This is a question almost every interviewer asks near the end of the interview. Everyone agrees that the candidate should always ask questions, but the motivation for asking the questions differ.

I've heard that I should ask questions so that the interviewer thinks that I am interested in the company. I should do research on the company and ask questions about what I found out to show that I've done my research and am motivated to work for the company.After having done a few interviews, I disagree with this. 

Personally, I could care less if you've done research or not. We need high quality developers, and we can't find enough who are willing to join us. I will never reject someone because he didn't ask any questions. As for the rest of the team, we've never rejected anyone because they had no idea what we are making. At our candidate summary meetings, we never said we don't want him because he didn't ask any questions.

My wife disagrees with me on this. In her industry, it does matter how much you know about the company. She would like to see that the candidate has done research into the company to know more about the goals and principles of the company. In my experiences with this company, this does not matter at all.

Now I am not suggesting you shouldn't ask questions, just that you shouldn't ask questions just to prove your interest. There are plenty of other questions to ask. You should find out if you're a good fit. In particular, you want to find out things like overtime, weekends, management, evaluations, or typical day. Specifically for programmers, you should ask things like source control, development process, build process, support roles, and involvement in making bigger decisions. There are other sites with some good questions, and you should definitely check them out.