Reality Check

Blog archive
exam

Programming tests separate wheat from chaff

I recently helped a recruiter friend’s efforts by developing an answer sheet for a programming test his company was using. It turns out some of his candidates with advanced degrees were being rejected due to poor performance on this particular test. His thinking was that if I could take the test and provide him with an answer sheet, he could better screen his potential candidates and not waste time passing along those who had little chance of being accepted.

The programming test asked only one question on Structured Query Language and a few  about code development for the C programming language. The C programming language questions, by the way, revolved around the subtleties of pointers, pointer pointers and function pointers, which I covered in detail in my 1993 book, “C Pointers and Dynamic Memory Management.” 

The use of programming tests to screen job candidates is a widely debated topic. Some arguments against these tests are the amount of time they take, the use of poor/limited/obscure test questions, or that the tests themselves are “offensive” or insulting in their focus on arcane knowledge. 

Proponents of testing say they can weed out incompetent programmers (some people will quit or refuse to take tests instead of embarrassing themselves, which is an indicator in and of itself). Others say tests can find exceptional programmers (Facebook uses testing this way) or screen new programmers (the infamous FizzBuzz question).  I should also note that there are variations of the programming test to include timed tests, and audition projects that have their own set of pros and cons.

Over my career I have clashed a few times with the human resources department over my desire to use programming tests in screening applicants. The HR department would not allow the testing, citing “legal reasons” related to questions that may be misconstrued or biased in some manner (like toward English speakers). Personally, I think that HR was being overly cautious because I was talking about programming questions not personal questions.

In my experience, programming tests are extremely effective in screening candidates and far superior to using interviews alone. I have been burned by hiring people who talked a good game but failed to deliver. While I understand that some people don’t test well or are insulted by the “gotcha” type questions, I consider these (and my experience confirms them) to be more the exception than the rule.

For me, the central and most important reason that programming tests work is they ensure the person can demonstrate a certain level of competency. So, what are the ramifications of this for government agencies? Using well-designed and well-written programming tests are worth the price of legal scrutiny to effectively screen candidates.

Michael C. Daconta (mdaconta@incadencecorp.com) is the Vice President of Advanced Technology at InCadence Strategic Solutions and the former Metadata Program Manager for the Homeland Security Department. His new book is entitled, The Great Cloud Migration: Your Roadmap to Cloud Computing, Big Data and Linked Data.

Posted by Michael C. Daconta on Dec 04, 2013 at 11:16 AM


Reader Comments

Thu, Dec 5, 2013 JoJo MD

Programming tests are important. Otherwise how can you tell that you are getting someone who has the skills that you need? As an interviewee, I do appreciate forewarning of a programming test, but I wish they were more common in the hiring process. Also, if you are giving more than a basic test, I think allowing the developer to be able to write and compile the program (without internet access) is good. Sometimes employers will look at someone and decide if the person is a good programmer or look at the purported job experience, which can be fake. This is a bad idea and is damaging for people who may look inept or have trouble presenting themselves, but are actually qualified. A test can allow employers to be unbiased and give jobs to qualified candidates, regardless of origin, color, gender etc. Tests can be difficult and nerve wracking, but writing software is a technical job and the pay is not inconsequential. Setting up a test is worth it. People in the US love to sue, but that obviously does not stop testing by all companies so IMO that is moot. If someone sues because they did not pass a programming test... I would ask them to take one in court. I would definitely give a test to anyone I was going to hire to do software development. I have only had to take two programming tests over my career, and I think that is crazy. I think they should be a part of any interview process for a development job. Sadly, I suspect that part of the reason that testing is not as common anymore is because the people who are doing the hiring may not feel competent enough to make or score a test. Even if the candidate made mistakes, I think this would even be OK depending on the level of the job and the type of test as long as he/she could have a discussion about how to proceed and what the issues were...indicating knowledge of the language and programming in general as it applies to the job. Any competent senior developer can be comfortable with doing this with an interviewee. Software is used for important systems. If the systems have to work, just like real engineers, we computer scientists should submit to testing. Saying you know how to do something properly, is not the same as knowing how. And doing badly on a test can be a way to identify our own weaknesses and correct them. That is good for both employers and workers. Employers are not balking at the liability of using credit checks and background checks to keep people out of jobs, why is a test that is related to the skills needed for the job any more of an issue?

Thu, Dec 5, 2013 earth

Let me add to my last post that I am not actually against tests at interviews. They give me a chance to see if the potential employer are a bunch of the self-important morons one finds at frat parties who’s grades depend on the fraternities test files and their advancement is due to networking instead of skill. Just as interviewers test the interviewee to see if they are a potential match to the job, the questions asked indicate whether the interviewers have the ability to come up with probing and insightful questions on the fly. If they don’t, they will likely be burned by their interviewing skills, and this is an indication that they might not be able to probe and get insights on the fly with programming problems. (one of the top skills needed in a programmer, and almost impossible to test for with a knowledge of X language test) A written test or pre prepared questions indicate the interviewers don’t have confidence in their own competence. As a manager I am more concerned with whether the candidate has the ability to do TDD and can use the debugging/refactoring exercise to improve their depth and breath of experience as well as whether they can fit within the existing group. To this end I find the most important answer is to the question: “describe your own sense of humor”.

Thu, Dec 5, 2013 earth

Anybody can create a test. But to create a test that actually measures what you want it to measure takes training. The majority of the people that create computer oriented tests have not had that training. The result, as the author mentioned, is a variety of ad hoc responses that, over a large population of tests, have a small population of them that work correctly. This gives some antidotal evidence that they work. My experience in both creating the tests and taking them is that that population is around 5% of the total. My suggestion is that if you want to use a test, find someone that has the appropriate training in test design and has a good knowledge of the duties the employee will need to fulfill, as well as the ability to design a test that measures the ability of a candidate to deal with the unknown technologies they will have to deal with in the future. (Good luck with that.) A one-size fits all test for government employee programmers isn’t going to give you any concept of whether the person can do the work in any particular position any more than giving all 13 year olds 34” pants and size 8 shirts to wear. Not with the depth and breadth of the technologies that exist in the field nor with the rate of change in the technological advancement. I had to pass three clearly arbitrary tests to get a certification to keep my job and a year later, the certification wasn’t even recognized by its creator, Microsoft. Standardized testing over any field with wide breath, deep depth and constant flux, is a money making scheme, no more, no less.

Please post your comments here. Comments are moderated, so they may not appear immediately after submitting. We will not post comments that we consider abusive or off-topic.

Please type the letters/numbers you see above

resources

HTML - No Current Item Deck
  • Transforming Constituent Services with Business Process Management
  • Improving Performance in Hybrid Clouds
  • Data Center Consolidation & Energy Efficiency in Federal Facilities