We have it easier than he does, but I'll tell you what our practicals look like:
* We test a lot of web applications, so we rigged up a sacrificial web app that we run on random EC2 instances and have candidates take an hour or two seeing what they can find in it.
* We test a lot of crazy custom client/server stuff, so we built a client/server system with a custom protocol that we have candidates reverse and then attack; we tried to calibrate this so it'd take less than 2 hours to beat.
* We write a lot of fuzzers, so we have candidates write a fuzzer for a file format.
Key attributes of good practical tests in our experience:
* They can't take too long to complete.
* They have to relate somehow to the actual work candidates would be doing within the first six months.
* They have to produce results that are objectively comparable in some fashion to those of other candidates, so we can learn from the trends. This is probably the most important thing I've learned about hiring in the past year.
We'll probably swap the fuzzer challenge for a memory corruption challenge sometime this quarter, because the fuzzer scores high on the first two attributes and low on the last.
I'd love to do a crypto challenge (mail sean AT matasano dot com if you'd like to do a bunch of crypto challenges, by the way; just say "Thomas told me to mail you"), but they score low on the first two attributes.
So if I was doing engineering interviews again (About 10 years of my career were as a full-time dev), I'd probably look to:
* Taking a time-inefficient algorithm and making it faster
* Taking a space-inefficient system and making it tighter
* Writing an importer for a specific file format
I agree that actually asking candidates to jump in and produce code for issues in Github is a losing strategy. In particular, because you're making it unnecessarily hard to learn from your hiring decisions; when you look at someone who is kicking ass on your team, can you go back to the hiring decision that selected them and see whether they outperformed or underperformed a metric and adjust the metric accordingly? If not, your hiring process is suboptimal.
Thanks. I guess this is a bit easier for you because you are a consultancy and have lots of different projects going on. You also work in a very technical domain (I assume - really, I know little about what you're doing).
Where I'm having a hard time seeing this play out, is for a business that has an in-house software platform supporting it. Or for shops that develop and maintain a products.
* We test a lot of web applications, so we rigged up a sacrificial web app that we run on random EC2 instances and have candidates take an hour or two seeing what they can find in it.
* We test a lot of crazy custom client/server stuff, so we built a client/server system with a custom protocol that we have candidates reverse and then attack; we tried to calibrate this so it'd take less than 2 hours to beat.
* We write a lot of fuzzers, so we have candidates write a fuzzer for a file format.
Key attributes of good practical tests in our experience:
* They can't take too long to complete.
* They have to relate somehow to the actual work candidates would be doing within the first six months.
* They have to produce results that are objectively comparable in some fashion to those of other candidates, so we can learn from the trends. This is probably the most important thing I've learned about hiring in the past year.
We'll probably swap the fuzzer challenge for a memory corruption challenge sometime this quarter, because the fuzzer scores high on the first two attributes and low on the last.
I'd love to do a crypto challenge (mail sean AT matasano dot com if you'd like to do a bunch of crypto challenges, by the way; just say "Thomas told me to mail you"), but they score low on the first two attributes.
So if I was doing engineering interviews again (About 10 years of my career were as a full-time dev), I'd probably look to:
* Taking a time-inefficient algorithm and making it faster
* Taking a space-inefficient system and making it tighter
* Writing an importer for a specific file format
I agree that actually asking candidates to jump in and produce code for issues in Github is a losing strategy. In particular, because you're making it unnecessarily hard to learn from your hiring decisions; when you look at someone who is kicking ass on your team, can you go back to the hiring decision that selected them and see whether they outperformed or underperformed a metric and adjust the metric accordingly? If not, your hiring process is suboptimal.