Once upon a time I worked for a small company that didn’t have dedicated customer support engineers. To be fair, we didn’t have the capacity for such a post. Instead, the development engineers manned the customer service phone line on a rota, because it would be good for our souls. It was too; more than one application note was written because someone had answered the same question three times that afternoon.
Forty years of perspective suggests that while it may have been good for us programmers, it was better for the company when we finally employed technical support specialists. Development and support are two very different disciplines requiring very different skills. In this article we will take a look at the similarities and differences between the two.
The first thing to notice is that technical support engineers have to talk to people. While development engineers are not universally geeky bearded individuals with absolutely no people skills (and anyone saying otherwise is spreading fake news, so there), they don’t in general deal with non-technical people. Tech support engineers do that all the time, which makes people skills at least as important as technical skills, if not more so. Consultant engineers like us at Kynesim are somewhat in the middle; we do actually have to talk to other people, but it’s only a core skill of a few of us, and our primary focus is technical design and development.
So a major part of any technical support job is contacting the customer to determine what the actual problem is. Most bug reports from non-technical customers amount to “It doesn’t work,” and bug reports from technical customers are often little better. There are some useful resources for those on the reporting side of bug reports — Simon Tatham’s article “How To Report Bugs Effectively” (https://www.chiark.greenend.org.uk/~sgtatham/bugs.html) and Eric Raymond’s venerable “How To Ask Questions The Smart Way” (http://www.catb.org/~esr/faqs/smart-questions.html) are my personal favorites — but teasing useful answers out of sometimes distraught or angry customers is a skill that is a major benefit to a support engineer.
The main routine of a development engineer’s job (to simplify hugely) is the cycle of the three D’s; design, develop and debug. We figure out what we’re going to write, write it, and then work out what we did wrong knowing what we wrote. Support engineers only have two D’s, but they are harder; diagnose and debug, not having written the code themselves. Figuring out what might be going wrong in a large, complex system is a skill in and of itself, and shouldn’t be underestimated. It is close kin to the test engineers’ skill of figuring out how you might be able to make something go wrong, and often involves a degree of lateral thinking that would make Edward de Bono proud.
My favorite example happened to a radio communications company that had a test room set up as a proper faraday cage so that they could routinely run tests in “clean air” to check the performance of their products. One day the throughput tests went through the floor. This caused a lot of head-scratching; none of the code anywhere near the radio transmit and receive routines had been changed, there wasn’t enough extra load to make that much difference, and none of the test hardware was faulty. After days of many very smart people staring at the code base and failing to find anything wrong, one of the test engineers decided to just stand in the test room, properly take in the environment and see if anything occurred to him. It did. After he ripped the unwisely-installed WiFi repeater off the ceiling, the throughput tests went back to normal.
This is a good example of the way support engineers often need to think outside the box, especially when diagnosing issues with embedded equipment. If that WiFi repeater had been installed on a customer’s site, the customer might well have complained about poor radio performance, and it would have been a lot harder for the support engineer to find out what in the environment had changed than when the engineer had physical access to the room. Even when the issues are purely internal to the software or firmware, it is terribly easy to get tunnel vision and convince yourself that the problem must lie in a particular section of code when it doesn’t. The number of times I have found that part of my code that could not possibly be responsible for what I’m seeing is doing exactly that… well, it’s been more often than I care to admit.
Remember when I said that support engineers only have two D’s to cope with? I lied. There is a third D that a good support engineer does right at the start of working with any code base, that underpins all the rest; they digest. Support engineers typically are not part of the team that created a product in the first place. They have to go into a code base, often with rather sketchy documentation, and understand how it fits together. Having a feel for the control and data flow is essential to figuring out where a fault might lie. Even getting to know the foibles of the development team may help in recognising what is going on. Just imagine trying to solve an issue with say Chrome from a standing start. Here, AI can be a huge help. It can digest a large codebase and help an engineer find their way around much more quickly than most people can unaided.
It feels weird to say that coding skills are less important for support engineers, at least compared to people skills, imagination and comprehension, but it’s true. They do have to be good; no one “just” fixes a bug in the same way that no one “just” implements anything. All changes have consequences. The particular skill that really pays off for support engineers is knowing when they have opened a can of worms. Recognising that the fix for a particular bug will have significant knock-on effects is very important; it tells you that you may need to revisit the design itself. Often this is a clear sign that it’s time to start thinking about the next generation of devices.
Communication, lateral thinking, understanding, and of course programming. These are the skills that support engineers need. Treasure them when you find them.