It's 7:30 pm. The office is quiet, most people have left for the evening. You texted home to say you'll be late for dinner around 6:00. The cleaning crew came by half an hour ago to empty the trash and sweep up. You are staring at your screen, looking at the same block of code over and over again waiting for an insight to appear. Your users are blocked by a bug and are waiting for your fix. You are now in the uncomfortable position of being in the critical path of a tapeout.
By 8:30 you decide to just go home. All the way home you are replaying the bug's symptoms in your head searching for a new approach -- something else to try that would illuminate the problem. At home you reheat some cold food in the microwave. The kids are already in bed. While you are eating you don't really pay attention to the food, your brain is still locked on the bug. What the heck is going on? You've checked everything and it all seems to be correct. Yet the problem persists.
Sleep is restless that night. You wake up at 6:54, six minutes before the alarm. As your brain regains consciousness and your eyes start to focus you remember that you still have to face the Bug of All Bugs. Laying in bed hoping for divine intervention, the alarm sounds. You slap the off button, perhaps a little too hard. Grumpily, you shower and dress, grabbing a granola bar on the way out of the door.
Back at the office you decide that you should not go it alone. You seek out Debbie, she's worked on this system and may have some ideas. She does -- but they are all things that you have already tried. She wishes you good luck and goes back to her keyboard. David points out that the module you are working on is old and full of a lot of cruft anyway. You should probably rewrite it. You tell him thanks for the advice and go back to your desk to sulk.
Debugging can be maddening. You can begin to question yourself. Your thoughts start turning negative: maybe things that I thought were true are not really true; maybe I have no idea what I am doing and should not be an engineer; I hate SystemVerilog, I hate C++, I hate Linux. A career as a barista is starting to look enticing.
Throughout your career, if you spend a lot of time with code, you will be in a similar position many times. You will encounter seemingly impenetrable bugs, get frustrated and feel like giving up. I hope this doesn't sound patronizing, but this is where learning takes place. Just like carbon under extreme heat and pressure forms diamonds, engineers under pressure become, well, better engineers.
As someone who has experienced his share of infuriating bugs here are a few things that I have learned:
The first rule of debugging: If everything looks correct, and the problem is still there, then one of those things that looks correct is not. Question everything, even those things that appear beyond reproach.
Let go of your assumptions. This is the key to debugging, particularly tough bugs. First you have to identify your assumptions and then differentiate between actual facts and assumptions. This can be quite difficult. You have to mentally step back and view the situation in a starkly objective manner. When you look at a line of suspicious code, make sure you really know what's going on. It's easy to think you know what's happening vs really knowing. Look at the order of operations, look at type promotions and conversions, look at overloading, dig into timing and event ordering, and so on. Remember the first rule of debugging.
Stay focused. While it may seem like the computer is acting like an impetuous child and is being capricious, in fact it is not. It is a highly deterministic device and does not carry personal vendettas.
Give yourself a break. When you are truly stuck and can think of no other options at the moment, step way from your computer. Go have lunch or a cup of coffee. Work on emails or another project for a while. At home put the whole thing out of your mind, spend time with your family, visit with friends, work on house projects, or even play your favorite video game -- something to engage your brain that doesn't involve work.
Do not underestimate the magic of taking a shower. I cannot tell you how many times, when faced with a difficult bug or confounding programming problem, that a useful insight popped into my brain while in the shower. As the warm water washes over you, you instinctively clear your brain and open up space for new ideas. Some people may use meditation for the same purpose.
Rest assured that you will find the source of the problem. It may take you five minutes, five hours, or five days, but you will find it. You may have an insight yourself, or you may get ideas from a colleague. You will eventually figure it out.
Bug hunting can improve your coding. Whatever turns out to be the source of the bug, you will know "never do that again." The pain of spending time with this particular bug will influence your coding style. Making your code transparent and simple (not necessarily simplistic) will make debugging much easier than trying to wade through dense, overly complex code.
When confronted with a difficult bug, give yourself the mental space to deal with it. Ignore outside pressure. Use all the tools you have at your command.
I wish you happy bug hunting!