As promised, here I am to write about the other bug. This one needs a bit a background to be understood. At the beginning of the current project (that means some 2 years ago) I designed a software timer for providing time services. User code that needs notification about time event may register the software timer specifying the time event. When the time event occurs the user code is called back.
Software timers are identified by a numeric id. Being an 8bit architecture with a severely constrained data memory, I picked an unsigned 8 bit number as timer id. This id indexes a private array containing the software timer parameters.
There are two kind of timers – repeating and one shot. A repeating timer repeatedly calls you back at given intervals, while an one-shot timer just calls you back once when it expires.
Repeating timers need to be explicitly canceled, but when I was designing the class, it felt like a good idea to let one-shot timer self-cancel pending their expiration.
Time passed, Software timer component being tried and tested became trusted. Then I got some mysterious bugs – code that used to work stopped being called back by timer. It took a while to discover what happened. After the one-shot timer executed its callback the user code still had some dangling references to the timer id.
Occasionally user code could inadvertently cancel the timer pointed to by the dangling reference resulting either in an assertion about a non-existing timer (turned into a reset in release code), or in a deletion of an innocent bystander timer.
I thought a while on the matter and I came to the conclusion that it is not good for the timer to self-delete. Rather is must be up to the user code to get rid of the timer. Unfortunately I cannot apply the lesson learned on the current project because the code base is too large and we are releasing, nonetheless I’ll keep a reminder for the next project.
While talking about timer callback, a colleague found another problem – if you perform timer registration and removal during the callback you are in for serious trouble. In fact the code for the timer dispatcher was about like this –

for( id = 0; id < TIMER_COUNT; ++id )
{
    if( timerHasExpired(id) )
    {
        callbackUserCode( id );
        if( timerIsOneShot( id ))
        {
            deleteTimer( id );
        }
        else
        {
            // timer is repeat
            reloadTimer( id );
        }
    }
}

This code appears good as long as nothing happens on the timers repository during callback. Consider what happens if a one-shot timer is canceled and a new timer is created in the callback code…
There are two ways to deal with this – either queue timer creation/deletion operation during the callback and execute them when the above code is completed. The other is to re-consider the current time nature after the callback, i.e. consider the timer repository “volatile” across the callback. This is our choice –

for( id = 0; id < TIMER_COUNT; ++id )
{
    if( timerHasExpired(id) )
    {
        callbackUserCode( id );
        if( timerHasExpired( id ))
        {
            if( timerIsOneShot( id ))
            {
                deleteTimer( id );
            }
            else
            {
                // timer is repeat
                reloadTimer( id );
            }
        }
    }
}

Lessons learned – 1) don’t provide classes that mixes automatic and explicit object disposal – either the user must always delete objects or the user must never delete objects. 2) critical runs are not just about interrupts, but they may occur in callback as well. Calling back code should be programmed defensively expecting that the user code may wreak havoc on the structure. Should this not be possible then a state variable declaring whether the class is in “callback execution” or “normal execution” must be provided and functions that manipulates class state have to be made aware of the state so to act directly or to defer actions at the end of “callback execution” state.

It is nearly three decades since I write bugs code and I am maturing the idea that each new bug I find should inspire a way to avoid it in the future or, at least, to reduce the chance of falling again in the same mistake. During the past weeks my team and I discovered two bugs and here is my ideas for improving the software quality.

The first bug is likely to be the longest-standing bug I had. As you may know, I am working on a proprietary field bus. We found evidence of the problem at least three years ago. Occasionally our devices lose a frame. It is not tragic – it is a frame in 250 – but, since the protocol does not account for transmission acknowledgment, it may result in wrong device behavior.

We got the assembly driver from our customer, therefore the code is granted to be correct. They use the same code in hundreds of devices and they said they never have such a bad figure in frame reception.

This bug had a chance to live so long for two reasons – first, the customer accepted our firmware with this known problem; second we were quite convinced that this problem should have appeared in their devices as well.

An event broke this impasse – first, we had big troubles in an installation and eventually, it turned out that one device has a slightly defective crystal that caused major frame dropout, wreaking havoc on all the system. This led to some investigation by the hardware guys that told me that crystal precision was very critical in device correctness. The crystal component has a tolerance of 0.5% and our device starts behaving weirdly when the frequency goes 0.4% away from the nominal frequency. This was clearly not acceptable and a solution was urged.

I went to some fruitless investigations before discovering that another firmware, mainly developed by our customer, was much more reliable than ours. After much head-scraping and head-banging against the nearest wall, I went through the aging documentation that the customer gave us to integrate the bus driver.

There I found that it was required to set a global variable, let’s call it FLEX with half of the value of a global constant.

Then I turned to the code and found an old comment of mine that went like this /* FLEX and GIZMO seem not to be used anywhere in the code, they must be private variables */

Of course, removing the comment and properly initializing the variable sent the problem the way of Dodos.

How this could have been prevented? That’s a simple shot – avoid the programmer manually setting obscure and arcane magic values in oddly named global variables. Either provide a complete setup function or let the magic value be computed at compile time.

I’ll write about the other bug, tomorrow. In the meantime, your comments are welcome, as always.

Once upon a time I was a strenuous ZX Spectrum fan, then time changed, technology evolved and, fighting back my feelings, I switched to Commodore Amiga (yes, it took me a detour over the Amstrad cpc, to get ready for change).Then I was fond of the Amiga, but time changed again, Commodore filed bankruptcy and I changed to anonymous PC clone with Windows 3.1.
Sad story, time changes and you must adapt and change as well, saying goodbye to your comfortable little home.
Now I am faced with a hard decision and it seems that it is time for change again. For the incoming project at the company I work for, I will have to lead the development team (and likely to put in some project management as well, but this is another story) and I have been already told that I don’t have to expect we are going to hire top programmer. On the contrary the company is like to get some instances of Joe Slightly Below Average junior programmer.
The project itself is ambitious (as all projects we tackle), there will be Linux, IP communication, complex system, audio and video communication, proprietary and open field bus communication, zig bee localization and the like… So for sure the team will have to find as much as possible from COTS (Components Off The Shelf) and integrate them.
Now I have to select (apparently I am still free of doing this) the language for the main application. Would have been it my project I would have chosen C++ blindly – no discussion. I am also curious about the new C++11 and this would have been a perfect occasion for getting used to the new tools in the language.
But… I have to deal with junior, possibly below the average programmers (and to make things worse, the development team will be far away from me, but this is another story as well).
From my last weekly report:

I am still undecided about the language to use for this project. There are really only two (maybe three) alternatives.

• Java. Java is quite accessible. It is likely that every college programmer has been exposed (at various degree) to Java. Java prevents you to do simple mistakes on pointers and memory management at the cost of allowing you to enter terrible pains and sufferance by mindlessly making complex mistakes. Java is mature and stable, performance could be sometimes troublesome. Integrating third part components is easy only if they are written in Java or have a Java wrapper.
• C++. C++ is unfortunately a less obvious choice, many people have it on their resume, just to pair C, but are quite lost at what is C++ today. C++ helps you strongly in avoiding the terrible pains and sufferances of complex mistakes, by letting yourself tie a knot around your neck with simple errors of pointers and memory management. Although C++ is with us since a long time, the latest standard is still young and many programmers still don’t have a clue about the previous one. Performance is not an issue. Integrating third party components should be easy.
• Python. Ok, I am not really a script language fan, but I reckon that for integration work, these could be the right tools. Write few “glue” code that just works and ignore the rest. Performance are not what you want to look at, but it is up to third party components to be quick. Though resources can be limited on an embedded system to use a scripting language as it was thought to be used.

Despite the silly wording, Java seems to be the reasonable answer, and for sure is the one preferred by management, but is it the right one?