On Thu, 2008-11-20 at 11:53 +0100, Bernd Petrovitsch wrote:
[....]
> Now that you ask;-):
> We probably have lots of bugs similar to all other serious software
> beyond the complexity of "hello world" (and I saw "hello world" with
> bugs:-). But almost all of them bite seldom enough that next no one is
> bothered enough;-)
> 
> One of these is: olsrd seg-faults "occasionally". One necessary
> condition for this seems to be an interface going up and down often
> enough and since - TTBOMK - I'm the only one who gets it quite
> "deterministic" after a few hours (or only 30 minutes) and the main

I like to add that I explicitly set some "deleted" fields to NULL to
trigger that bug as soon as possible (and had other assert()s and
run-time data checks in place to hunt down the bug).

> address is on the on the up/down interface, I suspect that that has
> something to do with the bug too (but I think I'm pretty alone with that
> suspicion).

I think I found the root cause of that (and the suspicion with the main
address was indeed wrong).
The root cause is: We walk a (linked) list of timers. Each timer has a
callback (which does what should be done at that time). And that
callback may also manipulate the timer list - e.g. cancel the timer
removing it from the list (and that is unavoidable).
That allows under certain circumstances - i.e. a removed timer is to be
fired in the same clock-tick and is e.g. the next element in the list -
that an already "officially deleted" timer fires (and seg-faults in
strange places - depending if and when the first illegal pointer is
dereferenced).
And there is no real small bugfix (as in "we add an if() somewhere") as
- at least in theory (which is enough for me) - given a timer in the
linked list: the previous and/or the current and/or the next and/or
*all* elements of that list could be deleted (so we have no chance to
keep any state information for a simple loop over the list).
We are now just thinking about a good solution - both in the "what can
we achieve if we touch that code" dimension and the "time for it needed"
dimension....

> But reading the code didn't show up anything even remotely suspicious.
> So ATM I'm (and others) are adding syslog(3) output now and then to nail
> that one when we find it sooner or later ....

Done:-)

> The short summary: IMHO the tip of the dev-repo is "stable" -  even with
> the above bug. And it is IMHO at least as stable as the "official stable
> release" version as I'm pretty convinced that the same bug is in there
> too somewhere.

For everyone else: An interface going down cancels several timers at
once - thus it was the prime candidate to trigger an quite unlikely bug.
As long as interfaces don't go up/down regularly (and IMHO and TTBOMK
only openvpn interfaces usually do this), it is pretty unlikely that the
bug will actually trigger IRL.
If you think that you run into the bug: start an olsrd with debug
symbols etc. under "gdb" (or gdbserver") and if it seg-faults and the
seg-fault is in the olsr_scheduler() function, you really had it.
For all other seg-faults: Post the output of "bt full" here for some
post-mortem inspection.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services




