<div dir="ltr">Nope, no evidence that the process is dying, the same pid's persist across the collapse events. Memory footprints are in the ~2meg range.<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Fri, Mar 28, 2014 at 3:06 AM, Russell Senior <span dir="ltr"><<a href="mailto:russell@personaltelco.net" target="_blank">russell@personaltelco.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>FWIW, the ipv6 routing tables on the "leaf" nodes are quite short, with mostly just a default route pointing at the central server, when olsrd is working. When the central server has the route collapse, the default route on the "leaf" nodes disappears.<br>
<br></div>I am thinking about memory exhaustion, maybe something his helpfully killing it off when the size becomes "too large" ... /me goes to look for evidence of that.<br></div><div class="HOEnZb"><div class="h5">
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On Fri, Mar 28, 2014 at 3:03 AM, Russell Senior <span dir="ltr"><<a href="mailto:russell@personaltelco.net" target="_blank">russell@personaltelco.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">The are single hop from the central server, which is the table I've been posting.<br></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Mar 28, 2014 at 3:01 AM, Henning Rogge <span dir="ltr"><<a href="mailto:hrogge@gmail.com" target="_blank">hrogge@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">What?<br>
<br>
but your routing tables only contains "ETX 1.0" paths... which means<br>
they are single hop!<br>
<br>
Henning<br>
<br>
On Fri, Mar 28, 2014 at 11:00 AM, Russell Senior<br>
<div><div><<a href="mailto:russell@personaltelco.net" target="_blank">russell@personaltelco.net</a>> wrote:<br>
> Without the ipv6 olsrd, the nodes can't route to each other, it seems. I<br>
> picked two I had turned off, and tried ping6'ing between them and got 100%<br>
> packet loss.<br>
><br>
><br>
> On Fri, Mar 28, 2014 at 2:54 AM, Henning Rogge <<a href="mailto:hrogge@gmail.com" target="_blank">hrogge@gmail.com</a>> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>> as far as I can see each "leaf" node can see each other leaf node over<br>
>> the OpenVPN, right?<br>
>><br>
>> So you are only using Olsrd to distribute HNAs?<br>
>><br>
>> Henning Rogge<br>
>><br>
>> On Fri, Mar 28, 2014 at 10:48 AM, Russell Senior<br>
>> <<a href="mailto:russell@personaltelco.net" target="_blank">russell@personaltelco.net</a>> wrote:<br>
>> > The central server, ::407, is running OpenVPN in server mode. The<br>
>> > "leaf"<br>
>> > nodes all connect to it via OpenVPN client mode with a tap interface.<br>
>> > We<br>
>> > statically provision the IPv6 addresses on the vpn.<br>
>> ><br>
>> > And yes, the OpenVPN links are still active. We are running an IPv4<br>
>> > instance of olsrd (same version) in parallel and those routes (to the<br>
>> > very<br>
>> > same devices) are not affected.<br>
>> ><br>
>> > We see the problem when particular (though varying) nodes olsrd ipv6<br>
>> > instances are started/stopped. Sometimes the nodes are running 0.6.6.1,<br>
>> > and<br>
>> > sometimes 0.6.4. It doesn't seem to be specific. The central server is<br>
>> > running 0.6.6.1 now, but we saw the same thing earlier (which is why I<br>
>> > upgraded) on 0.6.4.<br>
>> ><br>
>> > One other potential clue (it doesn't make very much sense, because I<br>
>> > know<br>
>> > there are much bigger networks than ours), I've never seen more than 186<br>
>> > ipv6 routes on ::407. We seem to see the problem when we try to exceed<br>
>> > that. I'm going to try to confirm that.<br>
>> ><br>
>> ><br>
>> > On Fri, Mar 28, 2014 at 2:34 AM, Henning Rogge <<a href="mailto:hrogge@gmail.com" target="_blank">hrogge@gmail.com</a>> wrote:<br>
>> >><br>
>> >> Hi,<br>
>> >><br>
>> >> I must admit that I am not convinced that its an Olsrd bug what we are<br>
>> >> seeing...<br>
>> >><br>
>> >> If I see it correctly Olsrd is running over the VPN interface<br>
>> >> connection (interface name "vpn"), right?<br>
>> >><br>
>> >> Is the VPN connection between the nodes still active during the route<br>
>> >> loss? Most of the nodes seem to have direct connections and the "30<br>
>> >> seconds until recovery" sounds like an ETX value slowly going down and<br>
>> >> then dropping the link.<br>
>> >><br>
>> >> Henning<br>
>> >><br>
>> >> On Fri, Mar 28, 2014 at 10:11 AM, Saverio Proto <<a href="mailto:zioproto@gmail.com" target="_blank">zioproto@gmail.com</a>><br>
>> >> wrote:<br>
>> >> > Hello Russel,<br>
>> >> ><br>
>> >> > looking at this:<br>
>> >> > <a href="https://personaltelco.net/~russell/olsrd/olsrd-routes-before.txt" target="_blank">https://personaltelco.net/~russell/olsrd/olsrd-routes-before.txt</a><br>
>> >> > <a href="https://personaltelco.net/~russell/olsrd/olsrd-routes-during.txt" target="_blank">https://personaltelco.net/~russell/olsrd/olsrd-routes-during.txt</a><br>
>> >> > <a href="https://personaltelco.net/~russell/olsrd/olsrd-routes-after.txt" target="_blank">https://personaltelco.net/~russell/olsrd/olsrd-routes-after.txt</a><br>
>> >> ><br>
>> >> > it looks like IPv6 routes are removed from the olsrd database. So I<br>
>> >> > is<br>
>> >> > actually the olsrd daemon involved.<br>
>> >> ><br>
>> >> > do you know if there is a previous stable version of olsrd where this<br>
>> >> > bug/behaviour is not present ?<br>
>> >> ><br>
>> >> > In my opinion the fastest way to track the bug is to try different<br>
>> >> > versions of olsrd with "git bisect" method.<br>
>> >> ><br>
>> >> > The first step is to tell us if there is a version of olsrd that is<br>
>> >> > not affected by this problem.<br>
>> >> ><br>
>> >> > thanks<br>
>> >> ><br>
>> >> > I cc: olsrd-dev<br>
>> >> ><br>
>> >> > Saverio<br>
>> >> ><br>
>> >> ><br>
>> >> > 2014-03-27 10:37 GMT+01:00 Russell Senior<br>
>> >> > <<a href="mailto:russell@personaltelco.net" target="_blank">russell@personaltelco.net</a>>:<br>
>> >> >>>>>>> "Henning" == Henning Rogge <<a href="mailto:henning.rogge@fkie.fraunhofer.de" target="_blank">henning.rogge@fkie.fraunhofer.de</a>><br>
>> >> >>>>>>> writes:<br>
>> >> >><br>
>> >> >> Henning> On 03/26/2014 07:41 PM, Russell Senior wrote:<br>
>> >> >>>> Anybody get a chance to look at the strace? I see a:<br>
>> >> >><br>
>> >> >> Henning> strace and packet dumps are much too lowlevel to directly<br>
>> >> >> Henning> hunt problems like this. Thats why Saverios question about<br>
>> >> >> Henning> txtinfo good, because it gives you a much more high-level<br>
>> >> >> Henning> view on what is going on.<br>
>> >> >><br>
>> >> >> I had not installed the modules previously, so that interface wasn't<br>
>> >> >> immediately available. It is now.<br>
>> >> >><br>
>> >> >> [...]<br>
>> >> >><br>
>> >> >> Henning> Okay, lets get back to the high-level view.<br>
>> >> >><br>
>> >> >> Henning> To interpret the events you described we need a list of<br>
>> >> >> Henning> nodes, with their interface IPs and the connectivity<br>
>> >> >> between<br>
>> >> >> Henning> them.<br>
>> >> >><br>
>> >> >> Here is the list of neighbors of 2001:470:e962::407. The addresses<br>
>> >> >> listed are on the public wifi. The OpenVPN addresses of each node<br>
>> >> >> are<br>
>> >> >> a permutation, e.g. if the public wifi addr is<br>
>> >> >> 2001:470:e962:wxyz::1,<br>
>> >> >> then the OpenVPN address of the node is 2001:470:e962::wxyz.<br>
>> >> >><br>
>> >> >> None of the nodes connect directly, everything goes through ::407.<br>
>> >> >><br>
>> >> >> From curl -6 http://localhost:$port/neighbors<br>
>> >> >><br>
>> >> >> <a href="https://personaltelco.net/~russell/olsrd/olsrd-neighbors.txt" target="_blank">https://personaltelco.net/~russell/olsrd/olsrd-neighbors.txt</a><br>
>> >> >><br>
>> >> >> Henning> I am also a bit worried about your usage of bridges<br>
>> >> >> Henning> connected to mesh interfaces. Normally you should no<br>
>> >> >> bridge<br>
>> >> >> Henning> any interface that OLSR uses for meshing. Mixing routing<br>
>> >> >> Henning> (L3) and bridging (L2) can go wrong in very creative ways.<br>
>> >> >><br>
>> >> >> I don't understand how the bridges could be a problem in this case.<br>
>> >> >> This is a hub and spoke topology. One openvpn server in the middle,<br>
>> >> >> nodes at the edges. None of the nodes interconnect otherwise. Olsr<br>
>> >> >> is broadcast on the wifi in case there are any olsrd devices nearby,<br>
>> >> >> but, again, there is no overlap in the wifi coverage (and if there<br>
>> >> >> were physically, they are on different SSIDs and wouldn't overlap<br>
>> >> >> logically).<br>
>> >> >><br>
>> >> >> Can you explain more about what in particularly would make you<br>
>> >> >> worry?<br>
>> >> >> This configuration has been stable for us on ipv4 for years and also<br>
>> >> >> on ipv6 until very recently, since late 2012 at least. So, I<br>
>> >> >> suspect<br>
>> >> >> a bug. Somewhere.<br>
>> >> >><br>
>> >> >> Henning> Txtinfo output would be good (especially /route) would be<br>
>> >> >> Henning> good to see... before the problem, during the problem and<br>
>> >> >> Henning> after the recovery.<br>
>> >> >><br>
>> >> >> I'm using curl -6 http://localhost:$port/routes to get the following<br>
>> >> >> data, before, during and after turning on an ipv6 olsrd on a<br>
>> >> >> particular node (2001:470:e962:11c1::1).<br>
>> >> >><br>
>> >> >> <a href="https://personaltelco.net/~russell/olsrd/olsrd-routes-before.txt" target="_blank">https://personaltelco.net/~russell/olsrd/olsrd-routes-before.txt</a><br>
>> >> >> <a href="https://personaltelco.net/~russell/olsrd/olsrd-routes-during.txt" target="_blank">https://personaltelco.net/~russell/olsrd/olsrd-routes-during.txt</a><br>
>> >> >> <a href="https://personaltelco.net/~russell/olsrd/olsrd-routes-after.txt" target="_blank">https://personaltelco.net/~russell/olsrd/olsrd-routes-after.txt</a><br>
>> >> >><br>
>> >> >> Henning> It would also help if you can reduce the number of nodes<br>
>> >> >> Henning> while still replicating the problem to a minimum.<br>
>> >> >><br>
>> >> >> I don't have that level of control, unfortunately. When I notice<br>
>> >> >> that<br>
>> >> >> the ipv6 routes have collapsed, I pick a likely seeming node (maybe<br>
>> >> >> because it had been plugged in recently) and turn off ipv6 olsrd,<br>
>> >> >> and<br>
>> >> >> over 30-60 seconds, magically the routes all come back. My luck in<br>
>> >> >> guessing the right node to turn off is a little bit "too good", if<br>
>> >> >> you<br>
>> >> >> know what I mean, so that I am not sure there is anything<br>
>> >> >> particularly<br>
>> >> >> unique about the node I choose. But, nevertheless, turning it off<br>
>> >> >> seems to help, generally.<br>
>> >> >><br>
>> >> >> FWIW, I'm including olsrd versions here. The central machine ::407<br>
>> >> >> is<br>
>> >> >> running 0.6.6.1, compiled from the tarball. The nodes have the<br>
>> >> >> following versions, all built from openwrt routing feed sources.<br>
>> >> >><br>
>> >> >><br>
>> >> >> <a href="https://personaltelco.net/~russell/olsrd/olsrd-versions-by-node.txt" target="_blank">https://personaltelco.net/~russell/olsrd/olsrd-versions-by-node.txt</a><br>
>> >> >><br>
>> >> >> Here is a table listing the frequency of each openwrt version:<br>
>> >> >><br>
>> >> >> 1 0.6.3-3<br>
>> >> >> 33 0.6.4-1<br>
>> >> >> 1 0.6.5.1-1<br>
>> >> >> 1 0.6.5.1-2<br>
>> >> >> 7 0.6.5.2-1<br>
>> >> >> 1 0.6.5.3-1<br>
>> >> >> 2 0.6.5.4-1<br>
>> >> >> 2 0.6.6-2<br>
>> >> >> 7 0.6.6-3<br>
>> >> >> 11 0.6.6.1-1<br>
>> >> >><br>
>> >> >><br>
>> >> >> --<br>
>> >> >> Russell Senior, President<br>
>> >> >> <a href="mailto:russell@personaltelco.net" target="_blank">russell@personaltelco.net</a><br>
>> >> >><br>
>> >> >> --<br>
>> >> >> Olsr-users mailing list<br>
>> >> >> <a href="mailto:Olsr-users@lists.olsr.org" target="_blank">Olsr-users@lists.olsr.org</a><br>
>> >> >> <a href="https://lists.olsr.org/mailman/listinfo/olsr-users" target="_blank">https://lists.olsr.org/mailman/listinfo/olsr-users</a><br>
>> >> ><br>
>> >> > --<br>
>> >> > Olsr-dev mailing list<br>
>> >> > <a href="mailto:Olsr-dev@lists.olsr.org" target="_blank">Olsr-dev@lists.olsr.org</a><br>
>> >> > <a href="https://lists.olsr.org/mailman/listinfo/olsr-dev" target="_blank">https://lists.olsr.org/mailman/listinfo/olsr-dev</a><br>
>> ><br>
>> ><br>
><br>
><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>