[Dnsmasq-discuss] Dnsmasq on high load

[Анатолий Мулярский]

Hi list,

I'm using dnsmasq as a caching DNS-server for 2000+ users.
Here is my configuration file:

#bogus-priv
#domain-needed
except-interface = lo
#filterwin2k
leasefile-ro
no-ping
no-poll
user = nobody
#pid-file.
cache-size=9500
dns-forward-max=4000

Periodically I got the message:
dnsmasq[2272]: failed to send packet: Resource temporarily unavailable

Used version of Dnsmasq is 2.72
Compile time options: no-IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP
no-DHCPv6 no-scripts no-TFTP no-conntrack no-ipset no-auth no-DNSSEC
no-loop-detect

Can someone suggest me how to optimize my configuration for high load
and get rid of the above message?

[Jim Alles]

Hello,
What hardware platform and OS are you using?

Also consider
​ adjusting​
ARP
​ neighbor table size due to the number of hosts​
:
http://forum.ipfire.org/viewtopic.php?t=9293

Post by Анатолий Мулярский
Hi list,
I'm using dnsmasq as a caching DNS-server for 2000+ users.

[Анатолий Мулярский]

3.10.28-AN5 #1 SMP Fri Jan 31 10:10:49 EET 2014 x86_64 Intel(R)
Core(TM) i5-4670 CPU @ 3.40GHz GenuineIntel GNU/Linux

cat /proc/sys/net/ipv4/neigh/default/gc_thresh1
4096
cat /proc/sys/net/ipv4/neigh/default/gc_thresh2
16384
cat /proc/sys/net/ipv4/neigh/default/gc_thresh3
32768

And the router has some protection by arptables.

Post by Jim Alles
Hello,
What hardware platform and OS are you using?
Also consider
​ adjusting​
ARP
​ neighbor table size due to the number of hosts​
http://forum.ipfire.org/viewtopic.php?t=9293

[Анатолий Мулярский]

As I know, the error message means EAGAIN error.
But what the reason?
I'll try to use log-facility=LOCAL0 but I doubt whether it helps.
How else can I get more debug information?

Post by Анатолий Мулярский
3.10.28-AN5 #1 SMP Fri Jan 31 10:10:49 EET 2014 x86_64 Intel(R)
cat /proc/sys/net/ipv4/neigh/default/gc_thresh1
4096
cat /proc/sys/net/ipv4/neigh/default/gc_thresh2
16384
cat /proc/sys/net/ipv4/neigh/default/gc_thresh3
32768
And the router has some protection by arptables.

Post by Jim Alles
Hello,
What hardware platform and OS are you using?
Also consider
​ adjusting​
ARP
​ neighbor table size due to the number of hosts​
http://forum.ipfire.org/viewtopic.php?t=9293

--
Best regards
Anatoly Muliarski

[Simon Kelley]

As I know, the error message means EAGAIN error. But what the
reason?

There's a recent change to dnsmasq, which limits it to waiting for one
second for the EAGAIN error to go away.

See retry_send() in src/util.c

/* Linux kernels can return EAGAIN in perpetuity when calling
sendmsg() and the relevant interface has gone. Here we loop
retrying in EAGAIN for 1 second max, to avoid this hanging
dnsmasq. */

You might try tweaking the code below that to make it wait longer, or
not have a timeout.

The reason for the EAGAIN is likely that the send queue on the socket
if full.


Cheers,

Simon.

I'll try to use log-facility=LOCAL0 but I doubt whether it helps.
How else can I get more debug information?

Post by Анатолий Мулярский
3.10.28-AN5 #1 SMP Fri Jan 31 10:10:49 EET 2014 x86_64 Intel(R)
cat /proc/sys/net/ipv4/neigh/default/gc_thresh1 4096 cat
/proc/sys/net/ipv4/neigh/default/gc_thresh2 16384 cat
/proc/sys/net/ipv4/neigh/default/gc_thresh3 32768
And the router has some protection by arptables.

Hello, What hardware platform and OS are you using?
Also consider ​ adjusting​ ARP ​ neighbor table size due to the
http://forum.ipfire.org/viewtopic.php?t=9293

-- Best regards Anatoly Muliarski

[Анатолий Мулярский]

Thank you for the advice, I'll try it later.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

As I know, the error message means EAGAIN error. But what the
reason?

There's a recent change to dnsmasq, which limits it to waiting for one
second for the EAGAIN error to go away.
See retry_send() in src/util.c
/* Linux kernels can return EAGAIN in perpetuity when calling
sendmsg() and the relevant interface has gone. Here we loop
retrying in EAGAIN for 1 second max, to avoid this hanging
dnsmasq. */
You might try tweaking the code below that to make it wait longer, or
not have a timeout.
The reason for the EAGAIN is likely that the send queue on the socket
if full.
Cheers,
Simon.

[Анатолий Мулярский]

I've looked into the code.
retry_send() is called multiply times from different places.
BUT the static variable retries does not reset before any call in a loop.
There can be a situation where retries is 999 and for the first EAGAIN
we get an error as the retries reaches 1000 as its upper limit.
The solution I can propose is to call retry_send() with a parameter
identifying the place of calling.
In the function retry_send() we must make the check for the parameter
for matching with one from the previous invokation. If it is not true
we must reset retries counter.
The second suggestion is to make the timeout value a configurable option.
And the last - in the error message it would be comfortable to get
more info about the problem request - the client address and its
request as minimal info.
Sorry if I want too much :)
But my problem it does not solve - it only can help identifying the
problem source.

Post by Анатолий Мулярский
Thank you for the advice, I'll try it later.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

As I know, the error message means EAGAIN error. But what the
reason?

There's a recent change to dnsmasq, which limits it to waiting for one
second for the EAGAIN error to go away.
See retry_send() in src/util.c
/* Linux kernels can return EAGAIN in perpetuity when calling
sendmsg() and the relevant interface has gone. Here we loop
retrying in EAGAIN for 1 second max, to avoid this hanging
dnsmasq. */
You might try tweaking the code below that to make it wait longer, or
not have a timeout.
The reason for the EAGAIN is likely that the send queue on the socket
if full.
Cheers,
Simon.

--
Best regards
Anatoly Muliarski

[Simon Kelley]

I've looked into the code. retry_send() is called multiply times
from different places. BUT the static variable retries does not
reset before any call in a loop. There can be a situation where
retries is 999 and for the first EAGAIN we get an error as the
retries reaches 1000 as its upper limit.

A good point, the problem is that retry_send() doesn't get called when
the sendto() or other syscall succeeds, so it can't reset retries.

By changing the boilerplate code from

while (sendto(....) == -1 && retry_send());

to

while (retry_send(sendto(....));

we can make retry_send reset the retries variable when the syscall
succeeds.

int retry_send(ssize_t rc)
{
if (rc != -1)
{
retries = 0;
return 0;
}

.... other code as before ....


I just checked in this change to the git repo (and some extra checks
on the return value of close(), for good measure.)

Cheers,

Simon.

The solution I can propose is to call retry_send() with a
parameter identifying the place of calling. In the function
retry_send() we must make the check for the parameter for matching
with one from the previous invokation. If it is not true we must
reset retries counter. The second suggestion is to make the timeout
value a configurable option. And the last - in the error message it
would be comfortable to get more info about the problem request -
the client address and its request as minimal info. Sorry if I want
too much :) But my problem it does not solve - it only can help
identifying the problem source.

Post by Анатолий Мулярский
Thank you for the advice, I'll try it later.
2015-03-10 19:31 GMT+02:00, Simon Kelley

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

As I know, the error message means EAGAIN error. But what
the reason?

There's a recent change to dnsmasq, which limits it to waiting
for one second for the EAGAIN error to go away.
See retry_send() in src/util.c
/* Linux kernels can return EAGAIN in perpetuity when calling
sendmsg() and the relevant interface has gone. Here we loop
retrying in EAGAIN for 1 second max, to avoid this hanging
dnsmasq. */
You might try tweaking the code below that to make it wait
longer, or not have a timeout.
The reason for the EAGAIN is likely that the send queue on the
socket if full.
Cheers,
Simon.

-- Best regards Anatoly Muliarski

[Chen Wei]

Post by Анатолий Мулярский
I'm using dnsmasq as a caching DNS-server for 2000+ users.
cache-size=9500
dns-forward-max=4000
dnsmasq[2272]: failed to send packet: Resource temporarily unavailable
Can someone suggest me how to optimize my configuration for high load
and get rid of the above message?

Sounds like the 10k problem.

My understanding is dnsmasq was designed to be small and portable. Its
select() loop works very well for most of us, but has limitation when
comes to high concurrency connections. FD_SETSIZE along has a upper
limit of 1024 on Linux.

Assuming most dnsmasq are running on Linux, I have contemplated a
simple wrap over select and epoll so dnsmasq can use the more efficient
epoll when available. But then should dnsmasq go multi-threading? and if
the features keeps piling up, could dnsmasq still small enough for
embedded devices?

[Simon Kelley]

Post by Chen Wei

Post by Анатолий Мулярский
I'm using dnsmasq as a caching DNS-server for 2000+ users.
cache-size=9500 dns-forward-max=4000
Periodically I got the message: dnsmasq[2272]: failed to send
packet: Resource temporarily unavailable
Can someone suggest me how to optimize my configuration for high
load and get rid of the above message?

Sounds like the 10k problem.
My understanding is dnsmasq was designed to be small and portable.
Its select() loop works very well for most of us, but has
limitation when comes to high concurrency connections. FD_SETSIZE
along has a upper limit of 1024 on Linux.

I don't think that's the problem. For UDP, you can handle as many
connections as you like with two (or even one) sockets. Once you want
random source ports for upstream connections, you need more, but
dnsmasq should limit the number of those sockets to avoid
file-descriptor problems.

Post by Chen Wei
Assuming most dnsmasq are running on Linux, I have contemplated a
simple wrap over select and epoll so dnsmasq can use the more
efficient epoll when available. But then should dnsmasq go
multi-threading? and if the features keeps piling up, could dnsmasq
still small enough for embedded devices?

A judgment I have to make frequently. In this case, I don't think
multi-threading is needed or desirable.


Cheers,

Simon.

[Rick Jones]

Post by Simon Kelley

Post by Chen Wei

Post by Анатолий Мулярский
I'm using dnsmasq as a caching DNS-server for 2000+ users.
cache-size=9500 dns-forward-max=4000
Periodically I got the message: dnsmasq[2272]: failed to send
packet: Resource temporarily unavailable
Can someone suggest me how to optimize my configuration for high
load and get rid of the above message?

Sounds like the 10k problem.
My understanding is dnsmasq was designed to be small and portable.
Its select() loop works very well for most of us, but has
limitation when comes to high concurrency connections. FD_SETSIZE
along has a upper limit of 1024 on Linux.

I don't think that's the problem. For UDP, you can handle as many
connections as you like with two (or even one) sockets. Once you want
random source ports for upstream connections, you need more, but
dnsmasq should limit the number of those sockets to avoid
file-descriptor problems.

Does dnsmasq make any setsockopt(SO_SNDBUF) settings? Perhaps the
SO_SNDBUF has filled thanks to Linux's intra-stack flow-control and an
attempt to (non blocking?) send has triggered the EAGAIN?

Just guessing,

rick jones

[Simon Kelley]

Post by Rick Jones

Post by Simon Kelley

Post by Chen Wei

Post by Анатолий Мулярский
I'm using dnsmasq as a caching DNS-server for 2000+ users.
cache-size=9500 dns-forward-max=4000
Periodically I got the message: dnsmasq[2272]: failed to
send packet: Resource temporarily unavailable
Can someone suggest me how to optimize my configuration for
high load and get rid of the above message?

Sounds like the 10k problem.
My understanding is dnsmasq was designed to be small and
portable. Its select() loop works very well for most of us, but
has limitation when comes to high concurrency connections.
FD_SETSIZE along has a upper limit of 1024 on Linux.

I don't think that's the problem. For UDP, you can handle as
many connections as you like with two (or even one) sockets. Once
you want random source ports for upstream connections, you need
more, but dnsmasq should limit the number of those sockets to
avoid file-descriptor problems.

Does dnsmasq make any setsockopt(SO_SNDBUF) settings? Perhaps the
SO_SNDBUF has filled thanks to Linux's intra-stack flow-control and
an attempt to (non blocking?) send has triggered the EAGAIN?
Just guessing,

No, it doesn't change the buffer size. I think your guess may be a
good one.


I wonder some adaptive buffer-size expansion could be created?



Cheers,

Simon.

Post by Rick Jones
rick jones
_______________________________________________ Dnsmasq-discuss
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss

[Rick Jones]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Rick Jones
Does dnsmasq make any setsockopt(SO_SNDBUF) settings? Perhaps the
SO_SNDBUF has filled thanks to Linux's intra-stack flow-control and
an attempt to (non blocking?) send has triggered the EAGAIN?
Just guessing,

No, it doesn't change the buffer size. I think your guess may be a
good one.
I wonder some adaptive buffer-size expansion could be created?

Presumably, these are transient conditions right? I suppose there are a
few choices - one would be to queue the request internally and send it
again "later." Another would be to simply drop the request outright
(non-Linux stacks likely would have done so anyway and not necessarily
told the caller).

The third would be to tweak the SO_[SND|RCV]BUF explicitly. Under Linux
for that to "take" the administrator will have to have tweaked
net.core.[rw]mem_max. Otherwise Linux will silently cap any request
above that value.

As for how much buffer, I suppose for the SO_SNDBUF decision it would be
how much delay is one willing to add. Then figure how many sends could
be drained by the NIC in that length of time. If the Linux version is
new enough, it may already have fq_codel employed as the default qdisc
and there may already be a fair bit of buffering below the socket buffer.

If there are UDP receive errors being recorded (ie because dnsmasq
wasn't keeping-up with the incoming requests) then the computation would
be just how long one might reasonably expect a dnsmasq process to remain
blocked by something else and compute from there.

Lots of choices and "it depends" :)

rick

[Simon Kelley]

Post by Rick Jones

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Post by Rick Jones
Does dnsmasq make any setsockopt(SO_SNDBUF) settings? Perhaps
the SO_SNDBUF has filled thanks to Linux's intra-stack
flow-control and an attempt to (non blocking?) send has
triggered the EAGAIN?
Just guessing,

No, it doesn't change the buffer size. I think your guess may be
a good one.
I wonder some adaptive buffer-size expansion could be created?

Presumably, these are transient conditions right? I suppose there
are a few choices - one would be to queue the request internally
and send it again "later."

Not generally possible - there's one packet buffer and dnsmasq has to
either answer a request or forward it upstream before it can handle
the next - this code tries to be small in resource use.

Post by Rick Jones
Another would be to simply drop the request outright (non-Linux
stacks likely would have done so anyway and not necessarily told
the caller).

That's actually what happens, I suspect it it didn't get logged, this
would probably not have been noticed.

Post by Rick Jones
The third would be to tweak the SO_[SND|RCV]BUF explicitly. Under
Linux for that to "take" the administrator will have to have
tweaked net.core.[rw]mem_max. Otherwise Linux will silently cap
any request above that value.
As for how much buffer, I suppose for the SO_SNDBUF decision it
would be how much delay is one willing to add. Then figure how
many sends could be drained by the NIC in that length of time. If
the Linux version is new enough, it may already have fq_codel
employed as the default qdisc and there may already be a fair bit
of buffering below the socket buffer.
If there are UDP receive errors being recorded (ie because dnsmasq
wasn't keeping-up with the incoming requests) then the computation
would be just how long one might reasonably expect a dnsmasq
process to remain blocked by something else and compute from
there.
Lots of choices and "it depends" :)

It would be good to hear from the OP if the fix to the retry code I
posted has cured this.


Cheers,

Simon.

Post by Rick Jones
rick
_______________________________________________ Dnsmasq-discuss
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss