Discussion:
[Dnsmasq-discuss] query-port option causes DNS error code 5(REFUSED)
Fred Douglas
2018-04-06 20:08:38 UTC
Permalink
I would like dnsmasq to stick to a single source port for its requests, so
that I can differentiate them from other DNS requests going out the same
interface.

The query-port option works as advertised when set to 0 (i.e. picks a
single random port and sticks to it). Any other value, however - below
1024, a little above 1024, way up in the 50000s - causes dnsmasq to respond
to all queries with a "REFUSED" (DNS error code 5).

My dnsmasq.conf is empty other than query-port, and I haven't made any
other weird changes to the system that should be relevant. This is Debian's
current [2.76+whatever security patches] version of dnsmasq.

Does anyone else get this behavior?

Fred
Simon Kelley
2018-04-07 08:29:36 UTC
Permalink
Just tried a simple test, and didn't see the same behaviour.

Use log-queries to check that the process is really failing in dnsmasq,
ie the problem is not REFUSED answers from upstream. A REFUSED answer
from dnsmasq only occurs if either there are no possible upstream server
to forward to, or if attempts to send UDP packets to all upstream
servers fail immediately, at kernel level. You're not accidentally
blocking packets from you special port, are you?


Cheers,

Simon.
Post by Fred Douglas
I would like dnsmasq to stick to a single source port for its requests,
so that I can differentiate them from other DNS requests going out the
same interface.
The query-port option works as advertised when set to 0 (i.e. picks a
single random port and sticks to it). Any other value, however - below
1024, a little above 1024, way up in the 50000s - causes dnsmasq to
respond to all queries with a "REFUSED" (DNS error code 5).
My dnsmasq.conf is empty other than query-port, and I haven't made any
other weird changes to the system that should be relevant. This is
Debian's current [2.76+whatever security patches] version of dnsmasq.
Does anyone else get this behavior?
Fred
_______________________________________________
Dnsmasq-discuss mailing list
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Fred Douglas
2018-04-07 13:47:51 UTC
Permalink
Thanks for the explanation of REFUSED's meaning! I bet it's that the
UDP sends are outright failing; I suspect that something is going
wrong with the bind at program start. I'll take a look at the logs and
report back on Monday.


For now, though, I can pretty confidently say I'm not accidentally
blocking the packets. All of my iptables rules are either for TCP, not
for the interface that goes to the internet (eth0), or are matching
UDP ports that these experiments aren't using.


I used query-port=0, observed the unchanging source port of the
(successful) resolutions, restarted dnsmasq with query-port=that_port
- and got the error. Even if I was getting unlucky, and that attempt
and my other attempts in the ephemeral range were failing because the
port happened to be in use when dnsmasq tried to bind it, that
shouldn't be the case for the lower numbered ports I was trying. (I'm
not making any other changes in between these experiments, either,
just changing query-port in dnsmasq.conf to commented, 0, or non-0,
and then `service dnsmasq restart`.)
Post by Simon Kelley
Just tried a simple test, and didn't see the same behaviour.
Use log-queries to check that the process is really failing in dnsmasq,
ie the problem is not REFUSED answers from upstream. A REFUSED answer
from dnsmasq only occurs if either there are no possible upstream server
to forward to, or if attempts to send UDP packets to all upstream
servers fail immediately, at kernel level. You're not accidentally
blocking packets from you special port, are you?
Cheers,
Simon.
* I would like dnsmasq to stick to a single source port for its requests,
*>>* so that I can differentiate them from other DNS requests going out the
*>>* same interface.
*>> >>* The query-port option works as advertised when set to 0 (i.e. picks a
*>>* single random port and sticks to it). Any other value, however - below
*>>* 1024, a little above 1024, way up in the 50000s - causes dnsmasq to
*>>* respond to all queries with a "REFUSED" (DNS error code 5).
*>> >>* My dnsmasq.conf is empty other than query-port, and I haven't made any
*>>* other weird changes to the system that should be relevant. This is
*>>* Debian's current [2.76+whatever security patches] version of dnsmasq.
*>> >>* Does anyone else get this behavior?
*>> >>* Fred
*>> >> >>* _______________________________________________
*>>* Dnsmasq-discuss mailing list
*>>* Dnsmasq-discuss at lists.thekelleys.org.uk
<http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss>
*>>* http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
<http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss>
*>>
Simon Kelley
2018-04-07 22:29:26 UTC
Permalink
Thanks for the explanation of REFUSED's meaning! I bet it's that the UDP sends are outright failing; I suspect that something is going wrong with the bind at program start. I'll take a look at the logs and report back on Monday.
When using port-randomisation, dnsmasq has to create and bind sockets
for each upstream interaction. Once you nail the port number using
query-port, it doesn't need to do that and will create and bind a
single socket at startup which it uses thereafter. A failure of that
process should cause a fatal error and abort at start-up.
For now, though, I can pretty confidently say I'm not accidentally blocking the packets. All of my iptables rules are either for TCP, not for the interface that goes to the internet (eth0), or are matching UDP ports that these experiments aren't using.
I used query-port=0, observed the unchanging source port of the (successful) resolutions, restarted dnsmasq with query-port=that_port - and got the error. Even if I was getting unlucky, and that attempt and my other attempts in the ephemeral range were failing because the port happened to be in use when dnsmasq tried to bind it, that shouldn't be the case for the lower numbered ports I was trying. (I'm not making any other changes in between these experiments, either, just changing query-port in dnsmasq.conf to commented, 0, or non-0, and then `service dnsmasq restart`.)
running dnsmasq under strace (run dnsmasq with the -d option) would be
useful, to see exactly what system calls it's making.

You have used --log-queries to make sure this REFUSED return code isn't
coming from usptream, haven't you?


Cheers,

Simon.
Post by Simon Kelley
Just tried a simple test, and didn't see the same behaviour.
Use log-queries to check that the process is really failing in dnsmasq,
ie the problem is not REFUSED answers from upstream. A REFUSED answer
from dnsmasq only occurs if either there are no possible upstream server
to forward to, or if attempts to send UDP packets to all upstream
servers fail immediately, at kernel level. You're not accidentally
blocking packets from you special port, are you?
Cheers,
Simon.
/I would like dnsmasq to stick to a single source port for its requests, />>/so that I can differentiate them from other DNS requests going out the />>/same interface. />>//>>/The query-port option works as advertised when set to 0 (i.e. picks a />>/single random port and sticks to it). Any other value, however - below />>/1024, a little above 1024, way up in the 50000s - causes dnsmasq to />>/respond to all queries with a "REFUSED" (DNS error code 5). />>//>>/My dnsmasq.conf is empty other than query-port, and I haven't made any />>/other weird changes to the system that should be relevant. This is />>/Debian's current [2.76+whatever security patches] version of dnsmasq. />>//>>/Does anyone else get this behavior? />>//>>/Fred />>//>>//>>/_______________________________________________ />>/Dnsmasq-discuss mailing list />>/Dnsmasq-discuss at lists.thekelleys.org.uk
<http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss> />>/http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss />>
_______________________________________________
Dnsmasq-discuss mailing list
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Simon Kelley
2018-04-10 20:49:40 UTC
Permalink
I still can't reproduce this, but tracing through the code, I found two
different problems which might have a bearing. I just pushed the fixes
to these to git, and tagged 2.80test1

The fixes (top two at
http://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=summary) should apply
to 2.76, if that's the easiest way to test, otherwise build from the
2.80test1 tarball.

I'd be very interested to know if they help.


Cheers,

Simon.
Ok, got the output of log-queries=extra. It is indeed the bind at
reading /run/dnsmasq/resolv.conf
ignoring nameserver 8.8.8.8 - cannot make/bind socket: Address already
in use
ignoring nameserver 8.8.4.4 - cannot make/bind socket: Address already
in use
That's query-port!=0. With =0 or unset, you get
reading /run/dnsmasq/resolv.conf
using nameserver 8.8.8.8#53
using nameserver 8.8.4.4#53
"ignoring nameserver - cannot make/bind" is printed when the
allocate_sfd function fails to allocate a socket set. allocate_sfd
returns null early when !daemon->osport, which I guess is why
query-port=0 sees the same good behavior as query-port unset. So, I
would guess the problem is inĀ allocate_sfd.
dnsmasq does not exit after that error happens, and I assume sees itself
as not having access to any resolvers, causing the REFUSEDs.
Post by Fred Douglas
Thanks for the explanation of REFUSED's meaning! I bet it's that
the UDP sends are outright failing; I suspect that something is
going wrong with the bind at program start. I'll take a look at the
logs and report back on Monday.
When using port-randomisation, dnsmasq has to create and bind sockets
for each upstream interaction. Once you nail the port number using
query-port, it doesn't need to do that and will create andĀ  bind a
single socket at startup which it uses thereafter. A failure of that
process should cause a fatal error and abort at start-up.
Post by Fred Douglas
For now, though, I can pretty confidently say I'm not accidentally
blocking the packets. All of my iptables rules are either for TCP,
not for the interface that goes to the internet (eth0), or are
matching UDP ports that these experiments aren't using.
Post by Fred Douglas
I used query-port=0, observed the unchanging source port of the
(successful) resolutions, restarted dnsmasq with
query-port=that_port - and got the error. Even if I was getting
unlucky, and that attempt and my other attempts in the ephemeral
range were failing because the port happened to be in use when
dnsmasq tried to bind it, that shouldn't be the case for the lower
numbered ports I was trying. (I'm not making any other changes in
between these experiments, either, just changing query-port in
dnsmasq.conf to commented, 0, or non-0, and then `service dnsmasq
restart`.)
running dnsmasq under strace (run dnsmasq with the -d option) would be
useful, to see exactly what system calls it's making.
You have used --log-queries to make sure this REFUSED return code isn't
coming from usptream, haven't you?
Cheers,
Simon.
Post by Fred Douglas
Post by Simon Kelley
Just tried a simple test, and didn't see the same behaviour.
Use log-queries to check that the process is really failing in
dnsmasq,
Post by Fred Douglas
Post by Simon Kelley
ie the problem is not REFUSED answers from upstream. A REFUSED answer
from dnsmasq only occurs if either there are no possible upstream
server
Post by Fred Douglas
Post by Simon Kelley
to forward to, or if attempts to send UDP packets to all upstream
servers fail immediately, at kernel level. You're not accidentally
blocking packets from you special port, are you?
Cheers,
Simon.
/I would like dnsmasq to stick to a single source port for its
requests, />>/so that I can differentiate them from other DNS
requests going out the />>/same interface. />>//>>/The query-port
option works as advertised when set to 0 (i.e. picks a />>/single
random port and sticks to it). Any other value, however - below
/>>/1024, a little above 1024, way up in the 50000s - causes dnsmasq
to />>/respond to all queries with a "REFUSED" (DNS error code 5).
/>>//>>/My dnsmasq.conf is empty other than query-port, and I
haven't made any />>/other weird changes to the system that should
be relevant. This is />>/Debian's current [2.76+whatever security
patches] version of dnsmasq. />>//>>/Does anyone else get this
behavior? />>//>>/Fred
/>>//>>//>>/_______________________________________________
/>>/Dnsmasq-discuss mailing list />>/Dnsmasq-discuss at
lists.thekelleys.org.uk <http://lists.thekelleys.org.uk>
Post by Fred Douglas
<http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss>
/>>/http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss />>
Post by Fred Douglas
_______________________________________________
Dnsmasq-discuss mailing list
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
_______________________________________________
Dnsmasq-discuss mailing list
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Fred Douglas
2018-04-11 14:55:34 UTC
Permalink
Ok, I have an interesting update coming from a very different direction.
Here is this whole big post I was just about to start a new thread for,
thinking it was a separate issue. It turns out that removing query-port=0
completely solves the bad behavior described below. (I tried both with the
unpatched 2.76, and with today's head of master `make install`ed; same
result).

====================

dnsmasq is listening on 192.168.42.1, and is configured to use 127.8.8.8,
which is a properly functioning DNAT to 8.8.8.8.

This setup works perfectly when I play the role of dnsmasq with `host
example.com 127.8.8.8`. Bizarrely, dnsmasq also works..... *exactly once*.
Then it starts returning REFUSED without even attempting to contact
127.8.8.8. Restarting dnsmasq gets you your single resolution opportunity
back. dnsmasq is configured with no-resolv, server=127.8.8.8, query-port=0,
and DNSMASQ_EXCEPT=lo.

host sdfgdsfgd.google.com 192.168.42.1 from the machine running dnsmasq
succeeds only the first time each dnsmasq restart
host sdfgdsfgd.google.com 192.168.42.1 from a client machine succeeds only
the first time each dnsmasq restart
host sdfgdsfgd.google.com 127.8.8.8 from the machine running dnsmasq always
succeeds (sanity check)
host sdfgdsfgd.google.com 127.8.8.9 from the machine running dnsmasq fails
(sanity check)
host sdfgdsfgd.google.com 8.8.8.8 from a client machine always succeeds
(sanity check)
("sdfgdsfgd" means I banged on the keyboard to avoid any possible caching.
Success is an NXDOMAIN, failure is REFUSED.)

I am sure the initial success really is taking the path I want; tcpdump
shows 4 packets:
192.168.42.1:54321->192.168.42.1:53
127.0.0.1->8.8.8.8
127.8.8.8->127.0.0.1
192.168.42.1:53->192.168.42.1
For the failed resolution, tcpdump shows no traffic involving 127.8.8.8 or
8.8.8.8, just the request and REFUSED on 192.168.42.1.

In dnsmasq's logs, successes show up as a "query[A]" line followed by a
"forwarded to 127.8.8.8", and then some "reply" lines. Failures show up as
just the query[A], with nothing else.

PS: I guessed that DNSMASQ_EXCEPT in /etc/default/dnsmasq would do the same
thing as except-interface in dnsmasq.conf. However, with just
except-interface=lo, resolv.conf still gets set to 127.0.0.1, whereas
DNSMASQ_EXCEPT=lo sees it get set to the normal 8.8.8.8 8.8.4.4. Is that a
bug, or does DNSMASQ_EXCEPT have some other meaning I'm not aware of?
I built and installed 2.80test1, and I still see the same "cannot
make/bind socket: Address already in use" in the logs. I should probably
mention I've had DNSMASQ_EXCEPT=lo in /etc/default/dnsmasq most of this
time, but I also tried without it just now - no difference. This is on a
Raspberry Pi running the current Raspbian. I will throw in some logging and
try to investigate further sometime in the next few days.
Post by Simon Kelley
I still can't reproduce this, but tracing through the code, I found two
different problems which might have a bearing. I just pushed the fixes
to these to git, and tagged 2.80test1
The fixes (top two at
http://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=summary) should apply
to 2.76, if that's the easiest way to test, otherwise build from the
2.80test1 tarball.
I'd be very interested to know if they help.
Cheers,
Simon.
Ok, got the output of log-queries=extra. It is indeed the bind at
reading /run/dnsmasq/resolv.conf
ignoring nameserver 8.8.8.8 - cannot make/bind socket: Address already
in use
ignoring nameserver 8.8.4.4 - cannot make/bind socket: Address already
in use
That's query-port!=0. With =0 or unset, you get
reading /run/dnsmasq/resolv.conf
using nameserver 8.8.8.8#53
using nameserver 8.8.4.4#53
"ignoring nameserver - cannot make/bind" is printed when the
allocate_sfd function fails to allocate a socket set. allocate_sfd
returns null early when !daemon->osport, which I guess is why
query-port=0 sees the same good behavior as query-port unset. So, I
would guess the problem is in allocate_sfd.
dnsmasq does not exit after that error happens, and I assume sees itself
as not having access to any resolvers, causing the REFUSEDs.
Post by Fred Douglas
Thanks for the explanation of REFUSED's meaning! I bet it's that
the UDP sends are outright failing; I suspect that something is
going wrong with the bind at program start. I'll take a look at the
logs and report back on Monday.
When using port-randomisation, dnsmasq has to create and bind
sockets
for each upstream interaction. Once you nail the port number using
query-port, it doesn't need to do that and will create and bind a
single socket at startup which it uses thereafter. A failure of that
process should cause a fatal error and abort at start-up.
Post by Fred Douglas
For now, though, I can pretty confidently say I'm not accidentally
blocking the packets. All of my iptables rules are either for TCP,
not for the interface that goes to the internet (eth0), or are
matching UDP ports that these experiments aren't using.
Post by Fred Douglas
I used query-port=0, observed the unchanging source port of the
(successful) resolutions, restarted dnsmasq with
query-port=that_port - and got the error. Even if I was getting
unlucky, and that attempt and my other attempts in the ephemeral
range were failing because the port happened to be in use when
dnsmasq tried to bind it, that shouldn't be the case for the lower
numbered ports I was trying. (I'm not making any other changes in
between these experiments, either, just changing query-port in
dnsmasq.conf to commented, 0, or non-0, and then `service dnsmasq
restart`.)
running dnsmasq under strace (run dnsmasq with the -d option) would
be
useful, to see exactly what system calls it's making.
You have used --log-queries to make sure this REFUSED return code
isn't
coming from usptream, haven't you?
Cheers,
Simon.
Post by Fred Douglas
Post by Simon Kelley
Just tried a simple test, and didn't see the same behaviour.
Use log-queries to check that the process is really failing in
dnsmasq,
Post by Fred Douglas
Post by Simon Kelley
ie the problem is not REFUSED answers from upstream. A REFUSED
answer
Post by Fred Douglas
Post by Simon Kelley
from dnsmasq only occurs if either there are no possible upstream
server
Post by Fred Douglas
Post by Simon Kelley
to forward to, or if attempts to send UDP packets to all upstream
servers fail immediately, at kernel level. You're not
accidentally
Post by Fred Douglas
Post by Simon Kelley
blocking packets from you special port, are you?
Cheers,
Simon.
/I would like dnsmasq to stick to a single source port for its
requests, />>/so that I can differentiate them from other DNS
requests going out the />>/same interface. />>//>>/The query-port
option works as advertised when set to 0 (i.e. picks a />>/single
random port and sticks to it). Any other value, however - below
/>>/1024, a little above 1024, way up in the 50000s - causes dnsmasq
to />>/respond to all queries with a "REFUSED" (DNS error code 5).
/>>//>>/My dnsmasq.conf is empty other than query-port, and I
haven't made any />>/other weird changes to the system that should
be relevant. This is />>/Debian's current [2.76+whatever security
patches] version of dnsmasq. />>//>>/Does anyone else get this
behavior? />>//>>/Fred
/>>//>>//>>/_______________________________________________
/>>/Dnsmasq-discuss mailing list />>/Dnsmasq-discuss at
lists.thekelleys.org.uk <http://lists.thekelleys.org.uk>
Post by Fred Douglas
<http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss>
/>>/http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
/>>
Post by Fred Douglas
_______________________________________________
Dnsmasq-discuss mailing list
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
_______________________________________________
Dnsmasq-discuss mailing list
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss
Continue reading on narkive:
Loading...