Discussion:
[Dnsmasq-discuss] dnsmasq not providing a response to client
Bill Warren
2016-09-08 04:30:21 UTC
Permalink
Hello Albert,

Thank you so much for the helpful response. This is my first time using tcpdump, so please excuse any novice mistakes.
Hello Bill,
Le Tue, 6 Sep 2016 19:17:56 -0400
Greetings from a new user of dnsmasq v.2.76 on FreeBSD v.10.3
dnsmasq is receiving queries and obtaining responses (confirmed in
--no-daemon mode).
Rather than paraphrasing the dnsmasq output, can you copy-paste it,
including [a reasonable amount of] the lines which you think are
irrelevant? I'm asking this because in your description, you don't
indicate what dnsmasq says about the response once it got it from the
upstream (I don't think it discards it, but hey, troubleshooting is
about checking what you don't think can go wrong).
Here is the output from dnsmasq during the client’s dig query. It was exactly the same when querying from the server (tcpdump later)
/usr/local/sbin/dnsmasq --no-daemon -x /var/run/dnsmasq.pid -C /usr/local/etc/dnsmasq.conf
dnsmasq: started, version 2.76 cachesize 150
dnsmasq: compile time options: IPv6 GNU-getopt no-DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect no-inotify
dnsmasq: reading /etc/resolv.conf
dnsmasq: using nameserver 8.8.8.8#53
dnsmasq: using nameserver 8.8.4.4#53
dnsmasq: read /etc/hosts - 2 addresses
dnsmasq: query[A] www.google.com from 192.168.1.161
dnsmasq: forwarded www.google.com to 8.8.8.8
dnsmasq: forwarded www.google.com to 8.8.4.4
dnsmasq: reply www.google.com is 216.58.219.196
dnsmasq: query[A] www.google.com from 192.168.1.161
dnsmasq: cached www.google.com is 216.58.219.196
dnsmasq: query[A] www.google.com from 192.168.1.161
dnsmasq: cached www.google.com is 216.58.219.196
However, the client never receives a response ...
results in
[…]
connection timed out; no servers could be reached
I disabled the pf firewall to ensure it wasn’t filtering traffic, to
no avail.
What about the server? Can you try dig on the same machine as dnsmasq
I cannot figure out why my clients aren’t getting the response from
dnsmasq even though it received and looked-up the query.
So it affects several clients. All the more a reason to check whether
the dnsmasq server itself can dig its own dnsmasq.
I should have mentioned, I did test this with "host www.google.com 127.0.0.1" on the dnsmasq server, with the same results. I do show the separate tcpdump output below.
Any suggestions would be greatly appreciated! I stumbled onto
dnsmasq and think it will be the perfect solution … once I get it
working properly.
In addition to trying dig on the server itself, I also suggest doing a
tcpdump on the server machine's interface while doing the dig, in order
to cross-check whether the server process physically sends the response
out.
Below is the output from “sudo tcpdump host 192.168.1.14” on the server (mostly). I needed the host parameter because dnsmasq is in a FreeBSD jail, but I couldn’t run tcpdump from within the jail so I ran it outside the jail. 192.168.1.14 is the jailed dnsmasq server.


tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 65535 bytes
00:26:13.418521 IP 192.168.1.14.33761 > google-public-dns-a.google.com.domain: 8349+ A? www.google.com. (32)
00:26:13.418538 IP 192.168.1.14.33761 > google-public-dns-b.google.com.domain: 8349+ A? www.google.com. (32)
00:26:13.529979 ARP, Request who-has 192.168.1.14 tell 192.168.1.1, length 46
00:26:13.529983 ARP, Reply 192.168.1.14 is-at d0:50:99:1b:92:f6 (oui Unknown), length 28
00:26:13.530774 IP google-public-dns-a.google.com.domain > 192.168.1.14.33761: 8349 1/0/0 A 172.217.2.4 (48)
00:26:13.530871 IP google-public-dns-b.google.com.domain > 192.168.1.14.33761: 8349 1/0/0 A 172.217.5.4 (48)
Then, same with digging from a client, but running two tcpdumps: one on
the server's physical interface, and one on the client's physical
interface.
Below is the tcpdump output from the server while querying from dig on client IP 192.168.1.161.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:43:21.228911 ARP, Request who-has 192.168.1.14 tell 192.168.1.161, length 46
23:43:21.228916 ARP, Reply 192.168.1.14 is-at d0:50:99:1b:92:f6 (oui Unknown), length 28
23:43:21.230111 IP 192.168.1.161.54740 > 192.168.1.14.domain: 18506+ A? www.google.com. (32)
23:43:21.230247 IP 192.168.1.14.8121 > google-public-dns-a.google.com.domain: 22525+ A? www.google.com. (32)
23:43:21.230260 IP 192.168.1.14.8121 > google-public-dns-b.google.com.domain: 22525+ A? www.google.com. (32)
23:43:21.331268 ARP, Request who-has 192.168.1.14 tell 192.168.1.1, length 46
23:43:21.331272 ARP, Reply 192.168.1.14 is-at d0:50:99:1b:92:f6 (oui Unknown), length 28
23:43:21.332149 IP google-public-dns-b.google.com.domain > 192.168.1.14.8121: 22525 1/0/0 A 216.58.219.196 (48)
23:43:21.332239 IP google-public-dns-a.google.com.domain > 192.168.1.14.8121: 22525 1/0/0 A 216.58.219.228 (48)
23:43:26.205531 IP 192.168.1.161.54740 > 192.168.1.14.domain: 18506+ A? www.google.com. (32)
23:43:29.839769 IP 192.168.1.1.37858 > 192.168.1.14.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
23:43:29.849956 IP 192.168.1.1.36023 > 192.168.1.14.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
23:43:31.207126 IP 192.168.1.161.54740 > 192.168.1.14.domain: 18506+ A? www.google.com. (32)
23:43:40.539728 IP 192.168.1.1.49200 > 192.168.1.14.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
23:43:40.549917 IP 192.168.1.1.46085 > 192.168.1.14.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
^C
15 packets captured
308 packets received by filter
0 packets dropped by kernel


Below is the output from the client on 192.168.1.161. admittedly, it included a lot of background chatter from other applications, but I was careful to exclude only truly unrelated transactions.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:11:00.193420 ARP, Request who-has 192.168.1.14 tell 192.168.1.161, length 28
00:11:00.277651 ARP, Reply 192.168.1.14 is-at f4:f2:6d:a1:34:6a (oui Unknown), length 46
00:11:00.277686 IP 192.168.1.161.58811 > 192.168.1.14.domain: 10137+ A? www.google.com. (32)
00:11:05.197627 IP 192.168.1.161.58811 > 192.168.1.14.domain: 10137+ A? www.google.com. (32)
00:11:10.198629 IP 192.168.1.161.58811 > 192.168.1.14.domain: 10137+ A? www.google.com. (32)

230 packets captured
280 packets received by filter
0 packets dropped by kernel
(Ideally, you should either copy-paste tcpdump output here if it's
short enough (but complete enough!), or write the dumps to files through
option -w or stdout redirection and make the files available somewhere,
providing just the URLs.)
Amicalement,
--
Albert.
Warm regards,
Bill
Bill Warren
2016-09-09 20:10:35 UTC
Permalink
Hi Albert,

I tried installing dnsmasq in a virtualized, fresh FreeBSD installation ... and it is working. I will go through my hardening configurations to see what, if anything, I can isolate as the cause.

to be continued …
Post by Bill Warren
Hello Albert,
Thank you so much for the helpful response. This is my first time using tcpdump, so please excuse any novice mistakes.
Hello Bill,
Le Tue, 6 Sep 2016 19:17:56 -0400
Greetings from a new user of dnsmasq v.2.76 on FreeBSD v.10.3
dnsmasq is receiving queries and obtaining responses (confirmed in
--no-daemon mode).
Rather than paraphrasing the dnsmasq output, can you copy-paste it,
including [a reasonable amount of] the lines which you think are
irrelevant? I'm asking this because in your description, you don't
indicate what dnsmasq says about the response once it got it from the
upstream (I don't think it discards it, but hey, troubleshooting is
about checking what you don't think can go wrong).
Here is the output from dnsmasq during the client’s dig query. It was exactly the same when querying from the server (tcpdump later)
/usr/local/sbin/dnsmasq --no-daemon -x /var/run/dnsmasq.pid -C /usr/local/etc/dnsmasq.conf
dnsmasq: started, version 2.76 cachesize 150
dnsmasq: compile time options: IPv6 GNU-getopt no-DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect no-inotify
dnsmasq: reading /etc/resolv.conf
dnsmasq: using nameserver 8.8.8.8#53
dnsmasq: using nameserver 8.8.4.4#53
dnsmasq: read /etc/hosts - 2 addresses
dnsmasq: query[A] www.google.com from 192.168.1.161
dnsmasq: forwarded www.google.com to 8.8.8.8
dnsmasq: forwarded www.google.com to 8.8.4.4
dnsmasq: reply www.google.com is 216.58.219.196
dnsmasq: query[A] www.google.com from 192.168.1.161
dnsmasq: cached www.google.com is 216.58.219.196
dnsmasq: query[A] www.google.com from 192.168.1.161
dnsmasq: cached www.google.com is 216.58.219.196
However, the client never receives a response ...
results in
[…]
connection timed out; no servers could be reached
I disabled the pf firewall to ensure it wasn’t filtering traffic, to
no avail.
What about the server? Can you try dig on the same machine as dnsmasq
I cannot figure out why my clients aren’t getting the response from
dnsmasq even though it received and looked-up the query.
So it affects several clients. All the more a reason to check whether
the dnsmasq server itself can dig its own dnsmasq.
I should have mentioned, I did test this with "host www.google.com 127.0.0.1" on the dnsmasq server, with the same results. I do show the separate tcpdump output below.
Any suggestions would be greatly appreciated! I stumbled onto
dnsmasq and think it will be the perfect solution … once I get it
working properly.
In addition to trying dig on the server itself, I also suggest doing a
tcpdump on the server machine's interface while doing the dig, in order
to cross-check whether the server process physically sends the response
out.
Below is the output from “sudo tcpdump host 192.168.1.14” on the server (mostly). I needed the host parameter because dnsmasq is in a FreeBSD jail, but I couldn’t run tcpdump from within the jail so I ran it outside the jail. 192.168.1.14 is the jailed dnsmasq server.
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 65535 bytes
00:26:13.418521 IP 192.168.1.14.33761 > google-public-dns-a.google.com.domain: 8349+ A? www.google.com. (32)
00:26:13.418538 IP 192.168.1.14.33761 > google-public-dns-b.google.com.domain: 8349+ A? www.google.com. (32)
00:26:13.529979 ARP, Request who-has 192.168.1.14 tell 192.168.1.1, length 46
00:26:13.529983 ARP, Reply 192.168.1.14 is-at d0:50:99:1b:92:f6 (oui Unknown), length 28
00:26:13.530774 IP google-public-dns-a.google.com.domain > 192.168.1.14.33761: 8349 1/0/0 A 172.217.2.4 (48)
00:26:13.530871 IP google-public-dns-b.google.com.domain > 192.168.1.14.33761: 8349 1/0/0 A 172.217.5.4 (48)
Then, same with digging from a client, but running two tcpdumps: one on
the server's physical interface, and one on the client's physical
interface.
Below is the tcpdump output from the server while querying from dig on client IP 192.168.1.161.
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:43:21.228911 ARP, Request who-has 192.168.1.14 tell 192.168.1.161, length 46
23:43:21.228916 ARP, Reply 192.168.1.14 is-at d0:50:99:1b:92:f6 (oui Unknown), length 28
23:43:21.230111 IP 192.168.1.161.54740 > 192.168.1.14.domain: 18506+ A? www.google.com. (32)
23:43:21.230247 IP 192.168.1.14.8121 > google-public-dns-a.google.com.domain: 22525+ A? www.google.com. (32)
23:43:21.230260 IP 192.168.1.14.8121 > google-public-dns-b.google.com.domain: 22525+ A? www.google.com. (32)
23:43:21.331268 ARP, Request who-has 192.168.1.14 tell 192.168.1.1, length 46
23:43:21.331272 ARP, Reply 192.168.1.14 is-at d0:50:99:1b:92:f6 (oui Unknown), length 28
23:43:21.332149 IP google-public-dns-b.google.com.domain > 192.168.1.14.8121: 22525 1/0/0 A 216.58.219.196 (48)
23:43:21.332239 IP google-public-dns-a.google.com.domain > 192.168.1.14.8121: 22525 1/0/0 A 216.58.219.228 (48)
23:43:26.205531 IP 192.168.1.161.54740 > 192.168.1.14.domain: 18506+ A? www.google.com. (32)
23:43:29.839769 IP 192.168.1.1.37858 > 192.168.1.14.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
23:43:29.849956 IP 192.168.1.1.36023 > 192.168.1.14.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
23:43:31.207126 IP 192.168.1.161.54740 > 192.168.1.14.domain: 18506+ A? www.google.com. (32)
23:43:40.539728 IP 192.168.1.1.49200 > 192.168.1.14.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
23:43:40.549917 IP 192.168.1.1.46085 > 192.168.1.14.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
^C
15 packets captured
308 packets received by filter
0 packets dropped by kernel
Below is the output from the client on 192.168.1.161. admittedly, it included a lot of background chatter from other applications, but I was careful to exclude only truly unrelated transactions.
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:11:00.193420 ARP, Request who-has 192.168.1.14 tell 192.168.1.161, length 28
00:11:00.277651 ARP, Reply 192.168.1.14 is-at f4:f2:6d:a1:34:6a (oui Unknown), length 46
00:11:00.277686 IP 192.168.1.161.58811 > 192.168.1.14.domain: 10137+ A? www.google.com. (32)
00:11:05.197627 IP 192.168.1.161.58811 > 192.168.1.14.domain: 10137+ A? www.google.com. (32)
00:11:10.198629 IP 192.168.1.161.58811 > 192.168.1.14.domain: 10137+ A? www.google.com. (32)
230 packets captured
280 packets received by filter
0 packets dropped by kernel
(Ideally, you should either copy-paste tcpdump output here if it's
short enough (but complete enough!), or write the dumps to files through
option -w or stdout redirection and make the files available somewhere,
providing just the URLs.)
Amicalement,
--
Albert.
Warm regards,
Bill
Albert ARIBAUD
2016-09-09 20:39:17 UTC
Permalink
Hi Bill,

Le Fri, 9 Sep 2016 16:10:35 -0400
Post by Bill Warren
Hi Albert,
I tried installing dnsmasq in a virtualized, fresh FreeBSD
installation ... and it is working. I will go through my hardening
configurations to see what, if anything, I can isolate as the cause.
I would have said as much from reading the second tcpdump, which shows
the answer from google to the dnsmasq server host (...1.14) but not the
answer from the server host to the original client. I bet the iptables
layer drops the packet for some reason.
Post by Bill Warren
to be continued …
Let us know when you find out.

Amicalement,
--
Albert.
Bill Warren
2016-09-09 23:29:10 UTC
Permalink
Hi Albert,

My issues were caused by running dnsmasq in a FreeBSD jail. Basic jails (using iocage as the jail manager, at least) use shared IP networking that is not a complete network stack. All other services I host inside jails work fine, but apparently I will need to change to use VNET/VIMAGE networking for the jails to allow dnsmasq to respond properly.

Sorry for the false alarm, since this is not a dnsmasq issue. I did learn a good amount, though :-)

Best regards,

Bill
Post by Albert ARIBAUD
Hi Bill,
Le Fri, 9 Sep 2016 16:10:35 -0400
Post by Bill Warren
Hi Albert,
I tried installing dnsmasq in a virtualized, fresh FreeBSD
installation ... and it is working. I will go through my hardening
configurations to see what, if anything, I can isolate as the cause.
I would have said as much from reading the second tcpdump, which shows
the answer from google to the dnsmasq server host (...1.14) but not the
answer from the server host to the original client. I bet the iptables
layer drops the packet for some reason.
Post by Bill Warren
to be continued …
Let us know when you find out.
Amicalement,
--
Albert.
Loading...