[Dnsmasq-discuss] Got bad packet: bad compression pointer

[Dave Taht]

I am testing the dnsmasq-full build on current lede-project head, and
enabled dnssec. Then :

***@dancer:/# host flent-fremont.bufferbloat.net
flent-fremont.bufferbloat.net has address 23.239.20.41
flent-fremont.bufferbloat.net has IPv6 address 2600:3c01::f03c:91ff:fe50:48d4
;; Got bad packet: bad compression pointer
111 bytes
40 41 81 80 00 01 00 00 00 01 00 00 0d 66 6c 65 @A...........fle
6e 74 2d 66 72 65 6d 6f 6e 74 0b 62 75 66 66 65 nt-fremont.buffe
72 62 6c 6f 61 74 03 6e 65 74 00 00 0f 00 01 c0 rbloat.net......
1a 00 06 00 01 00 00 0e 10 00 34 06 61 72 6e 6f ..........4.arno
6c 64 02 6e 73 0a 63 6c 6f 75 64 66 6c 61 72 65 ld.ns.cloudflare
03 63 6f 6d 00 03 64 6e 73 c0 f0 78 9d d7 47 00 .com..dns..x..G.
00 27 10 00 00 09 60 00 09 3a 80 00 00 0e 10 .'....`..:.....


Filed the bug here:
https://bugs.lede-project.org/index.php?do=details&task_id=392

I see a few other references to this phrase elsewhere on the net but
did not find a resolution.

In this case I've seen this with osx sierra, and "dancer" is a pretty
recent ubuntu box. The dnssec tests on the web seem to all pass, it
just shows up with host - and not consistently. I just had it happen
one time in 4, on a recent test.

[Simon Kelley]

Host makes A, AAAA and MX queries and it's the reply to the MX that's
failing. This all works fine here (dnsmasq and host both running on
the same x86_64 host. The reply to the MX query looks like this.


; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19992
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;flent-fremont.bufferbloat.net. IN MX

;; AUTHORITY SECTION:
bufferbloat.net. 1798 IN SOA arnold.ns.cloudflare.com.
dns.cloudflare.com. 2023610183 10000 2400 604800 3600

;; Query time: 50 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Jan 16 20:27:43 GMT 2017
;; MSG SIZE rcvd: 122


Comparing the packet dump you have with the correct answer I'm seeing,
there are a few not-important differences (transaction-id,
time-to-live and SOA serial), the only other difference is the second
domain name in the SOA record, dns.cloudflare.com. That's represented
as the label "dns" and then a compression pointer pointing back to
previous instance of "cloudflare.com" in arnold.ns.cloudflare.com. The
correct pointer is c0 45, the pointer in your dump is c0 f0. (the c0
is flags, the 45 (or f0) is an offset from the start of the packet).

This packet has been through some hairy code in dnsmasq which elides
DNSSEC records (RRSIGs etc) and has to fix up the pointers thus
affected. My guess is that that's the problem, somehow, but I'm at a
loss to say why it works for me and breaks for you.

Note that if my hypothesis is correct, you'll only see the effect when
the answer comes from upstream, and not when the answer comes from the
dnsmasq cache.

If you want to try and debug this, first check that you can see the
same error just doing the MX query with a cold cache, then look at
what's happening in rrfilter() in src/rrfilter.c

The other thing that would be useful is to capture the answer from
usptream complete with the NSEC and RRSIG records that need to be
removed. When I do that, the NSEC and RRSIG records come _after_ the
SOA record, so that the compression pointer in the SOA record doesn't
need to be touched at all, if the order of the records varied, that
could expose bugs in this code.

Not an answer, but some good clues......


Cheers,

Simon.

Post by Dave Taht
I am testing the dnsmasq-full build on current lede-project head,
flent-fremont.bufferbloat.net has address 23.239.20.41
flent-fremont.bufferbloat.net has IPv6 address
2600:3c01::f03c:91ff:fe50:48d4 ;; Got bad packet: bad compression
pointer 111 bytes 40 41 81 80 00 01 00 00 00 01 00 00 0d 66 6c 65
@A...........fle 6e 74 2d 66 72 65 6d 6f 6e 74 0b 62 75 66 66 65
nt-fremont.buffe 72 62 6c 6f 61 74 03 6e 65 74 00 00 0f 00 01 c0
rbloat.net...... 1a 00 06 00 01 00 00 0e 10 00 34 06 61 72 6e 6f
..........4.arno 6c 64 02 6e 73 0a 63 6c 6f 75 64 66 6c 61 72 65
ld.ns.cloudflare 03 63 6f 6d 00 03 64 6e 73 c0 f0 78 9d d7 47 00
.com..dns..x..G. 00 27 10 00 00 09 60 00 09 3a 80 00 00 0e 10
.'....`..:.....
https://bugs.lede-project.org/index.php?do=details&task_id=392
I see a few other references to this phrase elsewhere on the net
but did not find a resolution.
In this case I've seen this with osx sierra, and "dancer" is a
pretty recent ubuntu box. The dnssec tests on the web seem to all
pass, it just shows up with host - and not consistently. I just had
it happen one time in 4, on a recent test.

[Dave Taht]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Host makes A, AAAA and MX queries and it's the reply to the MX that's
failing. This all works fine here (dnsmasq and host both running on
the same x86_64 host. The reply to the MX query looks like this.
; (1 server found)
;; global options: +cmd
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19992
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
; EDNS: version: 0, flags:; udp: 512
;flent-fremont.bufferbloat.net. IN MX
bufferbloat.net. 1798 IN SOA arnold.ns.cloudflare.com.
dns.cloudflare.com. 2023610183 10000 2400 604800 3600
;; Query time: 50 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Jan 16 20:27:43 GMT 2017
;; MSG SIZE rcvd: 122
Comparing the packet dump you have with the correct answer I'm seeing,
there are a few not-important differences (transaction-id,
time-to-live and SOA serial), the only other difference is the second
domain name in the SOA record, dns.cloudflare.com. That's represented
as the label "dns" and then a compression pointer pointing back to
previous instance of "cloudflare.com" in arnold.ns.cloudflare.com. The
correct pointer is c0 45, the pointer in your dump is c0 f0. (the c0
is flags, the 45 (or f0) is an offset from the start of the packet).
This packet has been through some hairy code in dnsmasq which elides
DNSSEC records (RRSIGs etc) and has to fix up the pointers thus
affected. My guess is that that's the problem, somehow, but I'm at a
loss to say why it works for me and breaks for you.

It's a mips box supplying dns here. Different rules for alignment?

Note that if my hypothesis is correct, you'll only see the effect when
the answer comes from upstream, and not when the answer comes from the
dnsmasq cache.
If you want to try and debug this, first check that you can see the
same error just doing the MX query with a cold cache, then look at
what's happening in rrfilter() in src/rrfilter.c
The other thing that would be useful is to capture the answer from
usptream complete with the NSEC and RRSIG records that need to be
removed. When I do that, the NSEC and RRSIG records come _after_ the
SOA record, so that the compression pointer in the SOA record doesn't
need to be touched at all, if the order of the records varied, that
could expose bugs in this code.
Not an answer, but some good clues......

Don't even know if it's over ipv4 or ipv6 at the moment. will check harder.


Great clues, thx, I'll get on it after I resolv

https://bugs.lede-project.org/index.php?do=details&task_id=388

Cheers,
Simon.

Post by Dave Taht
I am testing the dnsmasq-full build on current lede-project head,
flent-fremont.bufferbloat.net has address 23.239.20.41
flent-fremont.bufferbloat.net has IPv6 address
2600:3c01::f03c:91ff:fe50:48d4 ;; Got bad packet: bad compression
pointer 111 bytes 40 41 81 80 00 01 00 00 00 01 00 00 0d 66 6c 65
@A...........fle 6e 74 2d 66 72 65 6d 6f 6e 74 0b 62 75 66 66 65
nt-fremont.buffe 72 62 6c 6f 61 74 03 6e 65 74 00 00 0f 00 01 c0
rbloat.net...... 1a 00 06 00 01 00 00 0e 10 00 34 06 61 72 6e 6f
..........4.arno 6c 64 02 6e 73 0a 63 6c 6f 75 64 66 6c 61 72 65
ld.ns.cloudflare 03 63 6f 6d 00 03 64 6e 73 c0 f0 78 9d d7 47 00
.com..dns..x..G. 00 27 10 00 00 09 60 00 09 3a 80 00 00 0e 10
.'....`..:.....
https://bugs.lede-project.org/index.php?do=details&task_id=392
I see a few other references to this phrase elsewhere on the net
but did not find a resolution.
In this case I've seen this with osx sierra, and "dancer" is a
pretty recent ubuntu box. The dnssec tests on the web seem to all
pass, it just shows up with host - and not consistently. I just had
it happen one time in 4, on a recent test.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAEBCAAGBQJYfTb0AAoJEBXN2mrhkTWiffYP/Ariw/tDKjfy8pd6/Z2Nixc/
MW5zcQAh421uxrSIA4NNhJeymiXdiwQNC8CAbvtSPNvwE87Ed8GlnfExnYSzWpig
UvjJS1fXRF+y57e8UcqQmZEXTNtE/wdc5Rs0nqv+TpLaBMXF4nVjABmSsNOzymrw
VQdMOHIrqGx4xmeiRU2JhUXPX57uxLTH1PmJ0PxHj5RPWm9xG8kzq3h7gtzA6hMs
j/SXApIM6trC34D+zrFt/4HrsneJuzFvEiolN9N1MurrdEW8SVlr8k37hayMU48w
fZa5W6EZ3UYS2YDMYMfUlYNDC2aktQO8KWF80aQvzdvDS8LWJNkJ1vcBZFV5gEht
J0AdZdK4b9J/j1Ejxxq37D3xWRG9VKuiyqPkvNlksLkrNeoS+aV3HdBDvqHbejz+
jDXSEXDvIOQ1Lg5RKFt7dkTZPlWpZnkzJqfFeIR4TsHYypqNUOSpuG+ar+4FfDaq
dMyNYbut75GN2HFuVMpedHiSjKaCQ1Oq90Zbwy4AO4sd81uMirpOILqB/siGghY+
e1NcfNR/RZJJxpAOS5I0f/TXycB7vgfLwJ+06m1JGwD+EyTKpyVqL16JsK18OsuM
vuBGaKA0/ReUQB+wPyJ1JVFlTmztmgaZLF/ZHg6PsjxkFvsn34GIsGEWMjfekXKc
irEZUnVMT8mWUyBPZ4p2
=aDdB
-----END PGP SIGNATURE-----
_______________________________________________
Dnsmasq-discuss mailing list
http://lists.thekelleys.org.uk/mailman/listinfo/dnsmasq-discuss

[Dave Taht]

so far I can only make it happen on mips. Doesn't happen on arm.
Haven't tried harder yet.

[Simon Kelley]

I won't have access to a MIPS system 'till the weekend.

I assume you're using the git head code?

Did you manage to see a dump of the upstream reply?


Simon.

Post by Dave Taht
so far I can only make it happen on mips. Doesn't happen on arm.
Haven't tried harder yet.

[Dave Taht]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
I won't have access to a MIPS system 'till the weekend.
I assume you're using the git head code?

No. Lede-project head. package claims to be dnsmasq-2.76-6. libc is musl.

Box under test was an archer c7v2. Can go try a few other mips boxes
like the wndr3800, but I've seen it there too. The arm box (that is
working) is an linksys-1200ac. (overall it's looking like a fine
release of lede)....

Did you manage to see a dump of the upstream reply?

Not yet. I'll touch bases with you later in the week.

Simon.

Post by Dave Taht
so far I can only make it happen on mips. Doesn't happen on arm.
Haven't tried harder yet.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAEBCAAGBQJYf8aDAAoJEBXN2mrhkTWiN9UP/2E9D6j/nd3RsubzHgZSvzB/
CJPNyk32jnAqZdIa9D3DOH2L9gN5GyBiAtv4iCz5KuzDnB9twBtQWOdzde5sZWWd
4t8tSsvJkr0pRZhhRQKelF2oW0k7Y0UM4mD90ZoabX9ytQG4ceTFkHKlwwPLvvTc
Osh9RmCpX1tsJoE/y+lMpEdT+GlhOe4z2Z9FZlTN7ke/uO9nIarekSIvnxgOnyac
vrHvgnjyyEHbfr0BNaupdwZz9d/dVABYkFTDUk4dg4tn6MW99AsbD2DaL9alx8U/
MsvbFarQe/w8fJkmBJOThWkLMvpO1854XAysc8/m5ldIEwcV4Yge29nYrmDhn9kH
Evo7wbKSH4AYGskYTiWnssczu1RhQOX9jCD31gv5CVOeTY4Dt7NR3WFCsAH2RYpR
jcstckC5R1fqfKtQt9B0l2SWmmLukRcMbGM1hiJbqGrZcb++gZ2RYl80AD0iQhkD
GjLNQAUKwlDwzB7JYXX+Fn0AVvP/G4qrmYBFlcxloddtrCiNqu4icTYIAb1zv0Lo
opM+0fFcfg1PPPobTQ7FLJQR/uAO93MWZJ43Ht90YEdk6aaBCf7Ego1fU0G6TjCV
iphmOqvhs96GFfhaBMYwFxvHb1tHNDT+Xzlsvkvk+S8SKyhNOg5GJOL2Dz78vlB/
fcImILW4vRf4rIkMDZKL
=kPYg
-----END PGP SIGNATURE-----

[Dave Taht]

The offputting part of your outline of what to check for was "some
hairy pointer code". :) I'm in the middle of some crash bugs
elsewhere, and I didn't realize how fast I could get you data without
thinking about the "hairy" parts.


dnssec and dnssec-check-unsigned are enabled, and I'm using cachesize
9999 (what's the limit nowadays?)

I put packet captures of the external interface on the router (comcast
upstream) and captures taken at the client, a log, and conf file,
here:

http://www.taht.net/~d/dnssecbug/

Basically hammering on nslookup for the two internal and internal
captures there.

Hammering on "dig" later, I was unable to trigger it on A, or AAAA
requests. Was able to easily trigger it on a MX request.

flent-freemont does not exist, btw. Flent-fremont, does. It will go
boom on both.



***@dancer:~/dnssecbug# dig flent-freemont.bufferbloat.net MX
;; Got bad packet: bad compression pointer
123 bytes
a5 c9 81 a0 00 01 00 00 00 01 00 01 0e 66 6c 65 .............fle
6e 74 2d 66 72 65 65 6d 6f 6e 74 0b 62 75 66 66 nt-freemont.buff
65 72 62 6c 6f 61 74 03 6e 65 74 00 00 0f 00 01 erbloat.net.....
c0 1b 00 06 00 01 00 00 0e 10 00 34 06 61 72 6e ...........4.arn
6f 6c 64 02 6e 73 0a 63 6c 6f 75 64 66 6c 61 72 old.ns.cloudflar
65 03 63 6f 6d 00 03 64 6e 73 c0 eb 78 9d d7 47 e.com..dns..x..G
00 00 27 10 00 00 09 60 00 09 3a 80 00 00 0e 10 ..'....`..:.....
00 00 29 02 00 00 00 00 00 00 00 ..)........
***@dancer:~/dnssecbug# dig flent-freemont.bufferbloat.net MX

; <<>> DiG 9.10.3-P4-Ubuntu <<>> flent-freemont.bufferbloat.net MX
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34631
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;flent-freemont.bufferbloat.net. IN MX

;; AUTHORITY SECTION:
bufferbloat.net. 3600 IN SOA arnold.ns.cloudflare.com.
dns.cloudflare.com. 2023610183 10000 2400 604800 3600

;; Query time: 72 msec
;; SERVER: 172.26.16.1#53(172.26.16.1)
;; WHEN: Wed Jan 18 12:42:02 PST 2017
;; MSG SIZE rcvd: 123

Post by Dave Taht

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
I won't have access to a MIPS system 'till the weekend.
I assume you're using the git head code?

No. Lede-project head. package claims to be dnsmasq-2.76-6. libc is musl.
Box under test was an archer c7v2. Can go try a few other mips boxes
like the wndr3800, but I've seen it there too. The arm box (that is
working) is an linksys-1200ac. (overall it's looking like a fine
release of lede)....

Did you manage to see a dump of the upstream reply?

Not yet. I'll touch bases with you later in the week.

Simon.

Post by Dave Taht
so far I can only make it happen on mips. Doesn't happen on arm.
Haven't tried harder yet.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAEBCAAGBQJYf8aDAAoJEBXN2mrhkTWiN9UP/2E9D6j/nd3RsubzHgZSvzB/
CJPNyk32jnAqZdIa9D3DOH2L9gN5GyBiAtv4iCz5KuzDnB9twBtQWOdzde5sZWWd
4t8tSsvJkr0pRZhhRQKelF2oW0k7Y0UM4mD90ZoabX9ytQG4ceTFkHKlwwPLvvTc
Osh9RmCpX1tsJoE/y+lMpEdT+GlhOe4z2Z9FZlTN7ke/uO9nIarekSIvnxgOnyac
vrHvgnjyyEHbfr0BNaupdwZz9d/dVABYkFTDUk4dg4tn6MW99AsbD2DaL9alx8U/
MsvbFarQe/w8fJkmBJOThWkLMvpO1854XAysc8/m5ldIEwcV4Yge29nYrmDhn9kH
Evo7wbKSH4AYGskYTiWnssczu1RhQOX9jCD31gv5CVOeTY4Dt7NR3WFCsAH2RYpR
jcstckC5R1fqfKtQt9B0l2SWmmLukRcMbGM1hiJbqGrZcb++gZ2RYl80AD0iQhkD
GjLNQAUKwlDwzB7JYXX+Fn0AVvP/G4qrmYBFlcxloddtrCiNqu4icTYIAb1zv0Lo
opM+0fFcfg1PPPobTQ7FLJQR/uAO93MWZJ43Ht90YEdk6aaBCf7Ego1fU0G6TjCV
iphmOqvhs96GFfhaBMYwFxvHb1tHNDT+Xzlsvkvk+S8SKyhNOg5GJOL2Dz78vlB/
fcImILW4vRf4rIkMDZKL
=kPYg
-----END PGP SIGNATURE-----

--
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

[Simon Kelley]

OK, that's a bit difficult to correlate the client and upstream, but I
can see in the responses coming back from upstream that the order of
the RRs in the Auth section of the "no-such name" replies is

NSEC
RRSIG (for NSEC)
SOA
RRSIG (for SOA)

That's different than for the google public DNS which returns

SOA
NSEC
RRSIG
RRSIG

Both are valid, but the first exercises code-paths the second doesn't,
since the SOA RR has to be moved when the NSEC and its RRSIG are
deleted from the packet.

The address of the upstream server in the captures is an RFC1918
address. Is there a comcast server at a public address that I can
point to and get the same answers? That should allow me to reproduce
the bug here.

I just tried the first 5 or 6 servers in google's answer to the query
"public dns" and they all reply in the SOA-first order. Need to find a
public DNS server that replies in the NSEC-first order.

Cheers,

Simon.

Post by Dave Taht
The offputting part of your outline of what to check for was "some
hairy pointer code". :) I'm in the middle of some crash bugs
elsewhere, and I didn't realize how fast I could get you data
without thinking about the "hairy" parts.
dnssec and dnssec-check-unsigned are enabled, and I'm using
cachesize 9999 (what's the limit nowadays?)
I put packet captures of the external interface on the router
(comcast upstream) and captures taken at the client, a log, and
http://www.taht.net/~d/dnssecbug/
Basically hammering on nslookup for the two internal and internal
captures there.
Hammering on "dig" later, I was unable to trigger it on A, or AAAA
requests. Was able to easily trigger it on a MX request.
flent-freemont does not exist, btw. Flent-fremont, does. It will
go boom on both.
Got bad packet: bad compression pointer 123 bytes a5 c9 81 a0 00 01
00 00 00 01 00 01 0e 66 6c 65 .............fle 6e 74 2d 66
72 65 65 6d 6f 6e 74 0b 62 75 66 66 nt-freemont.buff 65 72
62 6c 6f 61 74 03 6e 65 74 00 00 0f 00 01
erbloat.net..... c0 1b 00 06 00 01 00 00 0e 10 00 34 06 61 72 6e
...........4.arn 6f 6c 64 02 6e 73 0a 63 6c 6f 75 64 66 6c 61 72
old.ns.cloudflar 65 03 63 6f 6d 00 03 64 6e 73 c0 eb 78 9d d7 47
e.com..dns..x..G 00 00 27 10 00 00 09 60 00 09 3a 80 00 00 0e 10
..'....`..:..... 00 00 29 02 00 00 00 00 00 00 00
flent-freemont.bufferbloat.net MX
; <<>> DiG 9.10.3-P4-Ubuntu <<>> flent-freemont.bufferbloat.net MX
QUERY, status: NOERROR, id: 34631 ;; flags: qr rd ra ad; QUERY: 1,
ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;;
QUESTION SECTION: ;flent-freemont.bufferbloat.net. IN MX
;; AUTHORITY SECTION: bufferbloat.net. 3600 IN SOA
arnold.ns.cloudflare.com. dns.cloudflare.com. 2023610183 10000 2400
604800 3600
;; Query time: 72 msec ;; SERVER: 172.26.16.1#53(172.26.16.1) ;;
WHEN: Wed Jan 18 12:42:02 PST 2017 ;; MSG SIZE rcvd: 123

On Wed, Jan 18, 2017 at 11:48 AM, Simon Kelley

I won't have access to a MIPS system 'till the weekend.
I assume you're using the git head code?

Post by Dave Taht
No. Lede-project head. package claims to be dnsmasq-2.76-6.
libc is musl.
Box under test was an archer c7v2. Can go try a few other mips
boxes like the wndr3800, but I've seen it there too. The arm
box (that is working) is an linksys-1200ac. (overall it's
looking like a fine release of lede)....

Did you manage to see a dump of the upstream reply?

Post by Dave Taht
Not yet. I'll touch bases with you later in the week.

Simon.

Post by Dave Taht

Post by Dave Taht
so far I can only make it happen on mips. Doesn't happen on
arm. Haven't tried harder yet.

-- Dave Täht Let's go make home routers and wifi faster! With
better software! http://blog.cerowrt.org