[Dnsmasq-discuss] dnsmasq continuous integration

[Petr Menšík]

Hello everyone.

I maintain dnsmasq in Fedora and Red Hat Enterprise Linux.

We build for different sets of architectures and have some tests for
several packages. Dnsmasq is used for libvirt and network manager as a
dependency. In short, dnsmasq is important to us.

We lack something that we could attach after every build to test dnsmasq
is not (completely) broken. I do not know of any tests included in
dnsmasq repository. Is there any external repository that can be used to
validate dnsmasq still behaves properly? Is anyone using some kind of
continuos integration to ensure new build work at least the same as the
former one? I myself already made patches that broke some architectures
and some not. Such things are not easy to discover.

I currently assume no one has open source tests that can be used to
verify dnsmasq behavior right now. I would be glad if that was not true.
In the other case I would like to start some basic test suite, that can
be run to validate new build. I would like to make something useful that
could be merged into the repository sometime in the future.

I have a couple of questions:
* Do you know good and powerful enough framework to write such tests?
* Is there interest to cooperate on test suite? I think we could all
benefit from this, making dnsmasq more reliable.
* What language and library should be used for tests writing?
* Which tools would be useful for testing?

My kind of requirements:
- setup support of temporary addresses, network namespaces or
containers would be useful
- I would like to avoid reinventing the wheel, starting with bash
scripts, that would be simple at the start and hell to maintain later
- I think scripting languages are more suited for complicated test
setup with more than one daemon instance
- Preferred language would be python for me. Avocado [1] was
recommended to me. It is packaged in Fedora, but not in Debian. Because
Simon is Debian packager, I think something with good support on Debian
should be chosen. Do you know something?
- check library [2] seems interesting. I am afraid current code would
be not easily broken into unit tests written in C
- tests can be started as a single test or set of tests, failure
should be reported for each single test separately
- dig would be useful for dns queries. ldns-testns can be quite useful
for special upstream DNS servers.

What do you think? Any opinions would be appreciated.

[1] https://github.com/avocado-framework/avocado
[2] https://libcheck.github.io/check/


Best Regards,
Petr

[Simon Kelley]

I don't know of any such testing system, and I don't use one at present,
the best I can do is fairly strict dogfood testing: the latest code is
always running in the network router chez Kelleys.

If such a thing can be produced, I would be very interested in running it.


Cheers,

Simon.

Post by Petr Menšík
Hello everyone.
I maintain dnsmasq in Fedora and Red Hat Enterprise Linux.
We build for different sets of architectures and have some tests for
several packages. Dnsmasq is used for libvirt and network manager as a
dependency. In short, dnsmasq is important to us.
We lack something that we could attach after every build to test dnsmasq
is not (completely) broken. I do not know of any tests included in
dnsmasq repository. Is there any external repository that can be used to
validate dnsmasq still behaves properly? Is anyone using some kind of
continuos integration to ensure new build work at least the same as the
former one? I myself already made patches that broke some architectures
and some not. Such things are not easy to discover.
I currently assume no one has open source tests that can be used to
verify dnsmasq behavior right now. I would be glad if that was not true.
In the other case I would like to start some basic test suite, that can
be run to validate new build. I would like to make something useful that
could be merged into the repository sometime in the future.
* Do you know good and powerful enough framework to write such tests?
* Is there interest to cooperate on test suite? I think we could all
benefit from this, making dnsmasq more reliable.
* What language and library should be used for tests writing?
* Which tools would be useful for testing?
- setup support of temporary addresses, network namespaces or
containers would be useful
- I would like to avoid reinventing the wheel, starting with bash
scripts, that would be simple at the start and hell to maintain later
- I think scripting languages are more suited for complicated test
setup with more than one daemon instance
- Preferred language would be python for me. Avocado [1] was
recommended to me. It is packaged in Fedora, but not in Debian. Because
Simon is Debian packager, I think something with good support on Debian
should be chosen. Do you know something?
- check library [2] seems interesting. I am afraid current code would
be not easily broken into unit tests written in C
- tests can be started as a single test or set of tests, failure
should be reported for each single test separately
- dig would be useful for dns queries. ldns-testns can be quite useful
for special upstream DNS servers.
What do you think? Any opinions would be appreciated.
[1] https://github.com/avocado-framework/avocado
[2] https://libcheck.github.io/check/
Best Regards,
Petr

[Geert Stappers]

Post by Simon Kelley

Post by Petr Menšík
Hello everyone.
I maintain dnsmasq in Fedora and Red Hat Enterprise Linux.
We build for different sets of architectures and have some tests for
several packages. Dnsmasq is used for libvirt and network manager as a
dependency. In short, dnsmasq is important to us.
We lack something that we could attach after every build to test dnsmasq
is not (completely) broken. I do not know of any tests included in
dnsmasq repository. Is there any external repository that can be used to
validate dnsmasq still behaves properly? Is anyone using some kind of
continuos integration to ensure new build work at least the same as the
former one? I myself already made patches that broke some architectures
and some not. Such things are not easy to discover.
I currently assume no one has open source tests that can be used to
verify dnsmasq behavior right now. I would be glad if that was not true.
In the other case I would like to start some basic test suite, that can
be run to validate new build. I would like to make something useful that
could be merged into the repository sometime in the future.
* Do you know good and powerful enough framework to write such tests?
* Is there interest to cooperate on test suite? I think we could all
benefit from this, making dnsmasq more reliable.
* What language and library should be used for tests writing?
* Which tools would be useful for testing?
- setup support of temporary addresses, network namespaces or
containers would be useful
- I would like to avoid reinventing the wheel, starting with bash
scripts, that would be simple at the start and hell to maintain later
- I think scripting languages are more suited for complicated test
setup with more than one daemon instance
- Preferred language would be python for me. Avocado [1] was
recommended to me. It is packaged in Fedora, but not in Debian. Because
Simon is Debian packager, I think something with good support on Debian
should be chosen. Do you know something?
- check library [2] seems interesting. I am afraid current code would
be not easily broken into unit tests written in C
- tests can be started as a single test or set of tests, failure
should be reported for each single test separately
- dig would be useful for dns queries. ldns-testns can be quite useful
for special upstream DNS servers.
What do you think? Any opinions would be appreciated.

I don't know of any such testing system, and I don't use one at present,
the best I can do is fairly strict dogfood testing: the latest code is
always running in the network router chez Kelleys.
If such a thing can be produced, I would be very interested in running it.

I read that as "patches welcome" :-)


[1] https://github.com/avocado-framework/avocado
[2] https://libcheck.github.io/check/


Groeten
Geert Stappers