Update autotools, add Travis-CI support, and remove some unused code by grondo · Pull Request #93 · chaos/pdsh

grondo · 2017-06-21T18:15:22Z

The initial objective of these changes was to be able to update autotools for pdsh and remove autotools products from version control, in order to make addition of new modules much easier on the developer (e.g. see #91).

To make this easier, the use of libltdl was dropped in favor of straight dlopen(3) and dlsym(3), which should be fine for most if not all pdsh users nowadays.

Along the way many modules that were presumed unused were dropped, including

Quadrics/Elan support (qshell, mqshell, and the rmsquery module)
The SDR module used to get lists of hosts from SDRGetObjects on IBM SPs
The nodeattr module (everyone uses libgenders now)

After this work, the xpopen() function had no users so it was removed.

The use of META and config/ac_meta.m4 were removed.

Finally, support for Travis-CI was added, and many compiler warnings were fixed along the way.

Support for --enable-code-coverage and make check-code-coverage was also added, but I couldn't get this working in Travis yet to send results to codecov.io.

@garlick, @chu11, happy to take any feedback here.

@tabaer, you might want to rebase #91 on this and see how it goes -- it should be much smoother.

Fixes #79

Add /.libs to path for PDSH_MODULE_DIR in tests, so that we find and load the .so files instead of .la libtool archives.

Set pdsh_module_dir to src/modules/.libs so that we load .so module files, instead of attempting to load the libtool archive .la files.

Maintaining the autotools generated files in version control is a huge pain in the rear end. Every time someone runs ./bootstrap on a new system they create huge amounts of unnecessary churn. Yes, this requires that all developers and users working from a git checkout have compatible autotools installed on their system, so that they can successfully run ./bootstrap and ./configure. However, the alternative is a big mess and who knows how many years of therapist payments.

In the slurm and torque modules, the ltdl.h header is included but not used. Remove it in preparation for removal of libltdl.

Dump libltdl in favor of straight dlopen/dlsym for pdsh modules. The new libltdl wrapper is a whole thing of its own and a big burden to use (it wants to be installed as a subproject inside of the project which uses it) This likely means pdsh dynamic modules are no longer supported on systems. Likely those systems don't work now anyway. In the future if an unlikely port is needed to a system requiring libltdl, it should be possible to tackle re-implementation of libltdl (or some other portability layer) at that time.

Fix most autotools warnings by pushing the result of `autoupdate` plus some other various fixes by hand.

Remove non-portable use of $(wildcard) in test/Makefile.am. Just explicitly list the tests instead. This works better for newer automake parallel-tests anyway.

Remove support for launching jobs on older Quadrics/Elan HW via the old (m)qcmd rcmd modules. There are very likely no more users of this interface.

The etc/ subdir only had files related to Qsw/elan services, which are no longer supported, so remove them.

chu11 · 2017-06-21T18:48:39Z

-    AC_HELP_STRING([--without-pam],
-      [Do not build qshell/mqshell with pam support]),
+    AS_HELP_STRING([--without-pam],[Do not build qshell/mqshell with pam support]),
    [ case "$withval" in


we can probably remove ac_pam? It appears to be specific to qshell/mqshell?

Good catch forgot about that one.

PAM checks were only required by qshell/mqshell daemons, so the autoconf tests can now be removed.

Remove support for querying WCOLL from RMS resource manager via the rms misc module. There are likely no current users of this interface.

Remove the SDR module, which queried the SDR on IBM SP machines using a call to SDRGetObjects. There are likely no users of this module anymore.

Remove the nodeattr module. All users should be using the genders module instead.

The xpopen() call is no longer used after removal of the nodeattr module. Remove the unused code from the source.

Udpate pdsh manpage to remove modules that are deprecated or no longer included in the distribution.

Remove unsupported modules from the static modules macros.

Remove qsw modules from rcmd_rank_list used for default rcmd module priority.

Remove reference to unsupported modules from README and README.modules

Detect pdsh version from output of git describe.

Simplify the bootstrap file after autotools updates.

Fix make check for cases where builddir != srcdir

Add scripts/travis-ci-deps.sh script to download and build extra travis-ci build and test dependencies. For now this is only the updated autotools versions that have not made it into travis.

Remove unused variable `q` in pipecmd_format_arg() to silence warning on GCC 6.

Add missing return code checks for all write(2) calls to silence compiler warnings.

Add checks for return codes from write(2) to silence compiler warnings.

Fix compiler warning in _read_netgroup (suggested parenthesis)

Check return code of system(3) in interactive dsh to silence compiler warnings.

Silence compiler warning by checking return code of setuid(2) in interactive dsh, even though setuid(2) is probably not necessary here anymore since pdsh is using privilege separation.

remove unused variable in pcptest_argv_create

Check return codes on all write(2) calls to silence compiler warnings.

Fix compiler warning for failing to check return code of setuid, even though likely these calls are not used.

Fix compiler warning for unchecked return code of setXid calls.

Using the ax_code_coverage.m4 macro, enable a --enable-code-coverage option for ./configure which activates a `make check-code-coverage` target that builds an integrated code coverage report using lcov(1) when available.

garlick · 2017-06-21T23:34:49Z

There are a ton of great cleanups here. Thanks for doing all this painful work! Good riddance to the outdated modules too.

garlick · 2017-06-22T13:14:41Z

FWIW I had zero issues bootstrapping, building, and checking this branch from a freshly cloned repo on c9.io.

grondo · 2017-06-22T17:52:09Z

Thanks @garlick, @chu11. I'm going to merge this here and tag a pdsh-2.32. Then we should move the main project over to chaos/pdsh at some point.

grondo added 9 commits June 19, 2017 13:47

tests: look for built modules under .libs

e240511

Add /.libs to path for PDSH_MODULE_DIR in tests, so that we find and load the .so files instead of .la libtool archives.

pdsh/testconfig.c: load in-tree modules from .libs

6721633

Set pdsh_module_dir to src/modules/.libs so that we load .so module files, instead of attempting to load the libtool archive .la files.

remove unused ltdl.h from modules

3e8e2a1

In the slurm and torque modules, the ltdl.h header is included but not used. Remove it in preparation for removal of libltdl.

build: commit results of autoupdate, fix warnings

fb5d1fb

Fix most autotools warnings by pushing the result of `autoupdate` plus some other various fixes by hand.

build: remove non-portable Makefile function

5f19fc1

Remove non-portable use of $(wildcard) in test/Makefile.am. Just explicitly list the tests instead. This works better for newer automake parallel-tests anyway.

rcmd: remove qcmd/mqcmd remote execution module

91a956e

Remove support for launching jobs on older Quadrics/Elan HW via the old (m)qcmd rcmd modules. There are very likely no more users of this interface.

etc: remove the etc subdirectory

5689afd

The etc/ subdir only had files related to Qsw/elan services, which are no longer supported, so remove them.

chu11 reviewed Jun 21, 2017

View reviewed changes

grondo added 20 commits June 21, 2017 12:57

configure: remove --with-pam

1d9cecd

PAM checks were only required by qshell/mqshell daemons, so the autoconf tests can now be removed.

misc: remove rmsquery module

ffb94f3

Remove support for querying WCOLL from RMS resource manager via the rms misc module. There are likely no current users of this interface.

misc: remove SDR module

885f93a

Remove the SDR module, which queried the SDR on IBM SP machines using a call to SDRGetObjects. There are likely no users of this module anymore.

misc: remove nodeattr module

62ed648

Remove the nodeattr module. All users should be using the genders module instead.

remove unused xpopen implementation

ecfc7d1

The xpopen() call is no longer used after removal of the nodeattr module. Remove the unused code from the source.

doc: remove unsupported modules from pdsh manpage

632e0fb

Udpate pdsh manpage to remove modules that are deprecated or no longer included in the distribution.

pdsh.spec: remove unsupported modules

e6d5086

config/ac_static_modules.m4: remove some modules

26fd897

Remove unsupported modules from the static modules macros.

configure: remove old modules from rcmd_rank_list

e6e6832

Remove qsw modules from rcmd_rank_list used for default rcmd module priority.

update README and README.modules

bc18115

Remove reference to unsupported modules from README and README.modules

build: Remove META file

283bdbb

Detect pdsh version from output of git describe.

build: simplify bootstrap file

94314a7

Simplify the bootstrap file after autotools updates.

build: enable silent rules for automake

f4d196f

tests: t1001-genders.sh requires bash

6cc64f1

tests: update to TAP-capable test driver

9e9155f

tests: fix for vpath builds

e6fd07a

Fix make check for cases where builddir != srcdir

travis-ci: add script for travis-ci deps

165dda3

Add scripts/travis-ci-deps.sh script to download and build extra travis-ci build and test dependencies. For now this is only the updated autotools versions that have not made it into travis.

travis-ci: initial config for travis-ci.org

2508a99

pdsh/opt.c: fix compiler warning

625ac8a

tests: fix compiler warning in pcptest

e4d47b6

grondo added 12 commits June 21, 2017 12:58

common/pipecmd.c: remove unused variable

c931440

Remove unused variable `q` in pipecmd_format_arg() to silence warning on GCC 6.

xrcmd: add missing return code checks on write(2)

f60cccd

Add missing return code checks for all write(2) calls to silence compiler warnings.

mcmd: check write(2) return codes

1e474b9

Add checks for return codes from write(2) to silence compiler warnings.

netgroup: fix compiler warning

fe93b85

Fix compiler warning in _read_netgroup (suggested parenthesis)

pdsh: check return code of system(3)

f8779b3

Check return code of system(3) in interactive dsh to silence compiler warnings.

pdsh: check return code for setuid(2)

33b3a13

Silence compiler warning by checking return code of setuid(2) in interactive dsh, even though setuid(2) is probably not necessary here anymore since pdsh is using privilege separation.

tests/pcptest: remove unused variable

b057e18

remove unused variable in pcptest_argv_create

pcp_server: check return codes on write

71672bd

Check return codes on all write(2) calls to silence compiler warnings.

sshcmd,execcmd: check result of setuid

a0c91cc

Fix compiler warning for failing to check return code of setuid, even though likely these calls are not used.

privsep: fix compiler warnings

3054664

Fix compiler warning for unchecked return code of setXid calls.

travis-ci: Add -Werror to travis builds

5c14609

build: add code coverage target for make check

9cc020c

Using the ax_code_coverage.m4 macro, enable a --enable-code-coverage option for ./configure which activates a `make check-code-coverage` target that builds an integrated code coverage report using lcov(1) when available.

grondo force-pushed the autotools-update branch from 990a8e7 to 9cc020c Compare June 21, 2017 20:02

grondo merged commit b52893c into master Jun 22, 2017

grondo deleted the autotools-update branch June 22, 2017 17:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update autotools, add Travis-CI support, and remove some unused code#93

Update autotools, add Travis-CI support, and remove some unused code#93
grondo merged 41 commits into
masterfrom
autotools-update

grondo commented Jun 21, 2017 •

edited

Loading

Uh oh!

chu11 Jun 21, 2017

Uh oh!

grondo Jun 21, 2017

Uh oh!

garlick commented Jun 21, 2017

Uh oh!

garlick commented Jun 22, 2017

Uh oh!

grondo commented Jun 22, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

grondo commented Jun 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chu11 Jun 21, 2017

Choose a reason for hiding this comment

Uh oh!

grondo Jun 21, 2017

Choose a reason for hiding this comment

Uh oh!

garlick commented Jun 21, 2017

Uh oh!

garlick commented Jun 22, 2017

Uh oh!

grondo commented Jun 22, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

grondo commented Jun 21, 2017 •

edited

Loading