Collection of performance and scalability-related testing scripts.
This tool can use these providers to deploy VMs for testing:
- Vagrant
- IdM CI (uses Red Hat's internal infrastructure, needs authentication)
The controller must have the following software installed:
ansiblegitrsync
For using the Vagrant provider, these need to be installed as well:
vagrantlibvirt- Vagrant plugins
vagrant-libvirtwinrm(for AD support)winrm-elevated(for AD support)
Usage: ipaperftest [OPTIONS]
Options:
--test [EnrollmentTest|APITest|AuthenticationTest|GroupSizeTest]
Test to execute. [default: EnrollmentTest]
--client-image TEXT Image to use for clients.
--server-image TEXT Image to use for server.
--amount INTEGER Size of the test. [default: 1]
--replicas INTEGER RANGE Number of replicas to create. [default: 0;
0<=x<=64]
--threads INTEGER Threads to run per client during
AuthenticationTest. [default: 10]
--ad-threads INTEGER Active Directory login threads to run per
client during AuthenticationTest. [default:
0]
--sizelimit INTEGER IPA search size limit [default: 100]
--disable-selinux Disable the SSSD SELinux provider in all
clients, enable forking in pamtest
--command TEXT Command to execute during APITest.
--results-format [json|human] Format to use for results output [default:
json]
--results-output-file TEXT File to write results output to
--custom-repo-url TEXT URL from custom repo to be configured on the
server hosts. Make sure N-V-R is higher than
the packages available in the server image
so that your packages are used.
--provider [vagrant|idmci] Provider to use during test execution
[default: idmci]
--private-key TEXT Private key needed to access VMs in case the
default is not enough.
--sequential Run APITest commands sequentially from a
single client.
--idmci-lifetime INTEGER Lifetime in hours of IdM-CI hosts.
[default: 8]
--auth-spread INTEGER Time range in minutes to spread auths in
AuthenticationTest [default: 0]
--expected-result-type [time|time_unit|no_errors]
Type of expected result. [default:
no_errors]
--expected-result FLOAT Expected result of the test, in seconds.
--number-of-subgroups INTEGER Number of sub groups for Groupsize test
[default: 0]
--help Show this message and exit.
After executing the script, a sync directory will be created. There you will find logs gathered from all the machines
deployed, including performance monitoring using SAR.
A tarball will be created containing the sync directory and metadata like Ansible playbooks and Vagrantfile.
A result can be passed to the tool so that it fails if the actual result is longer than expected. The currently supported types of expected results are:
time: total time of execution of the test, excluding setup.time_unit: time of execution per each item (defined byamount), excluding setup.no_errors: the test will succeed as long as no errors are raised.
The package can be tested and developed in a python virtual environment.
To create the virtual environment run:
$ python3 -m venv --system-site-packages venv
$ venv/bin/pip install -e .
To use the environment:
$ source venv/bin/activate
If you are using the IdM CI provider, the Ansible Vault password file needs to be set up:
$ echo 'IDMCI_VAULT_PASSWORD' > ~/.idmci-ansible-vault-password-file
To run the tool:
$ source venv/bin/activate
$ ipaperftest
To run an enrollment test, with 100 clients and 2 replicas:
$ ipaperftest --test EnrollmentTest --replicas 2 --amount 100
Try to enroll n clients simultaneously, n being the amount especified using the amount option.
During this test, n client machines are created and configurated.
After this, they are all scheduled to launch ipa-client-install at the same time. A wait is added to ensure
all machines are properly configured before the client installation time. Once all the install processes exit,
the results are retrieved. To ensure that enrollment went well on both ends, clients successes are counted and
compared against the output of ipa host-find on the server.
If using replicas, the distribution of enrollments between servers will be shown after the test.
Perform authentication attempts against the server. The number of clients deployed is set using the amount option,
and the amount of authentication threads per client is defined using the threads option, so the total amount of
authentication attemps performed is amount * threads.
Before launching the authentications both server and clients are configured. Test users are created using the
create-test-data.py and set-password.py scripts, as explained below. After this, authentications are attempted
using the pamtest tool. A file named pamtest.log will be created for each client, containing logs from this run.
After the test execution, percentage of succeeded attempts will be shown, both per client and in total.
This tests runs the same command n times simultaneously. The command is specified using the command option. The
wildcard {id} is permitted for commands that required variability to run properly (for example, it can be used
as a username so that the user-add command can be run multiple times without failing). The amount of commands to
run is set using the amount options, and these commands will be run from clients deployed before the test. Each
client deployed will run a maximum of 25 commands. These commands are scheduled on the client using the at tool.
There is an option to run the test in sequential mode. When this mode is activated, only one client will be deployed, and commands will be executed sequentially from this single client.
After the execution of the test, output from the commands will be written to the sync directory.
Determine how long it takes to add one more user to a group. As groups grow in size the LDAP memberof and indexing calculations becomes more intense. This test creates a selected number of users as an LDIF, as members of a new group (allusers). It then calls ipa group-add-member to add one more user and returns the wall-clock time to do so. In IPA generally we try to keep times under 2s.
There is also the capability to create subgroups. Long ago subgroups was proposed as a workaround to a single group with a large number of members. The number of subgroups is specified iwth --number-of-subgroups. These are all members of a new top-level group and users are more or less equally assigned to the subgroups. Then one more member is added to a subgroup and the time returned.
Rather than declaring a bunch of new options some are reused. The available options are:
threads: number of users to createnumber-of-subgroups: number of subgroups to create (if not specified there is a single group)
Sample execution:
ipaperftest --test GroupSizeTest --threads 1500
ipaperftest --test GroupSizeTest --threads 1500 --number-of-subgroups 3
Find the limit of the IPA API to issue new certificates.
A set number of clients is enrolled then services for each client are created.
For each service an ipa-getcert request is issued. There is little effort made to ensure that these are all run at the same time but in the end this more closely mirrors a live installation.
Rather than declaring a bunch of new options some are reused. The available options are:
cert-requests: number of certificates to request for each clientclients: number of clients to enrollwsgi-processes: number of WSGI processes to enable (default=4)
Sample execution:
ipaperftest --test CertIssuanceTest --amount 70 --cert-requests 5
ipaperftest --test CertIssuanceTest --amount 70 --cert-requests 5 --wsgi-processes 8
For client authentication test we need a lot of users to test against. The combination of two scripts will create the users needed for testing with identical, unexpired passwords.
The create-test-data.py script generates LDIF data that can be imported directly
into LDAP. It must be run on a FreeIPA server as it uses the IPA framework (ipalib)
to determine the domain, base DN, and realm. The script creates:
- Users: one set of users per host, with usernames in the format
user{N}{hostname}(e.g.user0client000.example.com). Each user is created with a pre-hashed password (password), an SSH public key, and standard IPA attributes. - An
allusersgroup: a group containing all generated users as members. - Subgroups (optional): when
--number-of-subgroupsis specified, users are distributed evenly across subgroups which are then nested underallusers, instead of adding all users directly toallusers. - Services (optional): when
--servicesis specified, services are created for each host, managed by that host.
Usage: create-test-data.py [OPTIONS]
Options:
--users-per-host INTEGER Number of users to create for each host.
[default: 10]
--hosts INTEGER Number of hosts to create. [default: 500]
--host-prefix TEXT Hostname prefix. [default: client]
--services INTEGER Number of services per host to create.
[default: 0]
--outfile TEXT LDIF output file. If not specified, output is
written to stdout.
--number-of-subgroups INTEGER Number of subgroups to create. Users are
distributed evenly across subgroups, which
are all nested under the allusers group.
[default: 0]
--with-groups Create user groups.
--with-hostgroups Create host groups.
--with-sudo Create sudo rules.
--with-hbac Create HBAC rules.
--debug Debug logging.
--help Show this message and exit.
The total number of users created is users-per-host * hosts. With the defaults
(10 users per host, 500 hosts), this produces 5000 users.
In order to set passwords using a pre-hashed password IPA needs to be in migration mode:
$ kinit admin
$ ipa config-mod --enable-migration=true
Create 10 users for each of 500 hosts (defaults) and import them:
$ ./create-test-data.py > user.ldif
$ ldapadd -x -D 'cn=directory manager' -W < user.ldif
Create 20 users for each of 100 hosts, with 5 services per host, writing to a file:
$ ./create-test-data.py --users-per-host 20 --hosts 100 --services 5 --outfile user.ldif
$ ldapadd -x -D 'cn=directory manager' -W < user.ldif
Create users with 4 subgroups instead of a single flat group:
$ ./create-test-data.py --users-per-host 10 --hosts 500 --number-of-subgroups 4 > user.ldif
$ ldapadd -x -D 'cn=directory manager' -W < user.ldif
Time to add depends on the server but for 5000 users it was ~9 minutes.
Now reset all Kerberos credentials to the value of 'password':
$ ./set-password.py --dm-password <Directory Manager password>
Time to reset the passwords is ~11 minutes. This is done as the DM user requesting a keytab for each user which will set the Kerberos credentials. The LDAP password is set on the import.