When managing AD objects the SID of the objects are stored in FreeIPA
database, but a user would still use the human readable values, like
"AD\\user" or "user@ad.domain". This can cause idempotence issues in
many cases, and prevent some actions to be performed, like ensure
absence of the object.

The methods added allow the conversion of one or multiple objects, and
will be used by any module that manages AD objects.

Signed-off-by: Rafael Guterres Jeffman <rjeffman@redhat.com>

This patch adds the class EntryFactory to the ansible-freeipa module
utils. This class allows the handling of modules with multiple object
entries as list of objects. When the multi-object parameter is not used,
it creates a list of a single object, allowing for the same code idiom
to be used.

The entries created can be used both as objects, by acessing the values
as properties, or as dictionaires, by accessing the elements as
key-value pairs.

So far a datetime object was created for the expiration dates
krbpasswordexpiration and krbprincipalexpiration. This resulted in also
sending these objects to the API. With this change, the dates are
converted into strings using the LDAP_GENERALIZED_TIME_FORMAT defined in
ipalib.constants. This way only strings are used with the IPA API.

A new function has been added to ansible_freeipa_module:

- date_string: Convert datetime to gernalized time format string

This fuction is used on the result of user_show to convert the
expiration dates to the gernalized time format string.

The existing function date_format in ansible_freeipa_module has been
renamed to convert_date and fixed in the way that it also uses
date_string to return a gernalized time format string and not a
datetime object. This function was only used in the ipauser module so
far.

The error string returned by execute_ipa_commands in batch mode
additionally contains the whole parameter list for the command. This is
different to non batch mode execution and breaks tests that are checking
the returned error message.

A left over debug message also have been removed from the error
processing.

Certificates given by ansible could have leading and trailing white
space, but also multi line input is possible that also could have
leading and training white space and newlines.

New function:
- convert_input_certificates(module, certs, state)

New versions of pylint ignore Python 2 functions and types, evaluating
'unicode' as "undefined". ansible-freeipa will always define 'unicode'
when running under Python 3, and it is always defined under Python 2.

This patch fixes these false positives.

The method execute_ipa_commands has been extended to handle multi
commands with the batch command.

New constants for execute_ipa_commands debugging:

    DEBUG_COMMAND_ALL = 0b1111
    DEBUG_COMMAND_LIST = 0b0001
        Print the while command list
    DEBUG_COMMAND_COUNT = 0b0010
        Print the command number
    DEBUG_COMMAND_BATCH = 0b0100
        Print information about the batch slice size and currently executed
        batch slice

New parameters have been added to execute_ipa_commands:

    batch: bool
        Enable batch command use to speed up processing
    batch_slice_size: integer
        Maximum mumber of commands processed in a slice with the batch
        command
    keeponly: list of string
        The attributes to keep in the results returned.
        Default: None (Keep all)
    debug: integer
        Enable debug output for the exection using DEBUG_COMMAND_*

Batch mode can be enabled within the module with setting batch to True
for execute_ipa_commands.

Fixes: #1128 (batch command support)

normalize_sshpubkey is imported from ipalib.util and also added to
__all__ for use in modules.

FreeIPA PR https://github.com/freeipa/freeipa/pull/7286 moved
ipalib.install.kinit to ipalib.

It is first tried to import kinit_keytab and kinit_password from
ipalib.kinit, then ipalib.install.kinit and finally in some cases
where support for IPA 4.5.0 is needed still also ipapython.ipautil.

Related: https://github.com/freeipa/freeipa/pull/7286

Some parameters, in modules, have a specific data type, but allow the
use of an empty string to clear the parameter.

By providing a method to retrieve the parameter with the correct data
type, or optionally an empty string, allows for consistency of parameter
handling between different modules.

The parameter 'allow_empty_string' in 'module_params_get' is used to
allow an item in a list to be an empty string. The problem is that the
naming is misleading, as it is checking a list item rather than a
string.

This patch rename the parameter to 'allow_empty_list_item' so that it
more clearly refers to list itens instead of standalone strings, and do
not collide with future parameters that may test for empty strings which
are not part of lists.

The uri parameters auth_uri, dev_auth_uri, token_uri, userinfo_uri and
keys_uri have not been validated before. Also the base_url was not
normalized. The auth_uri, dev_auth_uri, token_uri and userinfo_uri need
to be set for new entries, but might be empty or empty string for reset
or updates.

The ipaidpclientsecret needs to be decoded from binary string in
find_idp result to not trigger no change ipd_mod calls.

The code for validate_uri and base_url normalization has been copied
from the ipaserver idp plugin.

ansible_freeipa_module:
urlparse from urllib.parse with a fallback to six.moves.urllib.parse is
imported and also exported. urlparse is needed for validate_uri in ipaidp
module.

Resolves: RHEL-17954, RHEL-17955, RHEL-17957 and RHEL-17958

There is a new idp management module placed in the plugins folder:

    plugins/modules/ipaidp.py

The idp module allows to ensure presence or absence of external Identity
Providers.

Here is the documentation for the module:

    README-idp.md

New idp example playbooks:

    playbooks/idp/idp-present.yml
    playbooks/idp/idp-absent.yml

New tests for the module:

    tests/idp/test_idp.yml
    tests/idp/test_idp_client_context.yml

The use of del os.environ assumes that the environment variable exists.
If the variable does not exist, this call will result in a traceback.
The solution is to use os.environ.pop(VARIABLE, None) instead.

This is the ansible-freeipa fix for https://pagure.io/freeipa/issue/9446
(Nightly test failure for replica installation with --setup-ca)

Fix ipa_command_invalid_param_choices for IPA 4.6 (RHEL-7)

- krbprincipalauthind in host_add does not have choices defined
- krbprincipalauthind in service_add does not have choices defined

api.Command[command].params[name].cli_metavar returns "STR" and
ast.literal_eval failes with a ValueError "malformed string".

There is no way to verify that the given values are valid or not in
this case. The check is done later on while applying the change
with host_add, host_mod, service_add and service_mod.

New IPAAnsibleModule.ipa_command_invalid_param_choices method to return
invalid parameter choices for an IPA command.

This is needed to verify for example if userauthtype and authind are
supporting the idp value.

Export Ansible's 'boolean' parsing function so it can be used to verify
if a string can be handled as a truthy value, allowing module parameters
to use strings instead of bools, as strings can be cleared by using
empty strings.

There is a new certificate management module placed in the plugins
folder:

    plugins/modules/ipacert.py

The certificate module allows to request, revoke, release and retrieve
certificates for users, hosts and services.

Here is the documentation for the module:

    README-cert.md

New example playbooks have been added:

    playbooks/cert/cert-hold.yml
    playbooks/cert/cert-release.yml
    playbooks/cert/cert-request-host.yml
    playbooks/cert/cert-request-service.yml
    playbooks/cert/cert-request-user.yml
    playbooks/cert/cert-retrieve.yml
    playbooks/cert/cert-revoke.yml

New tests for the module can be found at:

    tests/cert/test_cert_client_context.yml
    tests/cert/test_cert_host.yml
    tests/cert/test_cert_service.yml
    tests/cert/test_cert_user.yml

The module has been co-authored by Sam Morris (@yrro) and Rafael
Guterres Jeffman (@rjeffman).

The function exit_raw_json is a replacement for AnsibleModule.exit_json
without flterting out values for no_log parameters.

Ansible added checks for pylint to forbid print and also sys.exit and
fails with ansible-bad-function. As the check is not known outside of
ansible-test, the disable line needed also W0012:

    # pylint: disable=W0012,ansible-bad-function

A combination of ansible-freeipa modifications and a newer version of
IPA has brought a regression regarding different OS localization.

For properly setting environment to use "C" language, as required by
ansible-freeipa, the setting must be executed before importing the
module 'ipaserver.dcerpc', so setting environment language was moved
closer to the 'import os' statement, so that it is always set, as soon
as possible.

Note that 'import os' should always be imported before any FreeIPA
module.

This patch modifies the way that the certificate load function is
defined, depending on the dependency version, so that the resulting
identifier for the function is always set and static analysis tools,
like linters don't complain about variables being used before being
set.

The same idiom is applied to both the ipaclient role and the plugins
ansible_module_utils.

All imports that are only available after installing IPA need to be in a
try exception clause to be able to pass the fake execution test. The old
workaround "if 'ansible.executor' in sys.modules:" is not working with
this test anymore.

If the imports can not be done, all used and needed attributes are
defines with the value None.

A check has been added to IPAAnsibleModule.__init__ to make sure that it
fails if the imports have not been done successfully.

The FreeIPABaseModule class has been maked deprecated with
ansible-freeipa version 1.5.0. It is not used in the code any more
therefore it is time to finally remove it.

api_check_ipa_version was using packaging.version. IPA is using
pkg_resources.parse_version in ipaplatform.tasks.parse_ipa_version.

With this change tasks.parse_ipa_version from ipaplatform is used to
have exactly the same version comparison that also IPA has.

Additionally tasks is added to __all__.

The fallback function used to compare IPA versions was spliting the
version string into a tuple of strings, and the comparison of the tuple
would fail if comparing a field with one digit aginst a two-digit one,
for example, '8' with '10', as the string comparison would put '10'
before the '8'.

This patch forces the version fields to be converted to integers, so
a numerical comparison will be performed. If a version string field
cannot be converted to a number, than the string comparison will still
be used.

FreeIPA 4.9.10+ and 4.10 use proper mapping for boolean values, and
only searching for "TRUE" does not work anymore.

This patch fix ipadnszone plugin and IPAParamMapping class handling
of boolean values.

Python 3.11 dropped compat inspect.getargspec. As the roles and modules
need to support Python2 and Python3, the code for getargspec has been
copied from Python 3.10 and is added as a fallback as soon as getargspec
can not be imported from inspect. The copied getargspec is using
getfullargspec internally.

Fixes: #855 (Python's inspect.getargspec was removed in version 3.11)

When evaluating imports, pylint does not have access to IPA imports,
so they need to be ignored during import or usage.

When managing idranges, it might be needed to obtain the domain SID
from the domain name. As this method needs to use the IPA API object
and requires imorting some ipaserver modules, teh best place for this
method to be implemented is on ansible_module_utils.

This change modifies the comparison of the retrieved IPA object and the
provided arguments on ansible_freeipa_module.compare_args_ipa when the
provider argument is an empty string.

If an attribute is not available in 'ipa', its value is considered to be
a list with an empty string (['']), possibly forcing the conversion of
the 'args' attribute to a list for comparison. This allows, for example,
the usage of empty strings which should compare as equals to inexistent
attributes (None), as is done in IPA API.

So far it is possible to pass list parameters with empty strings to the
modules. The use of empty strings in list does not make a lot of sense,
though. The simple solution is to add a check to module_params_get for
empty strings in returned lists.

The option allow_empty_string can be set to True to allow an empty string
in the list with a list len of 1. The option defaults to False. It is
needed for some parameters the modules, like for example userauthtype in
the user module. It is using "" to reset to the default value.

module_params_get_lowercase has been changed to use module_params_get to
have one place to add the check.

Due to an issue in Ansible it is possible to use the empty string "" for
lists with choices, even if the empty list is not part of the choices.
Ansible issue https://github.com/ansible/ansible/issues/77108

Allows the creation of IPAAnsibleModule objects with specific
`ipa_arguments` which are defined in a dictionary of argumets in
the base class.

Every module using `delete_continue` should provide the proper behavior
and the module must be instantiated with:

  ansible_module = IPAAnsibleModule(
      ...,
      ipa_arguments=["delete_continue"]
  )

The plugin documentation must be extended with
'ipamodule_arguments.delete_continue'.

Ensuring absence of members (services and targets) that do not exist may
not fail as they are not members for servicedelegationtarget and
servicedelegationrule.

servicedelegation_normalize_principals in ansible_freeipa_module has
been extended with a check_exists argument that defaults to False. state
== "present" is now given as this argument to turn on the element exists
check only if elements should be added.

This function will be used in servicedelegation target and rule modules
to normalize principals given in the tasks. These can be service and host
principals and also aliases.

Note: The use of host principals requires IPA 4.9.0 or later. fail_json
is called if the version is lower.

servicedelegation_normalize_principals contains two embedded fuctions.
One is normalize_principal_name that has been copied from
ipaserver/plugins/servicedelegation.py, the other is the generic
function _check_exists to be able to check if a host or service exists.

Some attributes retrieved by the IPA API backend don't change, and are
used more than once, in different places of the code. IPA API domain
is one of these attributes.

This patch adds a cache to the attribute, so there is only one request
for the API, improving access time to the object and alowing multiple
calls with no efficiency penalty.

These functions have been added to get the basedb from api.env for use
with DN for example.

api_get_basedn is returning api.env.basedn
IPAAnsibleModule.ipa_get_basedn is a wrapper for api_get_basedn

This patch is needed to pass Automation Hub tests.