Additional group discovery and creation

docs/en/user-manual/advanced_configuration.md

@@ -681,6 +681,119 @@ In order to use the Okta connector, you will need to specify the `--connector ok
 
 Okta sync can use extended groups, attributes and after-mapping hooks.  The names of extended attributes must be valid Okta profile fields.
 
+## Additional Group Options
+
+It is possible for the User Sync Tool to sync group relationships that
+are not explicitly mapped out in `user-sync-config.yml`.  Any LDAP group
+that a user belongs to directly can be mapped and targeted to an Adobe
+profile or user group using the `additional_groups` configuration
+option.
+
+* Additional groups are identified with a regular expression
+* The name of the mapped Adobe group can be customized with a regular
+expression substitution string.
+
+Possible use cases:
+
+* Metadata such as department, employee type, etc
+* ACL groups for [Adobe Experience Manager](https://www.adobe.com/marketing/experience-manager.html)
+* Special-case group, role or profile assignment
+
+Note: This feature only works with the LDAP connector at this time.
+
+### Additional Group Rules
+
+`additional_groups` is defined in `user-sync-config.yml` in the `groups`
+object.  It specifies a list of rules to identify and filter groups
+present in the `memberOf` LDAP attribute, as well as rules that govern
+how corresponding Adobe groups should be named.  Groups that are
+discovered with this feature will be added to a user's list of
+targeted Adobe groups.
+
+### Additional Group Example
+
+Suppose an Adobe Experience Manager customer would like
+to sync all AEM users to the admin console.  They define a group
+mapping in `user-sync-config.yml` to map the LDAP group `AEM-USERS` to
+the Adobe group `Adobe Experience Manager`.
+
+```yaml
+    - directory_group: "AEM-USERS"
+      adobe_groups:
+        - "Adobe Experience Manager"
+```
+
+This example company's AEM users fall into two broad categories -
+authors and publishers.  These users already belong to LDAP groups that
+correspond to each role - `AEM-ACL-AUTHORS` and `AEM-ACL-PUBLISHERS`,
+respectively.  Suppose this company wishes to assign users to these
+additional groups when syncing users.  Assuming group membership
+information can be found in the `memberOf` user attribute, they can
+leverage the `additional_groups` config option.
+
+```yaml
+directory_users:
+  # ... additional directory config options
+  groups:
+    # ... group mappings, etc
+    additional_groups:
+      - source: "AEM-ACL-(.+)"
+        target: "AEM-(\\1)"
+```
+
+`additional_groups` contains a list of additional group rules. `source`
+is a regular expression that identifies the group.  Only groups that
+match a `source` regex will be included.  `target` is a regex
+substitution string that allows group names to be renamed.  In this
+case, any group beginning with `AEM-ACL` will be renamed to `AEM-[role]`.
+Each rule is executed on the list of groups a user directly belongs to.
+In this example, authors and publishers are added to their respective
+Adobe user group (`AEM-AUTHORS` or `AEM-PUBLISHERS`).
+
+Note: The company in this example can also add mappings for authors
+and publishers to the group mapping in `user-sync-config.yml`.  The
+advantage to using the additional groups mechanism is that it
+will apply dynamically to any LDAP group that matches the regex
+`AEM-ACL-(.+)`.  If additional AEM roles are introduced, they will
+be included in sync as long as they follow that naming convention -
+no configuration change would be needed.
+
+## Automatic Group Creation
+
+The User Sync Tool can be configured to automatically create targeted
+Adobe user groups that do not already exist.  This can be used in
+conjunction with the additional groups functionality detailed in the
+previous section, but it also applies to Adobe groups targeted in
+the group mapping as well as the extension config.
+
+`group_sync_options` is defined in the `directory_users` section in
+`user-sync-config.yml`.  It contains an object that currently has just
+one key - `auto_create`. `auto_create` is boolean and is `False` by
+default.
+
+To enable dynamic group creation, set `auto_create` to `True`:
+
+```yaml
+directory_users:
+  # ... additional directory config options
+  group_sync_options:
+    auto_create: True
+```
+
+With auto create enabled, a given Adobe group will be created if the
+following conditions are true:
+
+1. Group is targeted for at least one user
+2. Group does not currently exist
+3. The `--process-groups` command argument is set (or the equivalent
+   invocation option)
+
+New groups are always created as user groups.  The UMAPI does not
+support product profile creation, so the Sync Tool can't create them.
+If the Sync Tool is configured to target a misspelled profile name, or
+a profile that doesn't exist, it will automatically create a user group
+with the specified name.
+
 ---
 
 [Previous Section](usage_scenarios.md)  \| [Next Section](deployment_best_practices.md)

examples/config files - basic/1 user-sync-config.yml

@@ -167,13 +167,10 @@ directory_users:
     # is specified as a list of entries, each of which has a directory_group
     # setting (whose value is a single directory group) and an adobe_groups
     # setting (whose value is a list of 0 or more product configuration and
-    # user groups).  All of the values in the adobe_groups settings must
-    # match the name of product configurations and user groups which have
-    # already been created on the Adobe side.  (In this example, we pretend
-    # that "Acrobat DC Pro" is a product configuration and "Copy Editors"
-    # is a user group that the you have already created.  Possibly
-    # the "Copy Editors" user group has been assigned access to appropriate
-    # Adobe products, such as InDesign and InCopy.)
+    # user groups).  In this example, imagine that "Acrobat DC Pro" is a
+    # product configuration and "Copy Editors" is a user group, and that
+    # the "Copy Editors" user group will be assigned access to appropriate
+    # Adobe products, such as InDesign and InCopy.
     # [You will need to edit or remove these examples.]
     - directory_group: "Finance"
       adobe_groups:
@@ -186,6 +183,46 @@ directory_users:
         - "Copy Editors"
         - "Acrobat DC Pro"
 
+  # (optional) additional_groups (no default value)
+  # People who use their directory groups for ACLs on the Adobe side
+  # often have a very large number of groups that they want mapped
+  # over to (user) groups on the Adobe side.  To avoid having to
+  # specify those groups statically in their config file, and to
+  # update their config file when they change, they can instead
+  # use a naming convention for the groups and specify that here.
+  # The value of this attribute is a mapping from Python regular expressions
+  # that specify directory groups of interest to Python replacement expressions
+  # that specify how to construct the name of the target Adobe group
+  # that the directory group should be mapped to.  If a value is
+  # provided, then all users who are (directly) in groups whose
+  # CN matches one of the source regular expressions will be put in a user group
+  # on the Adobe side whose name is given by the target replacement expression.
+  # The simple example here (which should be removed) maps all the
+  # groups that start with "ACL-" or end with "-ACL" to an Adobe
+  # group that starts with "ACL-Grp-".
+  # (All of these regular expressions must match the entire group name.
+  # For details on Python regular expression matching and replacement,
+  # see https://docs.python.org/howto/regex.html )
+  # additional_groups:
+  #   - source: "ACL-(.+)"
+  #     target: "ACL-Grp-(\1)"
+  #   - source: "(.+)-ACL"
+  #     target: "ACL-Grp-(\1)"
+
+  # (optional) group_sync_options (default: all options false)
+  # Options that govern the automatic creation and/or deletion of Adobe user groups
+  # auto_create:
+  #   Automatically create target groups that do not exist.  Non-existent groups
+  #   will *always* be created as Adobe groups.  If targeting product profiles, they
+  #   must always be created manually in the Admin Console.
+  #   NOTE: auto_create applies to any targeted Adobe group that doesn't exist.
+  #         This includes groups targeted through group mapping, extension config,
+  #         and the additional_groups functionality.
+  #   The --process-groups command argument or equivalent invocation setting must
+  #   be enabled for groups to be auto-created
+  # group_sync_options:
+  #   auto_create: False
+
 # The limits section provides processing limits which can help ensure that
 # User Sync jobs do not exceed expected guardrails in their operation
 limits:

examples/config files - basic/3 connector-ldap.yml

@@ -72,14 +72,17 @@ all_users_filter: "(&(objectClass=user)(objectCategory=person)(!(userAccountCont
 group_filter_format: "(&(|(objectCategory=group)(objectClass=groupOfNames)(objectClass=posixGroup))(cn={group}))"
 
 # (optional) group_member_filter_format (default value given below)
-# group_users_filter specifies the query used to find all members of a group,
+# group_member_filter_format specifies the query used to find all members of a group,
 # where the string {group_dn} is replaced with the group distinguished name.
 # The default value just finds users who are immediate members of the group,
 # not those who are "indirectly" members by virtue of membership in a group
 # that is contained in the group.  If you want indirect containment, then
 # use this value instead of the default:
 # group_member_filter_format: "(memberOf:1.2.840.113556.1.4.1941:={group_dn})"
 group_member_filter_format: "(memberOf={group_dn})"
+# Note that this filter is &-combined with the all_users_filter so that
+# only users that would be selected by that filter will be returned as
+# members of the given group.
 
 # (optional) string_encoding (default value given below)
 # string_encoding specifies the Unicode string encoding used by the directory.
@@ -172,14 +175,9 @@ user_email_format: "{mail}"
 # are already pre-defined attribute names that are used for these fields:
 # - the Adobe first name is set from the LDAP "givenName" attribute
 # - the Adobe last name is set from the LDAP "sn" (surname) attribute
-# - the Adobe country is set from the LDAP "country" attribute
+# - the Adobe country is set from the LDAP "c" (country) attribute
 # If you need to override these values on the Adobe side, you can use the
 # custom extension mechanism (see the docs) to compute and set field values
-# by combining these and any other custom attributes needed.  Seed the
+# by combining these and any other custom attributes needed.  See the
 # User Sync documentation for full details.
-#
-# Finally, some LDAP systems use uids to identify groups, and place users in
-# groups via uid rather than name.  The User Sync implementation always reads
-# the uid attribute on all objects if the directory provides one, so it is
-# able to handle directories which function in this way even though the
-# configuration files always specify groups by name.
+

setup.py

@@ -52,7 +52,7 @@
           'pyldap==2.4.45',
           'PyYAML',
           'six',
-          'umapi-client>=2.10',
+          'umapi-client>=2.11',
       ],
       extras_require={
           ':sys_platform=="linux" or sys_platform=="linux2"':[

user_sync/app.py

@@ -307,6 +307,13 @@ def begin_work(config_loader):
             directory_connector_options['user_identity_type'] = rule_config['new_account_type']
         directory_connector.initialize(directory_connector_options)
 
+    additional_group_filters = None
+    additional_groups = rule_config.get('additional_groups', None)
+    if additional_groups and isinstance(additional_groups, list):
+        additional_group_filters = [r['source'] for r in additional_groups]
+
+    directory_connector.state.additional_group_filters = additional_group_filters
+
     primary_name = '.primary' if secondary_umapi_configs else ''
     umapi_primary_connector = user_sync.connector.umapi.UmapiConnector(primary_name, primary_umapi_config)
     umapi_other_connectors = {}

user_sync/config.py

@@ -444,6 +444,7 @@ def get_rule_options(self):
         options.update(self.invocation_options)
 
         # process directory configuration options
+        new_account_type = None
         directory_config = self.main_config.get_dict_config('directory_users', True)
         if directory_config:
             # account type
@@ -457,6 +458,15 @@ def get_rule_options(self):
             default_country_code = directory_config.get_string('default_country_code', True)
             if default_country_code:
                 options['default_country_code'] = default_country_code
+            additional_groups = directory_config.get_list('additional_groups', True) or []
+            additional_groups = [{'source': re.compile(r['source']), 'target': r['target']} for r in additional_groups]
+            options['additional_groups'] = additional_groups
+            sync_options = directory_config.get_dict_config('group_sync_options', True)
+            if sync_options:
+                options['auto_create'] = sync_options.get_bool('auto_create', True)
+        if not new_account_type:
+            new_account_type = user_sync.identity_type.ENTERPRISE_IDENTITY_TYPE
+            self.logger.debug("Using default for new_account_type: %s", new_account_type)
 
         # process exclusion configuration options
         adobe_config = self.main_config.get_dict_config('adobe_users', True)

user_sync/connector/directory_ldap.py

@@ -28,6 +28,7 @@
 import user_sync.error
 import user_sync.identity_type
 from user_sync.error import AssertionException
+from ldap import dn
 
 
 def connector_metadata():
@@ -121,6 +122,7 @@ def __init__(self, caller_options):
         self.connection = connection
         logger.debug('Connected')
         self.user_by_dn = {}
+        self.additional_group_filters = None
 
     def load_users_and_groups(self, groups, extended_attributes, all_users):
         """
@@ -209,6 +211,7 @@ def iter_users(self, users_filter, extended_attributes):
         user_attribute_names.extend(self.user_email_formatter.get_attribute_names())
         user_attribute_names.extend(self.user_username_formatter.get_attribute_names())
         user_attribute_names.extend(self.user_domain_formatter.get_attribute_names())
+        user_attribute_names.append('memberOf')
 
         extended_attributes = [six.text_type(attr) for attr in extended_attributes]
         extended_attributes = list(set(extended_attributes) - set(user_attribute_names))
@@ -289,6 +292,18 @@ def iter_users(self, users_filter, extended_attributes):
             elif last_attribute_name:
                 self.logger.warning('No country code attribute (%s) for user with dn: %s', last_attribute_name, dn)
 
+            uid_value = LDAPValueFormatter.get_attribute_value(record, six.text_type('uid'))
+            source_attributes['uid'] = uid_value
+
+            user['member_groups'] = []
+            if self.additional_group_filters:
+                member_groups = []
+                for f in self.additional_group_filters:
+                    for g in self.get_member_groups(record):
+                        if f.match(g) and g not in member_groups:
+                            member_groups.append(g)
+                user['member_groups'] = member_groups
+
             if extended_attributes is not None:
                 for extended_attribute in extended_attributes:
                     extended_attribute_value = LDAPValueFormatter.get_attribute_value(record, extended_attribute)
@@ -301,6 +316,36 @@ def iter_users(self, users_filter, extended_attributes):
 
             yield (dn, user)
 
+    def get_member_groups(self, user):
+        """
+        Get a list of member group common names for user
+        Assumes groups are contained in attribute memberOf
+        :param user:
+        :return:
+        """
+        group_names = []
+        groups = LDAPValueFormatter.get_attribute_value(user, 'memberOf')
+        for group_dn in map(dn.str2dn, groups):
+            group_cn = self.get_cn_from_dn(group_dn)
+            if group_cn:
+                group_names.append(group_cn)
+        return group_names
+
+    @staticmethod
+    def get_cn_from_dn(group_dn):
+        """
+        Take a DN parsed by ldap.dn.str2dn and locate and return the common name
+        Returns None if no common name is found
+        If common name is complex (e.g. cn=Bob [email protected]) then first part of CN is returned
+        :param group_dn:
+        :return:
+        """
+        for rdn in group_dn:
+            for rdn_part in rdn:
+                if rdn_part[0].lower() == 'cn':
+                    return rdn_part[1]
+        return None
+
     def iter_search_result(self, base_dn, scope, filter_string, attributes):
         """
         type: filter_string: str

user_sync/connector/umapi.py

@@ -146,6 +146,32 @@ def iter_users(self):
         except umapi_client.UnavailableError as e:
             raise AssertionException("Error contacting UMAPI server: %s" % e)
 
+    def get_groups(self):
+        return list(self.iter_groups())
+
+    def iter_groups(self):
+        try:
+            for g in umapi_client.GroupsQuery(self.connection):
+                yield g
+        except umapi_client.UnavailableError as e:
+            raise AssertionException("Error contacting UMAPI server: %s" % e)
+
+    def get_user_groups(self):
+        return list(self.iter_user_groups())
+
+    def iter_user_groups(self):
+        try:
+            for g in umapi_client.UserGroupsQuery(self.connection):
+                yield g
+        except umapi_client.UnavailableError as e:
+            raise AssertionException("Error contacting UMAPI server: %s" % e)
+
+    def create_group(self, name):
+        if name:
+            group = umapi_client.UserGroupAction(group_name=name)
+            group.create(description="Automatically created by User Sync Tool")
+            return self.connection.execute_single(group)
+
     def get_action_manager(self):
         return self.action_manager