Merge pull request #368 from bhunut-adobe/feature/exclude_unmapped

Added new parameter: --adobe-users all|mapped|group

Author: adorton-adobe

docs/en/user-manual/command_parameters.md

@@ -39,7 +39,8 @@ specific behavior in various situations.
 | `--adobe-only-user-list` _filename_ | Specifies a file from which a list of users will be read.  This list is used as the definitive list of "Adobe only" user accounts to be acted upon.  One of the `--adobe-only-user-action` directives must also be specified and its action will be applied to user accounts in the list.  The `--users` option is disallowed if this option is present: only account removal actions can be processed.  |
 | `--config-file-encoding` _encoding_name_ | Optional.  Specifies the character encoding for the contents of the configuration files themselves.  This includes the main configuration file, "user-sync-config.yml" as well as other configuration files it may reference.  Default is `utf8` for User Sync 2.2 and later and `ascii` for earlier versions.<br />Character encoding in the user source data (whether csv or ldap) is declared by the connector configurations, and that encoding can be different than the encoding used for the configuration files (e.g., you could have a latin-1 configuration file but a CSV source file that uses utf-8 encoding).<br />The available encodings are dependent on the Python version used; see the documentation [here for Python 2.7](https://docs.python.org/2.7/library/codecs.html#standard-encodings) or [here for Python 3.6](https://docs.python.org/3.6/library/codecs.html#standard-encodings) for more information.  |
 | `--strategy sync`<br />`--strategy push` | Available in release 2.2 and later. Optional.  Default operating mode is `--strategy sync`.   Controls whether User Sync reads user information from Adobe and compares to the directory information and then issues updates to Adobe, or simply pushes the directory input to Adobe without considering the existing user information on Adobe.  `sync` is the default and the subject of the description of most of this documentation.  `push` is useful when there is a large number of users on the Adobe side (>30,000) and known additions or changes to a small number of users are desired, and the list of those users is available in a csv file or a specific directory group.<br />If `--strategy push` is specified, `--adobe-only-user-action` cannot be specified as the determination of adobe-only users is not made.<br/>`--strategy push` will create new users, modify their group memberships for mapped groups only (if `--process-groups` is present),  update user information (if `--update-user-info` is present), and will not remove users from the organization or delete their accounts.  See [Handling Push Notifications](usage_scenarios.md#handling-push-notifications) for information on how to remove users via push notifications. |
-| `--connector ldap`<br />`--connector okta`<br />`--connector csv` _filename_ | Available in release 2.3 and later. Optional. Specifies the directory connector to be used (defaults to LDAP).  If you specify the use of a CSV input file with this argument, then you cannot also specify one with `--users`, but you can then specify other `--users` options (such as `mapped` or `group`) for use with the CSV file.  (The Okta connector does not support `--users all`, so you must specify a `--users` option of `mapped` or `group` if you use the Okta connector.)
+| `--connector ldap`<br />`--connector okta`<br />`--connector csv` _filename_ | Available in release 2.3 and later. Optional. Specifies the directory connector to be used (defaults to LDAP).  If you specify the use of a CSV input file with this argument, then you cannot also specify one with `--users`, but you can then specify other `--users` options (such as `mapped` or `group`) for use with the CSV file.  (The Okta connector does not support `--users all`, so you must specify a `--users` option of `mapped` or `group` if you use the Okta connector.) |
+| `--adobe-users all`<br />`--adobe-users mapped`<br />`--adobe-users group` _grp1,grp2_ | Available in release 2.4 and later. Optional. Specify the adobe users to be selected for sync. The default is all meaning all users found in Adobe Admin Console. Specifying group interprets the argument as a comma-separated list of groups (product profile or user-group) in the console, and only users in those groups are selected. Specifying mapped is the same as specifying group with all the adobe groups listed in the group mapping in the configuration file.
 {: .bordertablestyle }
 
 As of version 2.3 of User Sync, the values of most command-line parameters can also be specified in the main configuration file, in an optional section called `invocation_defaults`.  Here is an example use of that section:

docs/en/user-manual/usage_scenarios.md

@@ -43,7 +43,7 @@ This section provides detailed instructions for each of these scenarios.
 ## Update users and group memberships
 
 This is the most typical and common type of invocation. User Sync
-finds all changes to user information and to group membership information 
+finds all changes to user information and to group membership information
 on the enterprise
 side. It syncs the Adobe side by adding, updating, and removing
 users and user group and product configuration memberships.
@@ -89,7 +89,7 @@ parameter.
 ### View result
 
 When the synchronization succeeds, the Adobe Admin Console is
-updated.  After this command is executed, your user list and 
+updated.  After this command is executed, your user list and
 product configuration user list in the
 Admin Console shows that a user with a Federated identity has
 been added to the “Default Acrobat Pro DC configuration.”
@@ -168,7 +168,7 @@ download in `examples/csv inputs - user and remove lists/`.
 ./user-sync --users file user_list.csv
 ```
 
-Syncing from a file can be used in two situations.  First, Adobe users can be managed 
+Syncing from a file can be used in two situations.  First, Adobe users can be managed
 using a spreadsheet.  The spreadsheet lists users, the groups they are in, and
 information about them.  Second, if the enterprise directory can provide push notifications
 for updates, these notifications can be placed in a csv file and used to drive
@@ -183,7 +183,7 @@ users from the Adobe side.
 If you want to handle removals separately, you can instruct
 the tool to flag users that no longer exist in the enterprise
 directory but still exist on the Adobe side. The
-`--adobe-only-user-action write-file exiting-users.csv` parameter 
+`--adobe-only-user-action write-file exiting-users.csv` parameter
 writes out the list of user who
 are flagged for removal to a CSV file.
 
@@ -238,24 +238,49 @@ on the list generated in a prior run of User Sync.
 ./user-sync --adobe-only-user-list users-to-delete.csv --adobe-only-user-action delete
 ```
 
+### Limit Adobe users scope for syncing
+By supplying `adobe-users ...` argument you have an ability to control which Adobe users to be pull into User Sync Tool to be process for sync.
+With this argument you can specify limit by group name (`--adobe-users groups "..."`) or limit by adobe groups in group mapping in the configuration file (`--adobe-users mapped`)
+
+When not specifying `adobe-users groups | mapped` User Sync Tool will automatically default to all:
+```sh
+./user-sync –c user-sync-config.yml --adobe-users all
+```
+
+### Limit Adobe users to Adobe groups.
+
+This action limited Adobe users scope to specified group. Group can be either product profile or user-group in the Adobe Admin Console.
+
+```sh
+./user-sync –c user-sync-config.yml --adobe-users groups "group1, group2, group3"
+```
+
+### Limit Adobe users to mapped Adobe groups
+
+This action is the same as specifying `--adobe-users groups "..."`, where `...` is all the Adobe groups in the group mapping in the configuration file.
+
+```sh
+./user-sync –c user-sync-config.yml --adobe-users mapped
+```
+
 ## Handling Push Notifications
 
 If your directory system can generate notifications of updates you can use User Sync to
-process those updates incrementally.  The technique shown in this section can also be 
-used to process immediate updates where an administrator has updated a user or group of 
-users and wants to push just those updates immediately into Adobe's user management 
-system.  Some scripting may be required to transform the information coming from the 
-push notification to a csv format suitable for input to User Sync, and to separate 
+process those updates incrementally.  The technique shown in this section can also be
+used to process immediate updates where an administrator has updated a user or group of
+users and wants to push just those updates immediately into Adobe's user management
+system.  Some scripting may be required to transform the information coming from the
+push notification to a csv format suitable for input to User Sync, and to separate
 deletions from other updates, which mush be handled separately in User Sync.
 
-Create a file, say, `updated_users.csv` with the user update format illustrated in 
-the `users-file.csv` example file in the folder `csv inputs - user and remove lists`.  
+Create a file, say, `updated_users.csv` with the user update format illustrated in
+the `users-file.csv` example file in the folder `csv inputs - user and remove lists`.
 This is a basic csv file with columns for firstname, lastname, and so on.
 
     firstname,lastname,email,country,groups,type,username,domain
     John,Smith,[email protected],US,"AdobeCC-All",enterpriseID
     Jane,Doe,[email protected],US,"AdobeCC-All",federatedID
- 
+
 This file is then provided to User Sync:
 
 ```sh
@@ -274,8 +299,8 @@ This will handle deletions based on the notification and no other actions will b
 
 ## Action Summary
 
-At the end of the invocation, an action summary will be printed to the log (if the level is INFO or DEBUG). 
-The summary provides statistics accumulated during the run. 
+At the end of the invocation, an action summary will be printed to the log (if the level is INFO or DEBUG).
+The summary provides statistics accumulated during the run.
 The statistics collected include:
 
 - **Total number of Adobe users:** The total number of Adobe users in your admin console

examples/config files - basic/user-sync-config.yml

@@ -319,6 +319,10 @@ invocation_defaults:
   adobe_only_user_action: preserve
   # For argument --adobe-only-user-list, the default is empty (no value).
   adobe_only_user_list:
+  # For argument --adobe-users, the default is 'all'.
+  # if you want to specify group manually. Valid value format is
+  # ['group', 'groupA,groupB']
+  adobe_users: all
   # For argument --connector, the default is 'ldap'.
   connector: ldap
   # For argument --process-groups, the default is False (don't process).

tests/fixture/user-sync-config.yml

@@ -25,6 +25,7 @@ logging:
 invocation_defaults:
   adobe_only_user_action: preserve
   adobe_only_user_list: null
+  adobe_users: all
   connector: ldap
   process_groups: No
   strategy: sync

tests/test_config.py

@@ -136,3 +136,29 @@ def test_twostep_config(tmp_config_files, modify_ldap_config, caplog):
     assert 'two_steps_lookup' in options
     assert 'group_member_attribute_name' in options['two_steps_lookup']
     assert options['two_steps_lookup']['group_member_attribute_name'] == 'member'
+
+
+def test_adobe_users_config(tmp_config_files, modify_root_config):
+    (root_config_file, _, _) = tmp_config_files
+    args = app.process_args(['-c', root_config_file])
+
+    # test default
+    config_loader = ConfigLoader(args)
+    options = config_loader.load_invocation_options()
+    assert 'adobe_users' in options
+    assert options['adobe_users'] == ['all']
+
+    # test default invocation
+    modify_root_config(['invocation_defaults', 'adobe_users'], "mapped")
+    config_loader = ConfigLoader(args)
+    options = config_loader.load_invocation_options()
+    assert 'adobe_users' in options
+    assert options['adobe_users'] == ['mapped']
+
+    # test command line param
+    modify_root_config(['invocation_defaults', 'adobe_users'], "all")
+    args = app.process_args(['-c', root_config_file, '--adobe-users', 'mapped'])
+    config_loader = ConfigLoader(args)
+    options = config_loader.load_invocation_options()
+    assert 'adobe_users' in options
+    assert options['adobe_users'] == ['mapped']

user_sync/app.py

@@ -154,6 +154,13 @@ def process_args(args=None):
                              "users by also including --adobe-only-user-action and one of its arguments",
                         metavar='input_path',
                         dest='adobe_only_user_list')
+    parser.add_argument('--adobe-users',
+                        help="specify the adobe users to pull from UMAPI. Legal values are 'all' (the default), "
+                             "'group names' (one or more specified groups), 'mapped' (all groups listed in "
+                             "the configuration file)",
+                        nargs="+",
+                        metavar='all|mapped|group',
+                        dest='adobe_users')
     parser.add_argument('--connector',
                         help='specify a connector to use; default is LDAP (or CSV if --users file is specified)',
                         nargs='+',

user_sync/config.py

@@ -47,6 +47,7 @@ class ConfigLoader(object):
     invocation_defaults = {
         'adobe_only_user_action': ['preserve'],
         'adobe_only_user_list': None,
+        'adobe_users': ['all'],
         'connector': ['ldap'],
         'process_groups': False,
         'strategy': 'sync',
@@ -249,6 +250,29 @@ def load_invocation_options(self):
                                              (username_filter_pattern, e))
                 options['username_filter_regex'] = compiled_expression
 
+        # --adobe-users
+        if self.args['adobe_users'] is not None:
+            adobe_users_spec = options['adobe_users'] = self.args['adobe_users']
+        elif options['adobe_users'] is not None:
+            adobe_users_spec = options['adobe_users']
+        else:
+            adobe_users_spec = None
+
+        if adobe_users_spec is not None:
+            adobe_users_action = user_sync.helper.normalize_string(adobe_users_spec[0])
+            if adobe_users_action == 'all':
+                options['adobe_group_mapped'] = False
+            elif adobe_users_action == 'mapped':
+                options['adobe_group_mapped'] = True
+            elif adobe_users_action == 'group':
+                if len(adobe_users_spec) != 2:
+                    raise AssertionException(
+                        'You must specify the groups to read when using the adobe-users "group" option')
+                options['adobe_group_filter'] = []
+                for group in adobe_users_spec[1].split(','):
+                    options['adobe_group_filter'].append(user_sync.rules.AdobeGroup.create(group))
+            else:
+                raise AssertionException('Unknown option "%s" for adobe-users' % adobe_users_action)
         return options
 
     def get_logging_config(self):
@@ -552,6 +576,10 @@ def get_rule_options(self):
         if options.get('directory_group_mapped'):
             options['directory_group_filter'] = set(six.iterkeys(self.directory_groups))
 
+        # set the adobe group filter from the mapping, if requested.
+        if options.get('adobe_group_mapped') is True:
+            options['adobe_group_filter'] = set(user_sync.rules.AdobeGroup.iter_groups())
+
         return options
 
     def create_umapi_options(self, connector_config_sources):

user_sync/connector/umapi.py

@@ -135,10 +135,14 @@ def __init__(self, name, caller_options):
     def get_users(self):
         return list(self.iter_users())
 
-    def iter_users(self):
+    def iter_users(self, in_group=None):
         users = {}
         try:
-            for u in umapi_client.UsersQuery(self.connection):
+            if in_group:
+                u_query = umapi_client.UsersQuery(self.connection, in_group=in_group)
+            else:
+                u_query = umapi_client.UsersQuery(self.connection)
+            for u in u_query:
                 email = u['email']
                 if not (email in users):
                     users[email] = u

user_sync/rules.py

@@ -22,6 +22,7 @@
 import logging
 import six
 import re
+from itertools import chain
 
 import user_sync.connector.umapi
 import user_sync.error
@@ -37,6 +38,7 @@ class RuleProcessor(object):
     # rule processing option defaults
     # these are in alphabetical order!  Always add new ones that way!
     default_options = {
+        'adobe_group_filter': None,
         'after_mapping_hook': None,
         'default_country_code': None,
         'delete_strays': False,
@@ -850,14 +852,21 @@ def update_umapi_users_for_connector(self, umapi_info, umapi_connector):
         if self.will_process_strays:
             self.add_stray(umapi_info.get_name(), None)
 
+        if self.options['adobe_group_filter'] is not None:
+            umapi_users = self.get_umapi_user_in_groups(umapi_info, umapi_connector, self.options['adobe_group_filter'])
+        else:
+            umapi_users = umapi_connector.iter_users()
         # Walk all the adobe users, getting their group data, matching them with directory users,
         # and adjusting their attribute and group data accordingly.
-        for umapi_user in umapi_connector.iter_users():
+        for umapi_user in umapi_users:
             # get the basic data about this user; initialize change markers to "no change"
             user_key = self.get_umapi_user_key(umapi_user)
             if not user_key:
                 self.logger.warning("Ignoring umapi user with empty user key: %s", umapi_user)
                 continue
+            if umapi_info.get_umapi_user(user_key) is not None:
+                self.logger.debug("Ignoring umapi user. This user has already been processed: %s", umapi_user)
+                continue
             umapi_info.add_umapi_user(user_key, umapi_user)
             attribute_differences = {}
             current_groups = self.normalize_groups(umapi_user.get('groups'))
@@ -920,6 +929,14 @@ def map_email_override(self, umapi_user):
         if '@' in username and username != email:
             self.email_override[username] = email
 
+    @staticmethod
+    def get_umapi_user_in_groups(umapi_info, umapi_connector, groups):
+        umapi_users_iters = []
+        for group in groups:
+            if group.get_umapi_name() == umapi_info.get_name():
+                umapi_users_iters.append(umapi_connector.iter_users(in_group=group.get_group_name()))
+        return chain.from_iterable(umapi_users_iters)
+
     def is_umapi_user_excluded(self, in_primary_org, user_key, current_groups):
         if in_primary_org:
             self.primary_user_count += 1