diff --git a/Makefile b/Makefile index 8d4b2a1da..139d3b72e 100644 --- a/Makefile +++ b/Makefile @@ -8,7 +8,7 @@ ifeq ($(OS),Windows_NT) ifeq ($(rm_path),None) RM := rmdir /S /Q else - RM := $(rm_path) -rf + RM := $(rm_path) -rf endif else output_file_extension = "" diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index c7c20a55e..aead2c5ac 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,26 +1,22 @@ -# Release Notes for User Sync Tool Version 2.2.2 +# Release Notes for User Sync Tool Version 2.3 -These notes apply to v2.2.2 of 2017-11-19. +These notes apply to v2.3rc1 of 2017-11-20. ## New Features -[#294](https://github.com/adobe-apiplatform/user-sync.py/issues/294): Show statistics about users added to secondaries. +User Sync can now connect to Okta enterprise directories. Create an Okta configuration and use the new `--connector okta` command-line argument to select that connector. See [the docs](https://adobe-apiplatform.github.io/user-sync.py/en/user-manual/advanced_configuration.html#the-okta-connector) for details. -## Bug Fixes - -[#283](https://github.com/adobe-apiplatform/user-sync.py/issues/283): Don't import keyring unless needed. +There is a new command-line argument `--connector` for specifying whether to get directory information via LDAP file, by reading a CSV file, or via the Okta connector. The default connector is `ldap`. For CSV users, who formerly had to specify their input source with the `--users` argument, this optional argument offers the chance to specify `--users mapped` or `--users group ...` (since the CSV input can be specified with `--connector`). See [the docs](https://adobe-apiplatform.github.io/user-sync.py/en/user-manual/command_parameters.html) for details. -[#286](https://github.com/adobe-apiplatform/user-sync.py/issues/286): Allow specifying attributes for Adobe IDs. - -[#288](https://github.com/adobe-apiplatform/user-sync.py/issues/288): Escape special characters in user input to LDAP queries. +## Bug Fixes -[#293](https://github.com/adobe-apiplatform/user-sync.py/issues/293): Don't crash when existing users are added to secondaries. +[#305](https://github.com/adobe-apiplatform/user-sync.py/issues/305) General issues with Okta connector. -[#301](https://github.com/adobe-apiplatform/user-sync.py/issues/301): User Sync fails when adding more than 10 groups to a user. +[#306](https://github.com/adobe-apiplatform/user-sync.py/issues/306) v2.2.2 crashes if country code not specified. ## Compatibility with Prior Versions -There are no interface changes from prior versions. +All configuration and command-line arguments accepted in prior releases work in this release. The `--users file` argument is still accepted, and is equivalent to (although more limited than) specifying `--connector csv`. ## Known Issues diff --git a/docs/en/user-manual/advanced_configuration.md b/docs/en/user-manual/advanced_configuration.md index bd3dc0e48..369c7f38c 100644 --- a/docs/en/user-manual/advanced_configuration.md +++ b/docs/en/user-manual/advanced_configuration.md @@ -654,6 +654,33 @@ side, and removed users to be removed from the Adobe side. - Once the job has run, clear out the files (because their changes have been pushed) to prepare for the next batch. +## The Okta Connector + +In addition to LDAP and CSV, the User Sync tool supports [Okta](https://www.okta.com) as a source for user identity and product entitlement sync. Since Okta always uses email addresses as the unique ID for users, the Okta connector does not support username-based federation. + +Okta customers must obtain an API token for use with the Okta Users API. See the [Okta's Developer Documentation](http://developer.okta.com/docs/api/getting_started/api_test_client.html) +for more information. + +### Configuration + +To specify your Okta configuration file, use the key "okta" in `user-sync-config.yml`. + +```yaml +directory_users: + connectors: + okta: connector-okta.yml +``` + +There is a sample Okta connector file in the User Sync source tree. + +### Runtime + +In order to use the Okta connector, you will need to specify the `--connector okta` command-line parameter. (LDAP is the default connector.) In addition because the Okta connector does not support fetching all users, you must additionally specify a `--users` command line option of `group` or `mapped`. All other User Sync command-line parameters have their usual meaning. + +### Extensions + +Okta sync can use extended groups, attributes and after-mapping hooks. The names of extended attributes must be valid Okta profile fields. + --- [Previous Section](usage_scenarios.md) \| [Next Section](deployment_best_practices.md) diff --git a/docs/en/user-manual/command_parameters.md b/docs/en/user-manual/command_parameters.md index ddeabb916..ef49d3f95 100644 --- a/docs/en/user-manual/command_parameters.md +++ b/docs/en/user-manual/command_parameters.md @@ -39,6 +39,7 @@ specific behavior in various situations. | `--adobe-only-user-list` _filename_ | Specifies a file from which a list of users will be read. This list is used as the definitive list of "Adobe only" user accounts to be acted upon. One of the `--adobe-only-user-action` directives must also be specified and its action will be applied to user accounts in the list. The `--users` option is disallowed if this option is present: only account removal actions can be processed. | | `--config-file-encoding` _encoding_name_ | Optional. Specifies the character encoding for the contents of the configuration files themselves. This includes the main configuration file, "user-sync-config.yml" as well as other configuration files it may reference. Default is `utf8` for User Sync 2.2 and later and `ascii` for earlier versions.
Character encoding in the user source data (whether csv or ldap) is declared by the connector configurations, and that encoding can be different than the encoding used for the configuration files (e.g., you could have a latin-1 configuration file but a CSV source file that uses utf-8 encoding).
The available encodings are dependent on the Python version used; see the documentation [here](https://docs.python.org/2.7/library/codecs.html#standard-encodings) for more information. | | `--strategy sync`
`--strategy push` | Available in release 2.2 and later. Optional. Default operating mode is `--strategy sync`. Controls whether User Sync reads user information from Adobe and compares to the directory information and then issues updates to Adobe, or simply pushes the directory input to Adobe without considering the existing user information on Adobe. `sync` is the default and the subject of the description of most of this documentation. `push` is useful when there is a large number of users on the Adobe side (>30,000) and known additions or changes to a small number of users are desired, and the list of those users is available in a csv file or a specific directory group.
If `--strategy push` is specified, `--adobe-only-user-action` cannot be specified as the determination of adobe-only users is not made.
`--strategy push` will create new users, modify their group memberships for mapped groups only (if `--process-groups` is present), update user information (if `--update-user-info` is present), and will not remove users from the organization or delete their accounts. See [Handling Push Notifications](usage_scenarios.md#handling-push-notifications) for information on how to remove users via push notifications. | +| `--connector ldap`
`--connector okta`
`--connector csv` _filename_ | Available in release 2.3 and later. Optional. Specifies the directory connector to be used (defaults to LDAP). If you specify the use of a CSV input file with this argument, then you cannot also specify one with `--users`, but you can then specify other `--users` options (such as `mapped` or `group`) for use with the CSV file. (The Okta connector does not support `--users all`, so you must specify a `--users` option of `mapped` or `group` if you use the Okta connector.) {: .bordertablestyle } --- diff --git a/examples/config files - basic/1 user-sync-config.yml b/examples/config files - basic/1 user-sync-config.yml index d42788a2c..fafe66cfb 100644 --- a/examples/config files - basic/1 user-sync-config.yml +++ b/examples/config files - basic/1 user-sync-config.yml @@ -151,6 +151,11 @@ directory_users: # [Uncomment the next line if you have a custom csv configuration file.] #csv: "connector-csv.yml" + # (optional) okta (no default value) + # okta is a 3rd party federation provider compatible with Adobe Enterprise Federated ID. + # See https://developer.okta.com/ for Okta developer information. + # okta: "connector-okta.ytml" + # (optional) groups (no default value) # The groups setting specifies how groups in the enterprise directory map # to product configurations and user groups on the Adobe side (collectively diff --git a/examples/config files - basic/3 connector-ldap.yml b/examples/config files - basic/3 connector-ldap.yml index f425653dc..fe5397003 100755 --- a/examples/config files - basic/3 connector-ldap.yml +++ b/examples/config files - basic/3 connector-ldap.yml @@ -32,7 +32,6 @@ base_dn: "defines the base DN. e.g. DC=example,DC=com" # or network address) as the value below. #secure_password_key: ldap_password - # (optional) user_identity_type (default is inherited from main configuration) # user_identity_type specifies a default identity type for when directory users # are created on the Adobe side (one of adobeID, enterpriseID, federatedID). diff --git a/examples/config files - basic/5 connector-okta.yml b/examples/config files - basic/5 connector-okta.yml new file mode 100644 index 000000000..2490f6f0b --- /dev/null +++ b/examples/config files - basic/5 connector-okta.yml @@ -0,0 +1,36 @@ +# This is a sample configuration file for the okta connector type. +# +# Okta is an identity hosting company that supports being the Identity Provider +# for Adobe Enterprise Federated ID. +# +# This sample file contains all of the settable options for this protocol. +# It is recommended that you make a copy of this file and edit it for your needs. +# While you are at it, you will likely want to remove a lot of this commentary, +# in order to enhance the readability of your file. + +# connection settings (required) +# You must specify both of these settings. The token should be protected. +# For more information on getting an Okta API Token, see: +# http://developer.okta.com/docs/api/getting_started/getting_a_token.html +host: "sample-817042.oktapreview.com" +api_token: "00R_KJEaIcgAswrlO_sample_ZdgxC5scYZn8IZ-zi" + +# (required) group_filter_format (default given below) +# specifies the string format used to construct a group query. +# {group} is replaced with the name of the group to find. +group_filter_format: "{group}" + +# (required) all_users_filter (default given below) +# specifies the string filter used to find all users in the directory. +# Filter Examples: +# Filter user based on countryCode attribute in user profile +# all_users_filter: 'user.profile.countryCode == "MX"' +# Filter user based on status of ACTIVE +# all_users_filter: 'user.status == "ACTIVE"' +all_users_filter: 'user.status == "ACTIVE"' + +# (optional) default_identity_type (no default) +# specifies the identity type of the dashboard user to create. +# the valid values are: enterpriseID, federatedID +# If not specified, the default identity type from the main config file is used. +# user_identity_type: federatedID diff --git a/external/okta-0.0.3.1-py2.py3-none-any.whl b/external/okta-0.0.3.1-py2.py3-none-any.whl new file mode 100644 index 000000000..3a379b194 Binary files /dev/null and b/external/okta-0.0.3.1-py2.py3-none-any.whl differ diff --git a/setup.py b/setup.py index aee00ce4a..d133b14d0 100644 --- a/setup.py +++ b/setup.py @@ -45,13 +45,14 @@ license='MIT', packages=['user_sync', 'user_sync.connector'], install_requires=[ + 'keyring', + 'okta==0.0.3.1', + 'psutil', 'pycryptodome', 'pyldap==2.4.37', 'PyYAML', + 'six', 'umapi-client>=2.9', - 'psutil', - 'keyring', - 'six' ], extras_require={ ':sys_platform=="linux" or sys_platform=="linux2"':[ diff --git a/tests/connector/directory_okta_test.py b/tests/connector/directory_okta_test.py new file mode 100644 index 000000000..3703d459a --- /dev/null +++ b/tests/connector/directory_okta_test.py @@ -0,0 +1,399 @@ +# Copyright (c) 2016-2017 Adobe Systems Incorporated. All rights reserved. +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + + +import unittest +import mock +import logging +import okta +import json +import tests.helper +import copy + +from user_sync.error import AssertionException +from user_sync.connector.directory_okta import OktaDirectoryConnector, \ + OKTAValueFormatter, connector_load_users_and_groups + + +class TestOKTAValueFormatter(unittest.TestCase): + def test_get_extended_attribute_dict(self): + """Used to compare input and expected output data after OKTA formatting def Call""" + attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode'] + strtype = type('string') + expectedresult = {'countryCode': strtype, 'lastName': strtype, 'login': strtype, 'email': strtype, 'firstName': strtype} + + self.assertEqual(expectedresult, OKTAValueFormatter.get_extended_attribute_dict(attributes), 'Getting expected Output') + + +class TestOktaErrors(unittest.TestCase): + def setUp(self): + class MockResponse: + def __init__(self, status_code, data): + self.status_code = status_code + self.text = json.dumps(data) + + self.mock_response = MockResponse + + self.orig_directory_init = OktaDirectoryConnector.__init__ + + OktaDirectoryConnector.__init__ = mock.Mock(return_value=None) + directory = OktaDirectoryConnector({}) + directory.options = {'all_users_filter': None, 'group_filter_format': '{group}'} + directory.logger = mock.create_autospec(logging.Logger) + directory.groups_client = okta.UserGroupsClient('example.com', 'xyz') + + self.directory = directory + + def tearDown(self): + OktaDirectoryConnector.__init__ = self.orig_directory_init + + @mock.patch('okta.framework.ApiClient.requests') + def test_error_get_group(self, mock_requests): + # Mock an error response and make sure that the Okta connector catches the exception + # This will happen in the get_groups() method, which is the first time the UserGroupsClient is called + + mock_requests.get.return_value = self.mock_response(404, { + "errorCode": "E0000007", + "errorSummary": "Not found: Resource not found: users (UserGroup)", + "errorLink": "E0000007", + "errorId": "oaepKQbQ-_FQ7y5YxDQWFw5Vg", + "errorCauses": [] + }) + + self.assertRaises(AssertionException, + connector_load_users_and_groups, self.directory, ['group1', 'group2'], []) + + +class TestOktaGroupFilter(unittest.TestCase): + def setUp(self): + class MockResponse: + def __init__(self, status_code, data): + self.status_code = status_code + self.text = json.dumps(data) + + self.mock_response = MockResponse + + self.orig_directory_init = OktaDirectoryConnector.__init__ + + OktaDirectoryConnector.__init__ = mock.Mock(return_value=None) + directory = OktaDirectoryConnector({}) + + directory.logger = mock.create_autospec(logging.Logger) + directory.groups_client = okta.UserGroupsClient('example.com', 'xyz') + + self.directory = directory + + def tearDown(self): + OktaDirectoryConnector.__init__ = self.orig_directory_init + + @mock.patch('okta.framework.ApiClient.requests') + def test_success_group_filter(self, mock_requests): + # This test success group filter with valid Group + # This should return Okta GroupsClient Object and Contained property Profile. + + mock_requests.get.return_value = self.mock_response(200, [ + {"id": "00g9sq2jcqpk3LwCV0h7", + "objectClass": ["okta:user_group"], "type": "OKTA_GROUP", + "profile": {"name": "Group 1", "description": "null"}}]) + directory = self.directory + directory.options = {'all_users_filter': None, 'group_filter_format': '{group}'} + result = directory.find_group("Group 1") + self.assertEqual(result.profile.name, "Group 1") + + @mock.patch('okta.framework.ApiClient.requests') + def test_bad_group_filter_1(self, mock_requests): + # Test scenario where Group Filter is bad + # This will not return any group because it can't be find. + + mock_requests.get.return_value = self.mock_response(200, []) + directory = self.directory + directory.options = {'all_users_filter': None, + 'group_filter_format': '(BADFILTER){group}'} + result = directory.find_group("Group 1") + self.assertEqual(result, None) + + def test_bad_group_filter_2(self): + # Test another scenario of bad group filter - {groupA} instead of {group} + # This should throw an exception + + directory = self.directory + directory.options = {'all_users_filter': None, + 'group_filter_format': '{groupA}'} + self.assertRaises(AssertionException, directory.find_group, "Group 1") + + +class TestOktaUsersGroups(unittest.TestCase): + def setUp(self): + self.orig_directory_init = OktaDirectoryConnector.__init__ + + OktaDirectoryConnector.__init__ = mock.Mock(return_value=None) + directory = OktaDirectoryConnector({}) + directory.options = {'all_users_filter': None, 'group_filter_format': '{group}'} + directory.logger = mock.create_autospec(logging.Logger) + directory.groups_client = okta.UserGroupsClient('example.com', 'xyz') + + self.directory = directory + + def tearDown(self): + OktaDirectoryConnector.__init__ = self.orig_directory_init + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.iter_group_members') + def test_found_user_single_group(self, mock_members): + # There are 1 users in the Group. This test should return 1 users. + groups = ['group1'] + test_user = tests.helper.create_test_user_uid() + mock_members.return_value = [test_user] + directory = self.directory + results = directory.load_users_and_groups(groups, []) + self.assertEqual(len(list(results)), 1) + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.iter_group_members') + def test_found_user_multiple_groups(self, mock_members): + # There are 1 users in each Group. This test should total return 2 users. + groups = ['group1', 'group2'] + test_users = [] + for group in groups: + test_users.append([tests.helper.create_test_user_uid()]) + mock_members.side_effect = test_users + directory = self.directory + results = directory.load_users_and_groups(groups, []) + self.assertEqual(len(list(results)), 2) + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.iter_group_members') + def test_found_user_single_group_multiple_user(self, mock_members): + # There are 5 users in the Group. This test should return 5 users. + groups = ['group1'] + test_users = [] + user_count = 0 + while user_count < 5: + test_users.append(tests.helper.create_test_user_uid()) + user_count = user_count + 1 + mock_members.return_value = test_users + directory = self.directory + results = directory.load_users_and_groups(groups, []) + self.assertEqual(len(list(results)), 5) + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.iter_group_members') + def test_found_user_multiple_groups_multiple_user(self, mock_members): + # There are 5 users in each Group. This test should total return 10 users. + groups = ['group1', 'group2'] + total_test_users = [] + for group in groups: + test_users = [] + user_count = 0 + while user_count < 5: + test_users.append(tests.helper.create_test_user_uid()) + user_count = user_count + 1 + total_test_users.append(test_users) + mock_members.side_effect = total_test_users + directory = self.directory + results = directory.load_users_and_groups(groups, []) + self.assertEqual(len(list(results)), 10) + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.iter_group_members') + def test_no_user_single_group(self, mock_members): + # There are 0 users in the Group. This test should return 0 users. + groups = ['group1'] + test_users = [] + mock_members.return_value = test_users + directory = self.directory + results = directory.load_users_and_groups(groups, []) + self.assertEqual(len(list(results)), 0) + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.iter_group_members') + def test_no_user_multiple_groups(self, mock_members): + # There are 0 users in each Group. This test should total return 0 users. + groups = ['group1', 'group2'] + total_test_users = [[], []] + mock_members.side_effect = total_test_users + directory = self.directory + results = directory.load_users_and_groups(groups, []) + self.assertEqual(len(list(results)), 0) + + +# Used to Test UserGroupsClient , iter_group_members Definations + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.iter_group_members') + def test_same_user_in_multiple_groups(self, mock_members): + #User A in both Group 1 and Group 2. + groups = ['group1', 'group2'] + test_user = tests.helper.create_test_user_uid() + mock_members.side_effect = [[test_user], [copy.deepcopy(test_user)]] + directory = self.directory + result = directory.load_users_and_groups(groups, []) + result_list = list(result) + self.assertEqual(result_list[0]['groups'], ['group1','group2']) + +# Used to Test UserGroupsClient , iter_group_members Definitions +class TestOktaIterGroupMember(unittest.TestCase): + def setUp(self): + class MockResponse: + def __init__(self, status_code, data): + self.status_code = status_code + self.text = json.dumps(data) + self.links = {} + + self.mock_response = MockResponse + self.orig_directory_init = OktaDirectoryConnector.__init__ + OktaDirectoryConnector.__init__ = mock.Mock(return_value=None) + directory = OktaDirectoryConnector({}) + directory.logger = mock.create_autospec(logging.Logger) + directory.groups_client = okta.UserGroupsClient('example.com', 'xyz') + directory.user_identity_type = 'enterpriseID' + self.directory = directory + + def tearDown(self): + OktaDirectoryConnector.__init__ = self.orig_directory_init + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.find_group') + @mock.patch('okta.framework.ApiClient.requests') + def test_success_extended_attribute_key(self, mock_requests, mock_find_group): + # This test the extended_attribute feature + # Should return requested Extended Attribute Key in result['source_attributes'] + + mock_requests.get.return_value = self.mock_response(200, [{"id": "00u9s60df0cO5cU3Y0h7", + "status": "ACTIVE", + "profile": {"login": "testuser@xyz.com", + "mobilePhone": None, + "email": "testuser@xyz.com", + "secondEmail": None, "firstName": "Test", + "lastName": "User", + "countryCode": "US", + "additionalTest": "TestValue1234"}}]) + mockID = mock.Mock() + mockID.id = "TestGroup1" + mock_find_group.return_value = mockID + + directory = self.directory + directory.options = {'all_users_filter': 'user.status == "ACTIVE"', 'group_filter_format': '{group}'} + extended_attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode', 'additionalTest'] + iterGroupResponse = directory.iter_group_members("testGroup",directory.options['all_users_filter'], extended_attributes) + temp_var = list(iterGroupResponse) + + self.assertIn('additionalTest', temp_var[0]['source_attributes']) + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.find_group') + @mock.patch('okta.framework.ApiClient.requests') + def test_success_extended_attribute_value(self, mock_requests, mock_find_group): + # This test the extended_attribute feature + # Should return requested Extended Attribute value in result['source_attributes'] + + mock_requests.get.return_value = self.mock_response(200, [{"id": "00u9s60df0cO5cU3Y0h7", + "status": "ACTIVE", + "profile": {"login": "testuser@xyz.com", + "mobilePhone": None, + "email": "testuser@xyz.com", + "secondEmail": None, "firstName": "Test", + "lastName": "User", + "countryCode": "US", + "additionalTest": "TestValue1234"}}]) + mockID = mock.Mock() + mockID.id = "TestGroup1" + mock_find_group.return_value = mockID + + directory = self.directory + directory.options = {'all_users_filter': 'user.status == "ACTIVE"', 'group_filter_format': '{group}'} + extended_attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode', 'additionalTest'] + iterGroupResponse = directory.iter_group_members("testGroup", directory.options['all_users_filter'], + extended_attributes) + temp_var = list(iterGroupResponse) + + self.assertEqual(temp_var[0]['source_attributes']['additionalTest'], 'TestValue1234') + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.find_group') + @mock.patch('okta.framework.ApiClient.requests') + def test_non_existence_extended_attribute_key(self, mock_requests, mock_find_group): + # This test the extended_attribute feature + # user object should contain badattribute eventhough it does not exist in Okta User Profile + + mock_requests.get.return_value = self.mock_response(200, [{"id": "00u9s60df0cO5cU3Y0h7", + "status": "ACTIVE", + "profile": {"login": "testuser@xyz.com", + "mobilePhone": None, + "email": "testuser@xyz.com", + "secondEmail": None, "firstName": "Test", + "lastName": "User", + "countryCode": "US", + "additionalTest": "TestValue1234"}}]) + mockID = mock.Mock() + mockID.id = "TestGroup1" + mock_find_group.return_value = mockID + + directory = self.directory + directory.options = {'all_users_filter': 'user.status == "ACTIVE"', 'group_filter_format': '{group}'} + extended_attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode', 'badattribute'] + iterGroupResponse = directory.iter_group_members("testGroup", directory.options['all_users_filter'], + extended_attributes) + temp_var = list(iterGroupResponse) + + self.assertIn('badattribute', temp_var[0]['source_attributes']) + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.find_group') + @mock.patch('okta.framework.ApiClient.requests') + def test_non_existence_extended_attribute_value(self, mock_requests, mock_find_group): + # This test the extended_attribute feature + # user object should contain badattribute key eventhough it does not exist in Okta User Profile + + mock_requests.get.return_value = self.mock_response(200, [{"id": "00u9s60df0cO5cU3Y0h7", + "status": "ACTIVE", + "profile": {"login": "testuser@xyz.com", + "mobilePhone": None, + "email": "testuser@xyz.com", + "secondEmail": None, "firstName": "Test", + "lastName": "User", + "countryCode": "US", + "additionalTest": "TestValue1234"}}]) + mockID = mock.Mock() + mockID.id = "TestGroup1" + mock_find_group.return_value = mockID + + directory = self.directory + directory.options = {'all_users_filter': 'user.status == "ACTIVE"', 'group_filter_format': '{group}'} + extended_attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode', 'badattribute'] + iterGroupResponse = directory.iter_group_members("testGroup", directory.options['all_users_filter'], + extended_attributes) + temp_var = list(iterGroupResponse) + self.assertEqual(temp_var[0]['source_attributes']['badattribute'], None) + + @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.find_group') + @mock.patch('okta.framework.ApiClient.requests') + def test_invalid_missing_profile_user(self, mock_requests, mock_find_group): + # This test the extended_attribute feature + # user object should contain badattribute value of None eventhough it does not exist in Okta User Profile + + mock_requests.get.return_value = self.mock_response(200, [{"id": "00u9s60df0cO5cU3Y0h7", + "status": "ACTIVE", + "profile": {"login": "testuser@xyz.com", + "mobilePhone": None, + "email": "testuser@xyz.com", + "secondEmail": None, "firstName": "Test", + "lastName": None, + "countryCode": "US", + "additionalTest": "TestValue1234"}}]) + mockID = mock.Mock() + mockID.id = "TestGroup1" + mock_find_group.return_value = mockID + + directory = self.directory + directory.options = {'all_users_filter': 'user.status == "ACTIVE"', 'group_filter_format': '{group}'} + extended_attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode', 'badattribute'] + iterGroupResponse = directory.iter_group_members("testGroup", directory.options['all_users_filter'], + extended_attributes) + temp_var = list(iterGroupResponse) + self.assertEqual(temp_var[0]['source_attributes']['badattribute'], None) diff --git a/tests/helper.py b/tests/helper.py index 20c8d7914..250cffe5a 100644 --- a/tests/helper.py +++ b/tests/helper.py @@ -51,6 +51,22 @@ def create_test_user(groups): } return user +def create_test_user_uid(): + global next_user_id + firstName = 'User_%d' % next_user_id + uid = '0000%s' % next_user_id + next_user_id += 1 + user = { + 'uid': uid, + 'identity_type': 'enterpriseID', + 'firstname': firstName, + 'lastname': 'Test', + 'email': '%s_email@example.com' % firstName, + 'country': 'CA' if (next_user_id % 2 == 0) else 'US', + 'groups': [] + } + return user + def assert_equal_users(unit_test, expected_users, actual_users): actual_users_by_email = dict((user['email'], user) for user in actual_users) diff --git a/user_sync/app.py b/user_sync/app.py index 790b3cfe6..7a2085ca0 100644 --- a/user_sync/app.py +++ b/user_sync/app.py @@ -65,12 +65,13 @@ def process_args(): action='store_true', dest='test_mode') parser.add_argument('-c', '--config-filename', help='main config filename. (default: "%(default)s")', - default=user_sync.config.DEFAULT_MAIN_CONFIG_FILENAME, metavar='filename', dest='config_filename') + default=user_sync.config.DEFAULT_MAIN_CONFIG_FILENAME, + metavar='filename', dest='config_filename') parser.add_argument('--users', help="specify the users to be considered for sync. Legal values are 'all' (the default), " "'group names' (one or more specified groups), 'mapped' (all groups listed in " "the configuration file), 'file f' (a specified input file).", - nargs="*", metavar=('all|file|mapped|group', 'arg1'), dest='users') + nargs="+", metavar=('all|file|mapped|group', 'arg1'), dest='users', default=['all']) parser.add_argument('--user-filter', help='limit the selected set of users that may be examined for syncing, with the pattern ' 'being a regular expression.', @@ -110,6 +111,10 @@ def process_args(): help="whether to fetch and sync the Adobe directory against the customer directory " "or just to push each customer user to the Adobe side. Default is to fetch and sync.", dest='strategy', metavar='sync|push', default='sync') + parser.add_argument('--connector', + help='specify a connector to use; default is LDAP (or CSV if --users file is specified)', + nargs='+', metavar=['ldap|okta|csv','path-to-file.csv'], + dest='connector_spec', default=['ldap']) return parser.parse_args() @@ -209,7 +214,7 @@ def begin_work(config_loader): rule_processor = user_sync.rules.RuleProcessor(rule_config) if len(directory_groups) == 0 and rule_processor.will_manage_groups(): - logger.warn('no groups mapped in config file') + logger.warning('No group mapping specified in configuration but --process-groups requested on command line') rule_processor.run(directory_groups, directory_connector, umapi_connectors) @@ -233,8 +238,8 @@ def create_config_loader_options(args): """ config_options = { 'delete_strays': False, - 'directory_connector_module_name': None, 'directory_connector_overridden_options': None, + 'directory_connector_type': None, 'directory_group_filter': None, 'directory_group_mapped': False, 'disentitle_strays': False, @@ -249,23 +254,39 @@ def create_config_loader_options(args): 'username_filter_regex': None, } + # --connector + connector_type = user_sync.helper.normalize_string(args.connector_spec.pop(0)) + if connector_type in ["ldap", "okta"]: + if args.connector_spec: + raise AssertionException("Must not specify file (%s) with --connector %s" % + (args.connector_spec[0], connector_type)) + config_options['directory_connector_type'] = connector_type + elif connector_type == "csv": + if len(args.connector_spec) != 1: + raise AssertionException("Must specify a single file with CSV connector") + config_options['directory_connector_type'] = 'csv' + config_options['directory_connector_overridden_options'] = {'file_path': args.connector_spec.pop(0)} + else: + raise AssertionException("Unknown connector type: %s" % connector_type) + # --users users_args = args.users users_action = None if not users_args else user_sync.helper.normalize_string(users_args.pop(0)) if users_action is None or users_action == 'all': - config_options['directory_connector_module_name'] = 'user_sync.connector.directory_ldap' + if config_options['directory_connector_type'] == 'okta': + raise AssertionException('Okta connector module does not support "--users all"') elif users_action == 'file': + if config_options['directory_connector_type'] == 'csv': + raise AssertionException('You cannot specify "--users file" and "--connector csv file"') if len(users_args) == 0: raise AssertionException('Missing file path for --users %s [file_path]' % users_action) - config_options['directory_connector_module_name'] = 'user_sync.connector.directory_csv' + config_options['directory_connector_type'] = 'csv' config_options['directory_connector_overridden_options'] = {'file_path': users_args.pop(0)} elif users_action == 'mapped': - config_options['directory_connector_module_name'] = 'user_sync.connector.directory_ldap' config_options['directory_group_mapped'] = True elif users_action == 'group': if len(users_args) == 0: raise AssertionException('Missing groups for --users %s [groups]' % users_action) - config_options['directory_connector_module_name'] = 'user_sync.connector.directory_ldap' config_options['directory_group_filter'] = users_args.pop(0).split(',') else: raise AssertionException('Unknown argument --users %s' % users_action) @@ -308,7 +329,7 @@ def create_config_loader_options(args): if config_options.get('stray_list_output_path'): raise AssertionException('You cannot specify both --adobe-only-user-list and --output-adobe-users') # don't read the directory when processing from the stray list - config_options['directory_connector_module_name'] = None + config_options['directory_connector_type'] = None logger.info('--adobe-only-user-list specified, so not reading or comparing directory and Adobe users') config_options['stray_list_input_path'] = stray_list_input_path diff --git a/user_sync/config.py b/user_sync/config.py index be2f01824..81ee18525 100644 --- a/user_sync/config.py +++ b/user_sync/config.py @@ -46,8 +46,8 @@ def __init__(self, caller_options): # these are in alphabetical order! Always add new ones that way! 'delete_strays': False, 'config_file_encoding': 'utf8', - 'directory_connector_module_name': None, 'directory_connector_overridden_options': None, + 'directory_connector_type': None, 'directory_group_filter': None, 'directory_group_mapped': False, 'disentitle_strays': False, @@ -127,18 +127,22 @@ def get_directory_connector_module_name(self): """ :rtype str """ - options = self.options - return options['directory_connector_module_name'] + connector_type = self.options.get('directory_connector_type') + if connector_type: + return 'user_sync.connector.directory_' + connector_type + else: + return None def get_directory_connector_configs(self): connectors_config = None directory_config = self.main_config.get_dict_config('directory_users', True) if directory_config is not None: connectors_config = directory_config.get_dict_config('connectors', True) - # make sure neither ldap nor csv connectors get reported as unused + # make sure none of the standard connectors get reported as unused if connectors_config: connectors_config.get_list('ldap', True) connectors_config.get_list('csv', True) + connectors_config.get_list('okta', True) return connectors_config def get_directory_connector_options(self, connector_name): diff --git a/user_sync/connector/directory_okta.py b/user_sync/connector/directory_okta.py new file mode 100644 index 000000000..51813155b --- /dev/null +++ b/user_sync/connector/directory_okta.py @@ -0,0 +1,281 @@ +# Copyright (c) 2016-2017 Adobe Systems Incorporated. All rights reserved. +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +import okta +import six +from okta.framework.OktaError import OktaError + +import user_sync.config +import user_sync.connector.helper +import user_sync.helper +import user_sync.identity_type +from user_sync.error import AssertionException + + +def connector_metadata(): + metadata = { + 'name': OktaDirectoryConnector.name + } + return metadata + + +def connector_initialize(options): + """ + :type options: dict + """ + state = OktaDirectoryConnector(options) + return state + + +def connector_load_users_and_groups(state, groups, extended_attributes, all_users): + """ + :type state: OktaDirectoryConnector + :type groups: list(str) + :type extended_attributes: list(str) + :type all_users: bool + :rtype (bool, iterable(dict)) + """ + + return state.load_users_and_groups(groups, extended_attributes, all_users) + + +class OktaDirectoryConnector(object): + name = 'okta' + + def __init__(self, caller_options): + caller_config = user_sync.config.DictConfig('%s configuration' % self.name, caller_options) + builder = user_sync.config.OptionsBuilder(caller_config) + builder.set_string_value('group_filter_format', + '{group}') + builder.set_string_value('all_users_filter', + 'user.status == "ACTIVE"') + builder.set_string_value('user_identity_type', None) + builder.set_string_value('logger_name', self.name) + host = builder.require_string_value('host') + api_token = builder.require_string_value('api_token') + + options = builder.get_options() + + self.users_client = None + self.groups_client = None + self.logger = logger = user_sync.connector.helper.create_logger(options) + self.user_identity_type = user_sync.identity_type.parse_identity_type(options['user_identity_type']) + self.options = options + caller_config.report_unused_values(logger) + + if not host.startswith('https://'): + if "://" in host: + raise AssertionException("Okta protocol must be https") + host = "https://" + host + + self.user_by_uid = {} + + logger.debug('%s initialized with options: %s', self.name, options) + + logger.info('Connecting to: %s', host) + + try: + self.users_client = okta.UsersClient(host, api_token) + self.groups_client = okta.UserGroupsClient(host, api_token) + except OktaError as e: + raise AssertionException("Error connecting to Okta: %s" % e) + + logger.info('Connected') + + def load_users_and_groups(self, groups, extended_attributes, all_users): + """ + :type groups: list(str) + :type extended_attributes: list(str) + :type all_users: bool + :rtype (bool, iterable(dict)) + """ + if all_users: + raise AssertionException("Okta connector has no notion of all users, please specify a --users group") + + options = self.options + all_users_filter = options['all_users_filter'] + + self.logger.info('Loading users...') + self.user_by_uid = user_by_uid = {} + + for group in groups: + total_group_members = 0 + total_group_users = 0 + for user in self.iter_group_members(group, all_users_filter, extended_attributes): + total_group_members += 1 + + uid = user.get('uid') + if user and uid: + if uid not in user_by_uid: + user_by_uid[uid] = user + total_group_users += 1 + user_groups = user_by_uid[uid]['groups'] + if group not in user_groups: + user_groups.append(group) + + self.logger.debug('Group %s members: %d users: %d', group, total_group_members, total_group_users) + + return six.itervalues(user_by_uid) + + def find_group(self, group): + """ + :type group: str + :rtype UserGroup + """ + group = group.strip() + options = self.options + group_filter_format = options['group_filter_format'] + try: + results = self.groups_client.get_groups(query=group_filter_format.format(group=group)) + except OktaError as e: + self.logger.warning("Unable to query group") + raise AssertionException("Okta error querying for group: %s" % e) + + if results is None: + self.logger.warning("No group found for: %s", group) + else: + for result in results: + if result.profile.name == group: + return result + + return None + + def iter_group_members(self, group, filter_string, extended_attributes): + """ + :type group: str + :type filter_string: str + :type extended_attributes: list + :rtype iterator(str, str) + """ + + user_attribute_names = ["firstName", "lastName", "login", "email", "countryCode"] + extended_attributes = list(set(extended_attributes) - set(user_attribute_names)) + user_attribute_names.extend(extended_attributes) + + res_group = self.find_group(group) + if res_group: + try: + attr_dict = OKTAValueFormatter.get_extended_attribute_dict(user_attribute_names) + members = self.groups_client.get_group_all_users(res_group.id, attr_dict) + except OktaError as e: + self.logger.warning("Unable to get_group_users") + raise AssertionException("Okta error querying for group users: %s" % e) + # Filtering users based all_users_filter query in config + for member in self.filter_users(members, filter_string): + profile = member.profile + if not profile.email: + self.logger.warning('No email attribute for login: %s', profile.login) + continue + + user = self.convert_user(member, extended_attributes) + if not user: + continue + yield (user) + else: + self.logger.warning("No group found for: %s", group) + + def convert_user(self, record, extended_attributes): + profile = record.profile + + source_attributes = {} + user = user_sync.connector.helper.create_blank_user() + + source_attributes['id'] = user['uid'] = record.id + source_attributes['email'] = user['email'] = profile.email + + source_attributes['identity_type'] = user_identity_type = self.user_identity_type + if not user_identity_type: + user['identity_type'] = self.user_identity_type + else: + try: + user['identity_type'] = user_sync.identity_type.parse_identity_type(user_identity_type) + except AssertionException as e: + self.logger.warning('Skipping user %s: %s', profile.login, e) + return None + + source_attributes['login'] = profile.login + + user['username'] = '' + + if profile.firstName: + source_attributes['firstName'] = user['firstname'] = profile.firstName + else: + source_attributes['firstName'] = None + + if profile.lastName: + source_attributes['lastName'] = user['lastname'] = profile.lastName + else: + source_attributes['lastName'] = None + + if profile.countryCode: + source_attributes['countryCode'] = user['country'] = profile.countryCode + else: + source_attributes['countryCode'] = None + + if extended_attributes: + for extended_attribute in extended_attributes: + if extended_attribute not in source_attributes: + if hasattr(profile, extended_attribute): + extended_attribute_value = getattr(profile, extended_attribute) + source_attributes[extended_attribute] = extended_attribute_value + else: + source_attributes[extended_attribute] = None + + user['source_attributes'] = source_attributes.copy() + return user + + def iter_search_result(self, filter_string, attributes): + """ + type: filter_string: str + type: attributes: list(str) + """ + + attr_dict = OKTAValueFormatter.get_extended_attribute_dict(attributes) + + try: + self.logger.info("Calling okta SDK get_users with the following %s", filter_string) + if attr_dict: + users = self.users_client.get_all_users(query=filter_string, extended_attribute=attr_dict) + else: + users = self.users_client.get_all_users(query=filter_string) + except OktaError as e: + self.logger.warning("Unable to query users") + raise AssertionException("Okta error querying for users: %s" % e) + return users + + def filter_users(self, users, filter_string): + try: + return list(filter(lambda user: eval(filter_string), users)) + except SyntaxError as e: + raise AssertionException("Invalid syntax in predicate (%s): cannot evaluate" % filter_string) + except Exception as e: + raise AssertionException("Error filtering with predicate (%s): %s" % (filter_string, e)) + + +class OKTAValueFormatter(object): + @staticmethod + def get_extended_attribute_dict(attributes): + + attr_dict = {} + for attribute in attributes: + if attribute not in attr_dict: + attr_dict.update({attribute: str}) + + return attr_dict diff --git a/user_sync/rules.py b/user_sync/rules.py index 4db45ffd6..330c2235a 100644 --- a/user_sync/rules.py +++ b/user_sync/rules.py @@ -616,6 +616,7 @@ def create_umapi_commands_for_directory_user(self, directory_user, do_update=Fal Update the attributes of an existing user if do_update is True. :type directory_user: dict :type do_update: bool + :return user_sync.connector.umapi.Commands (or None if there's an error) """ identity_type = self.get_identity_type_from_directory_user(directory_user) commands = user_sync.connector.umapi.Commands(identity_type, directory_user['email'], @@ -631,7 +632,7 @@ def create_umapi_commands_for_directory_user(self, directory_user, do_update=Fal country = 'UD' else: self.logger.error("User cannot be added without a specified country code: %s", directory_user) - return + return None attributes['country'] = country if attributes.get('firstname') is None: attributes.pop('firstname', None) @@ -654,13 +655,14 @@ def create_umapi_user(self, user_key, groups_to_add, umapi_info, umapi_connector If we are pushing, we also remove the user from any mapped groups not in groups_to_add. (This way, when we push blindly, we manage the entire set of mapped groups.) :type user_key: str - :type update_attributes: bool :type groups_to_add: set :type umapi_info: UmapiTargetInfo :type umapi_connector: user_sync.connector.umapi.UmapiConnector """ directory_user = self.directory_user_by_user_key[user_key] commands = self.create_umapi_commands_for_directory_user(directory_user, self.will_update_user_info(umapi_info)) + if not commands: + return if self.will_manage_groups(): if self.push_umapi: groups_to_remove = umapi_info.get_mapped_groups() - groups_to_add diff --git a/user_sync/version.py b/user_sync/version.py index dbf184b9c..9f8a59545 100644 --- a/user_sync/version.py +++ b/user_sync/version.py @@ -18,4 +18,4 @@ # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # SOFTWARE. -__version__ = '2.2.2' +__version__ = '2.3rc1'