Dataset_librarian code updates (#1187)

* Changed location of installation of GCP cli

* Adding packaging tests

* Test Unit Tests

* Test Functional Tests

* Fix for downloader

* Adding an specific version to not have issues on instalation

* Interconnection experiment

* Test updated workflow

* Fixed out_report.xml path

* Interconnection for GCP

* Adding updates for wheel and setuptools

* Experiment requirements

* Changes on interconnection proyect and setup

* Changes to ignore copied files

* Adding fix to handle encoding issues

* Adding changes to readme

* Taking current folder to get packages

* Fix typo

Signed-off-by: Felipe Leza Alvarez <[email protected]>

* Inter connection sample

* Hot fix for python 3.8

* Fix for setup code

* ignoring sample file

* Renaming correctly interconnection

* Fix on upgrade for pip, wheel and setuptools

* Interoperability POC sample fix

* Fix for .sample copy

* Setup for bash

* Hotfix: change "test" strings for "test_unittest" so that unit tests won't use files used by samples.

* Fixing requirements and setup

* Removing wheel update from setup

* Fixes the problem to upload dataset to GCP.

* Fixed unwanted changes on main branch

* Fixed aws functional test name

* Upload a folder using the aws connector

* Last changes on Interoperability are applied

* Updated Licence

* Removinb code of conduct reference, we have not

* Skips row 0 from excel when creating dataframe.

* Refactor

* delete unused gitignore

* change access keys format

* Refactored names

* first version license header

* Remoivng readme files on this branch

* Deleting files created from setup

* headers

* Adding headers into main packages

* Adding headers on sample code

* Updating files for publishing

* Get GCP credentials

* Changed setup.sh file

* Removed init file on interoperability folder

* Big refactoring, moving data_connector into datasets

* Complete sample link

* Test WF

* Fixed path for unit tests

* Changed trigger to PR

* Create sample link

* Merging readmes

* Removing licence only for data connector

* Missed recursive flag

* delete unused names

* Ignore outputs of jupyter notebook

* Removing commented block

* Removing deprecated folder

* Include headers in init files

* Adding headers in all tests

* Removing coverage omit from tox configuration file (This only works on MZ env)

* Removed gcp auth commands instead of commenting them

* Adding headers

* Removed sensitive information on gcp

* Removing values to make it more easy for usersfill it

* Solving names to public repo

* Removing default values

* Removing sample values

* Removing values

* Fixing path on script

* Removing extra files

* add security file

* Updating files and structure for publishing

* Updates for packaging

* Updating readme

* Updating metadata

* Updating source code

* Updating git ignore

* Updating git ignore

* updating metadata

* Updating gitgitnore

* removing extracted files

* Updating repo

* removing dataset egg info

* Updating file permissions

* Updating file permissions

* removing key

* Updating imports

* Updating reame

* Removing error

* removing data_connector changes

* Updating conda recipes

* Fixed bugs in setup.sh
Added azure src and dependencies.

* Removing conda folders

* Updating blank space at the end of files

* Updating readme

* Validation/scans (#56)

* Fixed dataset_api requirements file

* Merging from data_connector

* Updating gitignore

* Returning depencencies

* Returning training code

* Creating and re naming sample files

* Adding format

* New readme proposals

* Fix on toml to avoid refactor

* Readme agenda

* Conda folder is unevitable

* Exclud conda and egg folders

* Adding badages in main readme... will see if we should use rst format for main readme only

* Simple entry point for sample doc

* Change header for sub_linked section

* Modifications to current lass invocation

* Adding relative link to documentation in AWS main readme file

* Terms and conditions requirements update

* Changes on Azure Readmi file

* Removing previous terms and conditions

* Updating path for datasets_urls

* Updating path for datasets_urls

* Removing data connector changes

* Updating blank last line

* Updated documentation with curren code functionality

* Update documentation

* Added code sample for upload, download and list blobs for oauth

* first definition on dcp readme for bigquery

* Sample connection with oauth

* Adding readme sample for gcp service account connection with GCP

* Connection documentation finished

* Updating TPP file

* updating with feedback

---------

Signed-off-by: gera-aldama <[email protected]>
Signed-off-by: Felipe Leza Alvarez <[email protected]>
Co-authored-by: Miguel Pineda <[email protected]>
Co-authored-by: Gerardo Dominguez <[email protected]>
Co-authored-by: gera-aldama <[email protected]>
Co-authored-by: Felipe Leza Alvarez <[email protected]>
Co-authored-by: aagalleg <[email protected]>
Co-authored-by: Leza Alvarez, Felipe <[email protected]>
Co-authored-by: ma-pineda <[email protected]>

Author: 5 people

datasets/data_connector/.gitignore

@@ -12,9 +12,11 @@ __pycache__/
 .venv
 
 # Folders
-local_channel/
 dist/
 inspect_package-conda/
 inspect_package-pip/
-conda_package/
 build/
+
+# conda
+conda/local_channel/
+conda/extracted

datasets/data_connector/MANIFEST.in

@@ -3,4 +3,6 @@ prune docker
 prune samples
 prune tests
 prune docs
-exclude *.txt __init__.py .gitignore poetry.lock tox.ini MANIFEST.in
+prune conda
+prune data_connector.egg-info
+exclude *.txt __init__.py .gitignore poetry.lock tox.ini MANIFEST.in

datasets/data_connector/README.md

@@ -1,13 +1,27 @@
+# Cloud Data Connector
+
+[![Intel: AI](https://img.shields.io/badge/intel-AI-0071C5)](https://www.intel.com/content/www/us/en/developer/topic-technology/artificial-intelligence/overview.html)
+[![Python](https://img.shields.io/badge/Python-3.8/3.9/3.10-green)](https://www.intel.com/content/www/us/en/developer/topic-technology/artificial-intelligence/overview.html)
+[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)
+[![security: SNYK](https://img.shields.io/badge/Security-SNYK-yellow)](https://github.com/PyCQA/bandit)
+[![security: BDBA](https://img.shields.io/badge/Security-BDBA-yellow)](https://github.com/PyCQA/bandit)
+[![security: Checkmarks](https://img.shields.io/badge/Security-Checkmarks-yellow)](https://github.com/PyCQA/bandit)
+
+
+_____
 ## Overview
 ---
 data_connector is a tool to connect to AzureML, Azure blob, GCP storage, GCP Big Query  and AWS storage S3. 
 The goal is provide all cloud managers in one place and provide documentation for an easy integration.
 
-For more details, visit the [Data Connector](repo link) GitHub repository.
+For more details, visit the [Data Connector](https://github.com/IntelAI/models/tree/master/datasets/data_connector) GitHub repository.
+<br/><br/>
 
 ## Hardware Requirements
 ---
 The hardware should comply with the same requirements that the cloud service.
+<br/><br/>
 
 ## How it Works
 ---
@@ -20,7 +34,7 @@ The package contains the following modules:
 | data_connector.azure |
 
 Each module is capable of connect, download and upload operation to it-s corresponding cloud service.
-
+<br/><br/>
 
 ## Getting Started with data_connector
 ---
@@ -33,23 +47,15 @@ conda activate venv
 
 You can install the package with:
 ```bash
-python -m pip install intel-cloud-data-connector
+python -m pip install cloud-data-connector
 ```
 
-Please follow module specific documentation for use case, hands-examples. This documentation can be found inside the package.
+Please follow module specific documentation for use case, hands-examples.
 1. data_connector/azure/README.md
 2. data_connector/azure/AzureML.md
 3. data_connector/aws/README.md
 4. data_connector/gcp/README.md
-
-<!---
-## Learn More
----
-For more information about data_connector, see these guides and software resources:
-- github/repo/link 
-TODO: Update public repo
--->
-
+<br/><br/>
 
 ## Getting Started with data_connector.azure
 ---
@@ -102,11 +108,7 @@ Data connector provides a tool to connect to Azure ML workspaces and upload conf
 ```
 How to get a [Connection String](https://learn.microsoft.com/en-us/answers/questions/1071173/where-can-i-find-storage-account-connection-string)? 
 
-<!---
-TODO: Update link from public repo
-![Azure Connection String Sample](data_connector/../../../docs/img/connection_string.png)
------
--->
+![Azure Connection String Sample](https://github.com/IntelAI/models/blob/master/datasets/data_connector/docs/img/connection_string.png)
 
 Also you can get connection strings using Azure CLI
 ```bash
@@ -137,12 +139,8 @@ Or just
     downloader = Downloader(connector=connector)
     downloader.download()
 ```
-<!---
-TODO: Update link from public repo
-[See sample](../../samples/azure/blob_sample.py)
------
--->
-
+[See sample Here](https://github.com/IntelAI/models/blob/master/datasets/data_connector/samples/azure/blob_sample.py)
+<br/><br/>
 
 
 ## Getting Started with data_connector.aws
@@ -327,7 +325,7 @@ uploader = Uploader(conection_object)
 # upload a file
 uploader.upload(bucket_name, 'path/to_local_file.csv', 'path/to_object_name.csv')
 ```
-
+<br/><br/>
 
 ## Getting Started with data_connector.gcp
 ---
@@ -420,6 +418,7 @@ For service account:
 python -m samples.gcp.bigquery -p <project_name> -c <credentials_path>
 ```
 User must provide the project name using flag (-p) and the local path to the JSON file with the credentials (-c).
+<br/><br/>
 
 
 ## Support

datasets/data_connector/data_connector/aws/README.md

@@ -1,4 +1,4 @@
-# Data Connector AWS S3 
+# Cloud Data Connector: AWS S3 
 
 Data Connector for AWS S3 allows you to connect to S3 buckets and list contents, download and upload files.
 
@@ -8,7 +8,8 @@ To access S3 buckets, you will need to sign up for an AWS account and create acc
 
 Access keys consist of an access key ID and secret access key, which are used to sign programmatic requests that you make to AWS.
 
-## Hot to get your access key ID and secret access key
+How to get your access key ID and secret access key
+---
 
 1. Open the IAM console at https://console.aws.amazon.com/iam/.
 2. On the navigation menu, choose Users.
@@ -34,31 +35,50 @@ You can add more configuration settings listed [here](https://boto3.amazonaws.co
 You need to import the DataConnector class.
 
 ```python
-from data_connector.aws.connector import Connector
+from data_connector.aws import Connector as aws_connector
 ```
 
 Connector class has the method connect(), it creates an AWS S3 object, by default the function will create a S3 connector using the credentials saved in your environment variables.
 
 ```python
-connector = Connector()
+aws_bucket_connector = aws_connector()
 ```
 
 Call the connect() method, this will return a connection object for S3. 
 
 ```python
-conection_object = connector.connect()
+aws_conection_object = aws_bucket_connector.connect()
 ```
 
+Downloader
+---
 Import the Downloader class and use the connection object returned by connect() function.
 
+
 ```python
-from data_connector.aws.downloader import Downloader
+from data_connector.aws import Downloader as aws_downloader
 
-downloader = Downloader(conection_object)
+aws_file_downloader = aws_downloader(aws_conection_object)
 ```
 
-The Downloader class has two methods:
 
+
+To download a file use the `download(container_obj, data_file, destiny)` method and specify the next parameters.
+
+- container_obj: The name of the bucket to download from.
+- data_file: The name of the file to download from.
+- destiny: The path to the file to download to.
+
+```python
+from data_connector.aws import Downloader as aws_downloader
+
+downloader = aws_downloader(aws_conection_object)
+file_name = "path/to_file.csv"
+downloader.download(bucket_name, file_name, 'path/to_destiny.csv')
+```
+List Blobs
+---
+[Downloader](#downloader) class has two methods:
 - list_blobs(container_obj): The function to get a list of the objects in a bucket.
 - download(container_obj, data_file, destiny): The function to download a file from a S3 bucket.
 
@@ -67,71 +87,62 @@ A first step with buckets is to list their content using the `list_blobs(contain
 - container_obj: The bucket name to list.
 
 ```python
-from data_connector.aws.downloader import Downloader
+from data_connector.aws import Downloader as aws_downloader
 
-downloader = Downloader(conection_object)
+aws_bucket_downloader = aws_downloader(aws_conection_object)
 
-list_blobs = downloader.list_blobs('MY_BUCKET_NAME')
+list_blobs = aws_bucket_downloader.list_blob(
+    'MY_BUCKET_NAME'
+)
 print(list_blobs)
 ```
 
-To download a file use the `download(container_obj, data_file, destiny)` method and specify the next parameters.
-
-- container_obj: The name of the bucket to download from.
-- data_file: The name of the file to download from.
-- destiny: The path to the file to download to.
-
-```python
-from data_connector.aws.downloader import Downloader
-
-downloader = Downloader(conection_object)
-file_name = "path/to_file.csv"
-downloader.download(bucket_name, file_name, 'path/to_destiny.csv')
-```
+Upload
+---
 
 You can import an Uploader class and use the upload method to send a file from you local machine to a bucket. You need to add the connector object to Uploader constructor.
 
 ```python
-from data_connector.aws.uploader import Uploader
-from data_connector.aws.connector import Connector
+from data_connector.aws import Uploader as aws_uploader
+from data_connector.aws  import Connector as aws_connector
 
-connector = Connector()
-conection_object = connector.connect()
-uploader = Uploader(conection_object)
+aws_bucket_connector = aws_connector()
+aws_conection_object = aws_bucket_connector.connect()
+aws_bucker_uploader = aws_uploader(aws_conection_object)
 
 ```
 Specify the next parameters in upload function.
 
 - container_obj: The name of the bucket to upload to.
 - data_file: The path to the file to upload.
-- object_name: The name of the file to upload to.
+- object_name: The name of file in cloud bucket
 
 ```python
-from data_connector.aws.uploader import Uploader
-
-uploader = Uploader(conection_object)
-uploader.upload(bucket_name, 'path/to_local_file.csv', 'path/to_object_name.csv')
+aws_bucker_uploader.upload('<bucket_name>', '<path/to_local_file>', '<path/to_object_name>')
 ```
 
+
+Samples
+---
 ### List objects in a bucket
 
 ```python
 # import the dataconnector package
-from data_connector.aws.connector import Connector
-from data_connector.aws.downloader import Downloader
+from data_connector.aws import Connector as aws_connector
+from data_connector.aws import Downloader as aws_downloader
 
 # specify a S3 bucket name
 bucket_name = 'MY_BUCKET_NAME'
 # create a connector
-connector = Connector()
+aws_bucket_connector = aws_connector()
 # connect to aws using default AWS access keys
 # connect() method uses the configurations settings for AWS account
-conection_object = connector.connect()
+aws_conection_object = aws_bucket_connector.connect()
 # list files from bucket
 # create a downloader to list files
-downloader = Downloader(conection_object)
+aws_downloader = aws_downloader(aws_conection_object)
 # use the list_blobs function
-list_blobs = downloader.list_blobs(bucket_name)
+list_blobs = aws_downloader.list_blobs(bucket_name)
 # list_blobs functions returns all objects in bucket
 print(list_blobs)
 
@@ -141,40 +152,40 @@ print(list_blobs)
 
 ```python
 # import the dataconnector package
-from data_connector.aws.connector import Connector
-from data_connector.aws.downloader import Downloader
+from data_connector.aws import Connector as aws_connector
+from data_connector.aws import Downloader as aws_connector
 
 # specify a S3 bucket name
 bucket_name = 'MY_BUCKET_NAME'
 # create a connector
-connector = Connector()
+aws_bucket_connector = aws_connector()
 # connect to aws using default aws access keys
-conection_object = connector.connect()
+aws_conection_object = aws_bucket_connector.connect()
 # download a file from bucket
 # create a Downloader object using a connector object
-downloader = Downloader(conection_object)
+aws_downloader = aws_connector(aws_conection_object)
 # specify the object name to download
 file_name = "path/to_file.csv"
 # download the object
-downloader.download(bucket_name, file_name, 'path/to_destiny.csv')
+aws_downloader.download(bucket_name, file_name, 'path/to_destiny.csv')
 ```
 
 ### Upload a file
 
 ```python
 # import dataconnector package
-from data_connector.aws.connector import Connector
-from data_connector.aws.uploader import Uploader
+from data_connector.aws import Connector as aws_connector
+from data_connector.aws import Uploader as aws_uploader
 
 # specify a S3 bucket name
 bucket_name = 'MY_BUCKET_NAME'
 # create a connector
-connector = Connector()
+aws_bucket_connector = aws_connector()
 # connect to aws using default aws access keys
-conection_object = connector.connect()
+aws_conection_object = aws_bucket_connector.connect()
 # Upload a file
 # create a uploader object using a connection object
-uploader = Uploader(conection_object)
+aws_bucket_uploader = aws_uploader(aws_conection_object)
 # upload a file
-uploader.upload(bucket_name, 'path/to_local_file.csv', 'path/to_object_name.csv')
+aws_bucket_uploader.upload(bucket_name, 'path/to_local_file.csv', 'path/to_object_name.csv')
 ```