Skip to content

Commit ba4bb30

Browse files
erezsherezsh
authored
Merge pull request #2 from datafold/dev
Initial code copied from data-diff
2 parents 362de4a + 3388b8d commit ba4bb30

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

68 files changed

+7016
-0
lines changed

‎.github/ISSUE_TEMPLATE/bug_report.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
name: Bug report
3+
about: Create a report to help us improve
4+
title: ''
5+
labels: ''
6+
assignees: ''
7+
8+
---
9+
10+
**Describe the bug**
11+
A clear and concise description of what the bug is.
12+
13+
**Describe the environment**
14+
15+
Describe which OS you're using, which sqeleton version, and any other information that might be relevant to this bug.

‎.github/ISSUE_TEMPLATE/feature_request.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
name: Feature request
3+
about: Suggest an idea for this project
4+
title: ''
5+
labels: enhancement
6+
assignees: ''
7+
8+
---
9+
10+
**Is your feature request related to a problem? Please describe.**
11+
A clear and concise description of what the problem is.
12+
13+
**Describe the solution you'd like**
14+
A clear and concise description of what you want to happen.
15+
16+
**Describe alternatives you've considered**
17+
A clear and concise description of any alternative solutions or features you've considered.
18+
19+
**Additional context**
20+
Add any other context or screenshots about the feature request here.

‎.github/ISSUE_TEMPLATE/request-support-for-a-database.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
name: Request support for a database
3+
about: 'Request a driver to support a new database '
4+
title: 'Add support for <database name>'
5+
labels: new-db-driver
6+
assignees: ''
7+
8+
---
9+
10+

‎.github/workflows/ci.yml

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: CI-COVER-VERSIONS
2+
3+
on:
4+
# push:
5+
# paths:
6+
# - '**.py'
7+
# - '.github/workflows/**'
8+
# - '!dev/**'
9+
pull_request:
10+
branches: [ master ]
11+
workflow_dispatch:
12+
13+
jobs:
14+
unit_tests:
15+
strategy:
16+
fail-fast: false
17+
matrix:
18+
os: [ubuntu-latest]
19+
python-version:
20+
- "3.7"
21+
- "3.8"
22+
- "3.9"
23+
- "3.10"
24+
25+
name: Check Python ${{ matrix.python-version }} on ${{ matrix.os }}
26+
runs-on: ${{ matrix.os }}
27+
steps:
28+
- uses: actions/checkout@v3
29+
30+
- name: Setup Python ${{ matrix.python-version }}
31+
uses: actions/setup-python@v3
32+
with:
33+
python-version: ${{ matrix.python-version }}
34+
35+
- name: Build the stack
36+
run: docker-compose up -d mysql postgres presto trino clickhouse vertica
37+
38+
- name: Install Poetry
39+
run: pip install poetry
40+
41+
- name: Install package
42+
run: "poetry install"
43+
44+
- name: Run unit tests
45+
env:
46+
PRESTO_URI: 'presto://[email protected]/postgresql/public'
47+
TRINO_URI: 'trino://[email protected]:8081/postgresql/public'
48+
CLICKHOUSE_URI: 'clickhouse://clickhouse:Password1@localhost:9000/clickhouse'
49+
VERTICA_URI: 'vertica://vertica:Password1@localhost:5433/vertica'
50+
SNOWFLAKE_URI: '${{ secrets.SNOWFLAKE_URI }}'
51+
REDSHIFT_URI: '${{ secrets.REDSHIFT_URI }}'
52+
run: |
53+
chmod +x tests/waiting_for_stack_up.sh
54+
./tests/waiting_for_stack_up.sh && TEST_ACROSS_ALL_DBS=0 poetry run unittest-parallel -j 16

‎.github/workflows/ci_full.yml

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
name: CI-COVER-DATABASES
2+
3+
on:
4+
# push:
5+
# paths:
6+
# - '**.py'
7+
# - '.github/workflows/**'
8+
# - '!dev/**'
9+
pull_request:
10+
branches: [ master ]
11+
workflow_dispatch:
12+
13+
jobs:
14+
unit_tests:
15+
strategy:
16+
fail-fast: false
17+
matrix:
18+
os: [ubuntu-latest]
19+
python-version:
20+
- "3.10"
21+
22+
name: Check Python ${{ matrix.python-version }} on ${{ matrix.os }}
23+
runs-on: ${{ matrix.os }}
24+
steps:
25+
- uses: actions/checkout@v3
26+
27+
- name: Setup Python ${{ matrix.python-version }}
28+
uses: actions/setup-python@v3
29+
with:
30+
python-version: ${{ matrix.python-version }}
31+
32+
- name: Build the stack
33+
run: docker-compose up -d mysql postgres presto trino clickhouse vertica
34+
35+
- name: Install Poetry
36+
run: pip install poetry
37+
38+
- name: Install package
39+
run: "poetry install"
40+
41+
- name: Run unit tests
42+
env:
43+
PRESTO_URI: 'presto://[email protected]/postgresql/public'
44+
TRINO_URI: 'trino://[email protected]:8081/postgresql/public'
45+
CLICKHOUSE_URI: 'clickhouse://clickhouse:Password1@localhost:9000/clickhouse'
46+
VERTICA_URI: 'vertica://vertica:Password1@localhost:5433/vertica'
47+
SNOWFLAKE_URI: '${{ secrets.SNOWFLAKE_URI }}'
48+
REDSHIFT_URI: '${{ secrets.REDSHIFT_URI }}'
49+
run: |
50+
chmod +x tests/waiting_for_stack_up.sh
51+
./tests/waiting_for_stack_up.sh && poetry run unittest-parallel -j 16

‎.gitignore

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
pip-wheel-metadata/
24+
share/python-wheels/
25+
*.egg-info/
26+
.installed.cfg
27+
*.egg
28+
MANIFEST
29+
30+
# PyInstaller
31+
# Usually these files are written by a python script from a template
32+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
33+
*.manifest
34+
*.spec
35+
36+
# Installer logs
37+
pip-log.txt
38+
pip-delete-this-directory.txt
39+
40+
# Unit test / coverage reports
41+
htmlcov/
42+
.tox/
43+
.nox/
44+
.coverage
45+
.coverage.*
46+
.cache
47+
nosetests.xml
48+
coverage.xml
49+
*.cover
50+
*.py,cover
51+
.hypothesis/
52+
.pytest_cache/
53+
54+
# Translations
55+
*.mo
56+
*.pot
57+
58+
# Django stuff:
59+
*.log
60+
local_settings.py
61+
db.sqlite3
62+
db.sqlite3-journal
63+
64+
# Flask stuff:
65+
instance/
66+
.webassets-cache
67+
68+
# Scrapy stuff:
69+
.scrapy
70+
71+
# Sphinx documentation
72+
docs/_build/
73+
74+
# PyBuilder
75+
target/
76+
77+
# Jupyter Notebook
78+
.ipynb_checkpoints
79+
80+
# IPython
81+
profile_default/
82+
ipython_config.py
83+
84+
# pyenv
85+
.python-version
86+
87+
# pipenv
88+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
89+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
90+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
91+
# install all needed dependencies.
92+
#Pipfile.lock
93+
94+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
95+
__pypackages__/
96+
97+
# Celery stuff
98+
celerybeat-schedule
99+
celerybeat.pid
100+
101+
# SageMath parsed files
102+
*.sage.py
103+
104+
# Environments
105+
.env
106+
.venv
107+
env/
108+
venv/
109+
ENV/
110+
env.bak/
111+
venv.bak/
112+
113+
# Spyder project settings
114+
.spyderproject
115+
.spyproject
116+
117+
# Rope project settings
118+
.ropeproject
119+
120+
# mkdocs documentation
121+
/site
122+
123+
# mypy
124+
.mypy_cache/
125+
.dmypy.json
126+
dmypy.json
127+
128+
# Pyre type checker
129+
.pyre/
130+
131+
# Mac
132+
.DS_Store
133+
134+
# IntelliJ
135+
.idea
136+
137+
# VSCode
138+
.vscode

‎dev/Dockerfile.prestosql.340

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
FROM openjdk:11-jdk-slim-buster
2+
3+
ENV PRESTO_VERSION=340
4+
ENV PRESTO_SERVER_URL=https://repo1.maven.org/maven2/io/prestosql/presto-server/${PRESTO_VERSION}/presto-server-${PRESTO_VERSION}.tar.gz
5+
ENV PRESTO_CLI_URL=https://repo1.maven.org/maven2/io/prestosql/presto-cli/${PRESTO_VERSION}/presto-cli-${PRESTO_VERSION}-executable.jar
6+
ENV PRESTO_HOME=/opt/presto
7+
ENV PATH=${PRESTO_HOME}/bin:${PATH}
8+
9+
WORKDIR $PRESTO_HOME
10+
11+
RUN set -xe \
12+
&& apt-get update \
13+
&& apt-get install -y curl less python \
14+
&& curl -sSL $PRESTO_SERVER_URL | tar xz --strip 1 \
15+
&& curl -sSL $PRESTO_CLI_URL > ./bin/presto \
16+
&& chmod +x ./bin/presto \
17+
&& apt-get remove -y curl \
18+
&& rm -rf /var/lib/apt/lists/*
19+
20+
VOLUME /data
21+
22+
EXPOSE 8080
23+
24+
ENTRYPOINT ["launcher"]
25+
CMD ["run"]

‎dev/dev.env

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
POSTGRES_USER=postgres
2+
POSTGRES_PASSWORD=Password1
3+
POSTGRES_DB=postgres
4+
5+
MYSQL_DATABASE=mysql
6+
MYSQL_USER=mysql
7+
MYSQL_PASSWORD=Password1
8+
MYSQL_ROOT_PASSWORD=RootPassword1
9+
10+
CLICKHOUSE_USER=clickhouse
11+
CLICKHOUSE_PASSWORD=Password1
12+
CLICKHOUSE_DB=clickhouse
13+
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
14+
15+
# Vertica credentials
16+
APP_DB_USER=vertica
17+
APP_DB_PASSWORD=Password1
18+
VERTICA_DB_NAME=vertica
19+
20+
# To prevent generating sample demo VMart data (more about it here https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/GettingStartedGuide/IntroducingVMart/IntroducingVMart.htm),
21+
# leave VMART_DIR and VMART_ETL_SCRIPT empty.
22+
VMART_DIR=
23+
VMART_ETL_SCRIPT=

‎dev/presto-conf/standalone/catalog/jmx.properties

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
connector.name=jmx

‎dev/presto-conf/standalone/catalog/memory.properties

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
connector.name=memory

‎dev/presto-conf/standalone/catalog/postgresql.properties

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
connector.name=postgresql
2+
connection-url=jdbc:postgresql://postgres:5432/postgres
3+
connection-user=postgres
4+
connection-password=Password1
5+
allow-drop-table=true

‎dev/presto-conf/standalone/catalog/tpcds.properties

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
connector.name=tpcds

‎dev/presto-conf/standalone/catalog/tpch.properties

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
connector.name=tpch

0 commit comments

Comments
 (0)