-
|
If I have a private GitHub repository, eg. |
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 11 replies
-
|
If you use fast deploys in your workflow, the following steps can install a private package as a dependency to your dagster cloud project. If you have
Once the |
Beta Was this translation helpful? Give feedback.
-
|
^ this one currently does not work anymore, due to: #12340 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
@johannkm thanks for this guide, I've implemented it and it works. But FYI, this line I need to install two dependencies, should I add all three steps for each dependency - or just the first two and the last third is shared? PS; when you have some time I would greatly appreciate your answer here: #17823 |
Beta Was this translation helpful? Give feedback.
-
Thanks for the report. We have changed it to
It would be just the first two steps repeated for each wheel and the last step is only needed once. |
Beta Was this translation helpful? Give feedback.
-
|
Is there an equivalent for "non fast" deploys? |
Beta Was this translation helpful? Give feedback.
-
|
@shalabhc are the python wheels built by Github Action cached or saved anywhere? I have an issue where I can see from Github Actions logs that the latest branch of private module is correctly pulled. But the "Python Executable Deploy" step is failing due to Is is possible to check the logs where packages of the pipeline are installed? |
Beta Was this translation helpful? Give feedback.
-
|
I'm getting the same issue @CharlesFr , were you ever able to fix it? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the instructions here, these were super helpful. I am not sure if this is related to the issue above. My build was successful but I later got an |
Beta Was this translation helpful? Give feedback.
-
|
I ran into this issue as well. If you examine the log lines above you can see |
Beta Was this translation helpful? Give feedback.
-
|
Warning for anyone doing this off the most recent starter templates that use Ubuntu 22.04. This induces deploy_pex.py to use docker to build instead of "local" which leads to your private dependency not being available. |
Beta Was this translation helpful? Give feedback.
-
|
I don't know if it related to this, but I am using this approach and getting error: I have tried using both ubuntu-20-04 and ubuntu-22.04 without success - how would you resolve this issue? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
The ubuntu-20.04 runner was deprecated as of yesterday, so switching to it is no longer a viable workaround. I'm now trying with ubuntu-24.04, but still encountering the same issue: Has anyone managed to resolve this using a different setup or configuration? |
Beta Was this translation helpful? Give feedback.
-
|
I am seeing the same, has anyone solved this? |
Beta Was this translation helpful? Give feedback.
-
|
Opened #30788 for this |
Beta Was this translation helpful? Give feedback.
-
|
I heard from Dagster support:
After that, it is possible to use the workaround above. Sample working yml which uses a pre-built wheel available in the repository checkout under name: Dagster Cloud Serverless Deployment
on:
push:
branches:
- "main"
- "master"
pull_request:
types: [opened, synchronize, reopened, closed]
concurrency:
# Cancel in-progress deploys to same branch
group: ${{ github.ref }}/deploy
cancel-in-progress: true
env:
DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
ENABLE_FAST_DEPLOYS: 'true'
PYTHON_VERSION: '3.12'
DAGSTER_CLOUD_YAML_PATH: '.'
DAGSTER_CLOUD_FILE: 'dagster_cloud.yaml'
DAGSTER_CLOUD_ORGANIZATION: 'YOUR_ORG'
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
jobs:
dagster_cloud_default_deploy:
name: Dagster Serverless Deploy
runs-on: ubuntu-22.04
steps:
- name: Prerun Checks
id: prerun
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.head_ref || github.ref }}
# Validate dagster_cloud.yaml and the connection to dagster.cloud
- name: Validate configuration
id: ci-validate
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci check --project-dir ${{ env.DAGSTER_CLOUD_YAML_PATH }} --dagster-cloud-yaml-path ${{ env.DAGSTER_CLOUD_FILE }}"
- name: Configure dependency resolution to use a prebuilt wheel
run: |
mkdir -p $GITHUB_WORKSPACE/deps &&
echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
echo " file://$GITHUB_WORKSPACE/local_wheels/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV
# Parse dagster_cloud.yaml, detect if this is branch deployment and initialize the build session
- name: Initialize build session
id: ci-init
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
project_dir: ${{ env.DAGSTER_CLOUD_YAML_PATH }}
dagster_cloud_yaml_path: ${{ env.DAGSTER_CLOUD_FILE }}
# A full deployment name. If this run is for a pull request, this value will be used as
# the base deployment for the branch deployment.
deployment: 'prod'
# If using fast build, build the PEX
# First ensure the correct Python version is installed
- name: Set up Python ${{ env.PYTHON_VERSION }} for target
id: setup-python-version
if: steps.prerun.outputs.result == 'pex-deploy'
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install setuptools
if: steps.prerun.outputs.result == 'pex-deploy'
run: ${{ steps.setup-python-version.outputs.python-path }} -m pip install setuptools
shell: bash
- name: Run PEX build
id: run-pex-build
if: steps.prerun.outputs.result == 'pex-deploy'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci build --build-strategy=python-executable --python-version ${{ env.PYTHON_VERSION }} --pex-deps-cache-from='${{ github.repository }}' --pex-deps-cache-to='${{ github.repository }}'"
# Otherwise, enable buildx for caching and build the Docker image
- name: Set up Docker Buildx
if: steps.prerun.outputs.result == 'docker-deploy'
uses: docker/setup-buildx-action@v2
- name: Run Docker build
id: run-docker-build
if: steps.prerun.outputs.result == 'docker-deploy'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci build --build-strategy=docker --python-version ${{ env.PYTHON_VERSION }}"
# Deploy all code locations in this build session to Dagster Cloud
- name: Deploy to Dagster Cloud
id: ci-deploy
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci deploy"
# Update a PR comment - this runs always() so the comment is updated on success and failure
- name: Update PR comment for branch deployments
id: ci-notify
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci notify --project-dir=${{ env.DAGSTER_CLOUD_YAML_PATH }}"
# Generate a summary that shows up on the Workflow Summary page
- name: Generate a summary
id: ci-summary
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci status --output-format=markdown >> $GITHUB_STEP_SUMMARY" |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for my ignorance, but could you outline the process more throughout? |
Beta Was this translation helpful? Give feedback.
-
|
you can't link them in the install_requires=[
"my-package",
# ...
]and then put a - name: Configure dependency resolution to use a prebuilt wheel
run: |
mkdir -p $GITHUB_WORKSPACE/deps &&
echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
echo " file://$GITHUB_WORKSPACE/local_wheels/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENVthe path itself doesn't matter, it can be anywhere, same it doesn't matter whether it's pre-built or build in a previous actions step. In my case, I have a |
Beta Was this translation helpful? Give feedback.
If you use fast deploys in your workflow, the following steps can install a private package as a dependency to your dagster cloud project. If you have
ENABLE_FAST_DEPLOYS: "true"in your.github/workflows/deploy.ymlthen you use fast deploys.deploy.ymlfile, add the following to the top ofsteps:section in thedagster-cloud-default-deployjob.