Zappa events intermittently failing to create with PutBucketNotificationConfiguration exception #1419
Description
I have an Django DRF microservice application that runs in AWS Lambda. The relevant dependencies and dependency versions are listed below.
- Django==5.1.11
- djangorestframework==3.15.2
- zappa==0.59.0
My zappa_settings.py file is included below
"prod": {
"aws_region": "us-east-1",
"django_settings": "my-app.settings.settings",
"profile_name": "myit-infra-sis-prod",
"project_name": "myapp",
"runtime": "python3.12",
"s3_bucket": "myit-sis2-prod-deploy",
"layers": [
"arn:aws:lambda:us-east-1:111111111111:layer:sis-django-py312:2",
"arn:aws:lambda:us-east-1:111111111111:layer:myapp-libs-py312:1"
],
"exclude": [
"htmlcov",
"*.tox",
"*.json",
"local_settings.py",
"boto3*",
"botocore*",
"moto*",
"tox*",
"PIL*",
"viewflow*",
"faker*",
"piptools",
"factory_boy*",
"factory*",
"fakeredis*",
"django_fakeredis*",
"virtualenv*",
"validate_email*",
"troposphere*",
"__pycache__*"
],
"exclude_glob": [
"*.gz",
"*.rar",
"*.zip",
"*.log",
"*.pyc"
],
"regex_excludes": [
"fixtures",
"__pycache__"
],
"callbacks": {
"zip": "zappa_package_cleaner.main"
},
"slim_handler": false,
"manage_roles": false,
"events": [
{
"function": "lambda.scheduled_instructor_managed_waitlist_report_daily",
"expression": "cron(0 10 ? * 2-6 *)",
"description": "Runs Mon–Fri at 06:00 AM ET (10:00 UTC)"
},
{
"function": "lambda.wf_instructor_admin_notifications_daily_morning_edt",
"expression": "cron(0 13 ? * 2-6 *)",
"description": "Runs Mon–Fri at 9:00 AM EDT (UTC-4)"
},
{
"function": "lambda.wf_instructor_admin_notifications_daily_morning_est",
"expression": "cron(0 14 ? * 2-6 *)",
"description": "Runs Mon–Fri at 9:00 AM EST (UTC-5)"
},
{
"function": "lambda.wf_instructor_admin_notifications_daily_noon_edt",
"expression": "cron(0 19 ? * 2-6 *)",
"description": "Runs Mon–Fri at 3:00 PM EDT (UTC-4)"
},
{
"function": "lambda.wf_instructor_admin_notifications_daily_noon_est",
"expression": "cron(0 20 ? * 2-6 *)",
"description": "Runs Mon–Fri at 3:00 PM EST (UTC-5)"
},
{
"function": "lambda.wf_student_pending_notifications_daily_morning_edt",
"expression": "cron(0 13 ? * 2-6 *)",
"description": "Runs Mon–Fri at 09:00 AM EDT (UTC-4)."
},
{
"function": "lambda.wf_student_pending_notifications_daily_morning_est",
"expression": "cron(0 14 ? * 2-6 *)",
"description": "Runs Mon–Fri at 09:00 AM EST (UTC-5)."
},
{
"function": "lambda.wf_student_pending_notifications_daily_noon_edt",
"expression": "cron(0 19 ? * 2-6 *)",
"description": "Runs Mon–Fri at 03:00 PM EDT (UTC-4)."
},
{
"function": "lambda.wf_student_pending_notifications_daily_noon_est",
"expression": "cron(0 20 ? * 2-6 *)",
"description": "Runs Mon–Fri at 03:00 PM EST (UTC-5)."
},
{
"function": "lambda.wf_deactivate_expired_wfegroups_daily_edt",
"expression": "cron(0 4 * * ? *)",
"description": "Runs daily at 12:00 AM EDT (UTC-4)."
},
{
"function": "lambda.wf_deactivate_expired_wfegroups_daily_est",
"expression": "cron(0 5 * * ? *)",
"description": "Runs daily at 12:00 AM EST (UTC-5)."
},
{
"function": "lambda.unit_rules_refresh",
"expression": "cron(0 22 * * ? *)"
},
{
"function": "lambda.deactivate_waitlist",
"expression": "cron(0 4 * * ? *)"
},
{
"function": "lambda.refresh_restricted_class_query",
"expression": "rate(30 minutes)"
},
{
"function": "lambda.refresh_workflow_approvers",
"expression": "cron(0 1 * * ? *)",
"description": "Runs daily at 9 PM ET (UTC-4)."
},
{
"function": "students.external_registration.utils.process_s3_file",
"event_source": {
"arn": "arn:aws:s3:::myit-sis2-prod-ext-reg",
"events": [
"s3:ObjectCreated:*"
]
}
},
{
"function": "lambda.process_download_s3_function",
"event_source": {
"arn": "arn:aws:s3:::myit-sis2-prod-job",
"events": [
"s3:ObjectCreated:*"
]
}
},
{
"function": "lambda.sqs_event_handler",
"event_source": {
"arn": "arn:aws:sqs:us-east-1:111111111111:student-record-processing-queue",
"enabled": true
}
},
{
"function": "lambda.exportfiletask_expire",
"expression": "cron(0 1 * * ? *)",
"description": "Runs daily at 1:00 AM EST (UTC-5)."
}
],
"timeout_seconds": 900,
"memory_size": 8192
}
}
The following is a snippet from my shell script that runs during the Jenkins pipeline deployment
if [[ ${DEPLOY:-'false'} == 'true' ]]
then
echo
echo "----------------"
echo "> zappa update <"
echo "----------------"
set +e
zappa update $ZAPPA_NAME --disable_progress 2>&1 | tee zappa_update_log.txt
deploy_exit_status=$?
set -e
if [[ "$deploy_exit_status" -ne "0" ]]
then
echo "zappa update failed"
exit 1
fi
# Check for partial errors.
if grep -qE "ERROR:|Could not create event" zappa_update_log.txt; then
echo "zappa partial update failed."
exit 1
fi
echo
echo "------------------"
echo "> ./manage check <"
echo "------------------"
zappa manage $ZAPPA_NAME check
echo
echo "--------------"
echo "> tag lambda <"
echo "--------------"
# 1.31.12 is the highest version compatible with boto3==1.33.2 per requirements.txt
# if boto3 is upgraded, this can be upgraded too
pip install -q awscli==1.31.12
set -x
aws lambda tag-resource \
--tags git-branch=$GIT_BRANCH,git-commit=$GIT_COMMIT \
--resource $LAMBDA_ARN
set +x
fi
Intermittently ( but often ) when the zappa update command runs we receive the following error creating the s3 bucket events
ERROR:Unable to add S3 event source
Traceback (most recent call last):
File "/home/jenkins/agent/workspace/sis-myapp-prod-deploy/buildenv/lib64/python3.12/site-packages/kappa/event_source/s3.py", line 55, in add
response = self._s3.call(
^^^^^^^^^^^^^^
File "/home/jenkins/agent/workspace/sis-myapp-prod-deploy/buildenv/lib64/python3.12/site-packages/kappa/awsclient.py", line 86, in call
data = op(**kwargs)
^^^^^^^^^^^^
File "/home/jenkins/agent/workspace/sis-myapp-prod-deploy/buildenv/lib64/python3.12/site-packages/botocore/client.py", line 553, in _api_call
return self._make_api_call(operation_name, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/jenkins/agent/workspace/sis-myapp-prod-deploy/buildenv/lib64/python3.12/site-packages/botocore/client.py", line 1009, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations
Problem creating s3:ObjectCreated:* event schedule for students.external_registration.utils.process_s3_file!
It does not always happen on this one event. It randomly happens on any s3 bucket event. We either rerun the pipeline until all the events create appropriately, or we just create the event manually on the bucket via the AWS Management Console.
Has anyone ran into this issue before? Is it fixed in an upgraded version?