Incorrect senza wait behavior for updates/creation

[kenden]

The command senza wait does not work properly.
An error is returned.
The option wait --deletion brings better results but even after the completion, the program end up with an error (syscode 1). This is a problem for continuous integration systems.

After creating a stack, I get:

senza create --region eu-central-1 senza-template.yml stack2 Environment="production" DockerImgVer="v801" AMIImage="ami-20ba574f"
Generating Cloud Formation template.. OK
Creating Cloud Formation stack project-stack2.. OK

$ senza wait --region eu-central-1 senza-template.yml stack2
ERROR: LaunchConfig CREATE_FAILED: AMI ami-28fb705b is invalid: The image id '[ami-28fb705b]' does not exist
ERROR: project-stack2 ROLLBACK_IN_PROGRESS: The following resource(s) failed to create: [LaunchConfig]. . Rollback requested by user.
ERROR: Stack project-stack2 has status DELETE_COMPLETE

$ # what? let's check this out....

$ senza events --region eu-central-1 senza-template.yml stack2
Stack Name│Ver.  │Resource Type                     │Resource ID      │Status            │Status Reason              │Event Time
project stack2 CloudFormation::Stack              project-stack2 CREATE_IN_PROGRESS User Initiated                 60s ago
project stack2 EC2::SecurityGroup                 SecurityGroupELB  CREATE_IN_PROGRESS                                57s ago
project stack2 EC2::SecurityGroup                 SecurityGroupELB  CREATE_IN_PROGRESS Resource creation Initiated    40s ago
project stack2 EC2::SecurityGroup                 SecurityGroupELB  CREATE_COMPLETE                                   39s ago
project stack2 EC2::SecurityGroup                 SecurityGroupEC2  CREATE_IN_PROGRESS                                37s ago
project stack2 ElasticLoadBalancing::LoadBalancer ELBHealthcheck    CREATE_IN_PROGRESS                                36s ago
project stack2 ElasticLoadBalancing::LoadBalancer ELBHealthcheck    CREATE_IN_PROGRESS Resource creation Initiated    35s ago
project stack2 ElasticLoadBalancing::LoadBalancer ELBHealthcheck    CREATE_COMPLETE                                   34s ago
project stack2 EC2::SecurityGroup                 SecurityGroupEC2  CREATE_IN_PROGRESS Resource creation Initiated    21s ago
project stack2 EC2::SecurityGroup                 SecurityGroupEC2  CREATE_COMPLETE                                   19s ago
project stack2 AutoScaling::LaunchConfiguration   LaunchConfig      CREATE_COMPLETE                                   16s ago
project stack2 AutoScaling::LaunchConfiguration   LaunchConfig      CREATE_IN_PROGRESS Resource creation Initiated    16s ago
project stack2 AutoScaling::LaunchConfiguration   LaunchConfig      CREATE_IN_PROGRESS                                16s ago
project stack2 AutoScaling::AutoScalingGroup      ASG               CREATE_IN_PROGRESS                                12s ago
project stack2 AutoScaling::AutoScalingGroup      ASG               CREATE_IN_PROGRESS Resource creation Initiated    11s ago

$ # looks correct... What about with the wait --deletion option?

$ senza wait --deletion --region eu-central-1 senza-template.yml stack2
Waiting up to 1800 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1794 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1789 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1784 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1778 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1773 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1768 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1763 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1756 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1751 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1746 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1741 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1735 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1730 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1725 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1719 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
Waiting up to 1714 more secs for stack project-stack2 (CREATE_IN_PROGRESS)..
ERROR: Stack project-stack2 has status CREATE_COMPLETE

$ echo $?
1  # <-------- CREATE_COMPLETE results in error (1) ?

After updating a stack, I get:

$ senza update --region eu-central-1 senza-template.yml stack1 Environment="test" DockerImgVer="v304" AMIImage="ami-20ba575f"
Generating Cloud Formation template.. OK
Updating Cloud Formation stack project-stack1.. OK

 $ senza wait --region eu-central-1 senza-template.yml stack1
ERROR: ASG CREATE_FAILED: Received 0 SUCCESS signal(s) out of 1.  Unable to satisfy 100% MinSuccessfulInstancesPercent requirement
ERROR: project-stack1 ROLLBACK_IN_PROGRESS: The following resource(s) failed to create: [ASG]. . Rollback requested by user.
ERROR: Stack project-stack1 has status DELETE_COMPLETE

$ senza wait --deletion --region eu-central-1 senza-template.yml stack1    # <----- more success with --deletion... Why?
Waiting up to 1800 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1795 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1789 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1784 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1779 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1773 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1767 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1762 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1757 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1752 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1747 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1741 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1736 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1731 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1726 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1721 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1715 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1710 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1704 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1699 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1694 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1688 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1683 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1678 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1673 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1668 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1662 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1657 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1652 more secs for stack project-stack1 (UPDATE_IN_PROGRESS)..
Waiting up to 1647 more secs for stack project-stack1 (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)..
ERROR: Stack project-stack1 has status UPDATE_COMPLETE
$ echo $?
1                             # <-------------------------------------------------------- UPDATE_COMPLETE results in error?

[kenden]

My current workaround for this issue ( updated after the fix for #195)

Inside a deploy.sh file:

set +e  # do NOT exit if there is an error
senza wait --region ${EC2_REGION} senza-template.yml ${STACK_VERSION}
echo "If it is says 'UPDATE COMPLETE', it's good. Not an error but a bug in Senza (#234)."  # explain to devs why we get an error.
set -e  # activate exit on error again
senza status --region ${EC2_REGION} senza-template.yml ${STACK_VERSION} | grep UPDATE_COMPLETE  # fail if status is not UPDATE_COMPLETE

It would be nice to just have:

senza wait --region ${EC2_REGION} senza-template.yml ${STACK_VERSION}

[kenden]

I believe the update issue is a duplicate of #195

[kenden]

I confirm that the update issue is partially solved by #195

I haven't checked the "wait for creation" scenario yet.
The update issue is still not 100% OK though, it still returns an incorrect syscode 1:

$ senza update --region eu-central-1 senza-template.yml v1
Generating Cloud Formation template.. OK
Updating Cloud Formation stack project-stack.. OK

$ senza wait --stacktrace-visible  --region eu-central-1 senza-template.yml v1
Waiting up to 1800 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1795 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1789 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1783 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1778 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1773 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1768 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1763 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1758 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1752 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1747 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1742 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1737 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1732 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1727 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1721 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1716 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1711 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1705 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1700 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1695 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1690 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1685 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1680 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1675 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1669 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1664 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1658 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1653 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1648 more secs for stack project-stack (UPDATE_IN_PROGRESS)..
Waiting up to 1643 more secs for stack project-stack (UPDATE_COMPLETE_CLEANUP_IN_PROGRESS)..
ERROR: Stack project-stack has status UPDATE_COMPLETE   <-- INCORRECT "ERROR" message
$ echo $?
1          <---- INCORRECT

[hjacobs]

@kenden yes, the update part is broken --- actually "nobody" uses CF update here in Zalando AFAIK 😏