Map state machines in DISTRIBUTED mode bug

[ParsonsProjects]

This is a Bug Report

Description

What went wrong?

When using the DISTRIBUTED mode for a Map state the following error is thrown from compileIamRole.js.

What did you expect should have happened?

Serverless file deploys correctly.

What was the config you used?

Type: Map
  ItemProcessor:
    ProcessorConfig:
      Mode: DISTRIBUTED
      ExecutionType: STANDARD
    StartAt: Lambda
    States:
      Lambda:
        Type: Task
        Resource: "arn:aws:states:::lambda:invoke"
        Parameters:
          "Payload.$": "$"
          FunctionName:
            "Fn::GetAtt": [AssetLambdaFunction, Arn]
        End: true
        Retry:
          - ErrorEquals:
              - States.TaskFailed
            IntervalSeconds: 1
            MaxAttempts: 2
            BackoffRate: 2.0
  Next: Done
  Label: FileAnalysis
  MaxConcurrency: 1000
  ItemBatcher:
    MaxItemsPerBatch: 1
  ItemReader:
    Resource: "arn:aws:states:::s3:getObject"
    ReaderConfig:
      InputType: "JSON"
    Parameters:
      Bucket.$: $.object.bucket
      Key.$: $.object.key

What stacktrace or error message from your provider did you see?

TypeError: Cannot read property 'States' of undefined
    at C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless-step-functions\lib\deploy\stepFunctions\compileIamRole.js:22:42
    at C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\lodash\lodash.js:3585:27
    at C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\lodash\lodash.js:4967:15
    at baseForOwn (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\lodash\lodash.js:3032:24)
    at C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\lodash\lodash.js:4936:18
    at baseMap (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\lodash\lodash.js:3584:7)
    at map (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\lodash\lodash.js:9622:14)
    at Function.flatMap (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\lodash\lodash.js:9325:26)
    at getTaskStates (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless-step-functions\lib\deploy\stepFunctions\compileIamRole.js:12:12)
    at C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless-step-functions\lib\deploy\stepFunctions\compileIamRole.js:576:26
    at Array.forEach (<anonymous>)
    at ServerlessStepFunctions.compileIamRole (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless-step-functions\lib\deploy\stepFunctions\compileIamRole.js:566:32)   
    at processImmediate (internal/timers.js:464:21)
From previous event:
    at package:compileFunctions (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless-step-functions\lib\index.js:122:10)
    at PluginManager.runHooks (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless\lib\classes\plugin-manager.js:530:15)
    at async PluginManager.invoke (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless\lib\classes\plugin-manager.js:564:9)
    at async PluginManager.spawn (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless\lib\classes\plugin-manager.js:585:5)
    at async before:deploy:deploy (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless\lib\plugins\deploy.js:40:11)
    at async PluginManager.runHooks (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless\lib\classes\plugin-manager.js:530:9)
    at async PluginManager.invoke (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless\lib\classes\plugin-manager.js:563:9)
    at async PluginManager.run (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless\lib\classes\plugin-manager.js:604:7)
    at async Serverless.run (C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless\lib\serverless.js:170:5)
    at async C:\Users\alan_\Work\Projects\craftsman-serverless-api\node_modules\serverless\scripts\serverless.js:787:9
child_process.js:790
    err = new Error(msg);
          ^

Error: Command failed: npx cross-env NODE_ENV=development bash bin/deploy.sh -o encoder
    at checkExecSyncError (child_process.js:790:11)
    at execSync (child_process.js:863:15)
    at file:///C:/Users/alan_/Work/Projects/craftsman-serverless-api/scripts/deploy.mjs:30:3
    at ModuleJob.run (internal/modules/esm/module_job.js:183:25)
    at async Loader.import (internal/modules/esm/loader.js:178:24)
    at async Object.loadESM (internal/process/esm_loader.js:68:5)
    at async handleMainPromise (internal/modules/run_main.js:59:12) {
  status: 1,
  signal: null,
  output: [ null, null, null ],
  pid: 28572,
  stdout: null,
  stderr: null
}

Additional Data

• 3.21.0:
• 3.11.1:
• Windows:

Solution

There is a very simple solution for this, updating the switch case for Map to use either Iterator or ItemProcessor

function getTaskStates(states) {
  return _.flatMap(states, (state) => {
    switch (state.Type) {
      case 'Task': {
        return [state];
      }
      case 'Parallel': {
        const parallelStates = _.flatMap(state.Branches, branch => _.values(branch.States));
        return getTaskStates(parallelStates);
      }
      case 'Map': {
        const mapStates = state?.Iterator?.States || state?.ItemProcessor?.States;
        return getTaskStates(mapStates);
      }
      default: {
        return [];
      }
    }
  });
}

[timorthi]

I'm also running into this issue, which has actually been addressed in #536.

The above change hasn't been released yet but you can get around it by pinning the plugin against the latest commit in master in your package.json:

"serverless-step-functions": "serverless-operations/serverless-step-functions#42bd42350c33c418762bff6f75cb8b2d6e2d3729"

[akshaydk]

I tried above configuration with valid=false. SLS file is not generating the IAM policies required to download the file from s3 bucket. Any suggestion for fixing that?

[keithrozario]

encountered the same issue as well :(

[akshaydk]

This is fixed in the latest release! You can try and leave feedback.

[cwardcode]

@akshaydk

For a non-distributed map (INLINE), I was receiving the above error on the last release, but that seems to be resolved now. Unfortunately, though I'm now receiving this:

✖ Stack env-my-map-step-function failed to deploy (65s)
Environment: darwin, node 16.18.0, framework 3.26.0 (local), plugin 6.2.2, SDK 4.3.2
Credentials: Local, environment variables
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
UPDATE_FAILED: FeatureDashmyDashmapDashfunctionRole (AWS::IAM::Role)
The policy failed legacy parsing (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 578bcbb5-63be-4ec6-b2f6-7b6783a479eb; Proxy: null)

If a separate issue needs to be filed, please let me know and I'll do so.

Edit: Looks like this is actually a result of this bug that was filed almost 3 years ago

[lopburny]

@cwardcode I tried to reproduce with INLINE mode, but everything seems fine with the latest version. Good to raise a separate issue with some details so I can help you.