Cloud Code on AWS lambda

[flovilmart]

AWS Lambda seems a good candidate to host CloudCode functions.
Explore possible intergrations via HTTPS over and custom deployment package.

[alexnaldo]

Will be a great option!!!
Integrating Parse Server with Lambda opens many opportunities to integrate it with AWS.

[flovilmart]

I belive the effort is minimal, but I'll try to make an example.

[alexnaldo]

Without Parse.com we will need to think about infrastructure again, a big problem for startups. Using Lambda, API Getaway and others Amazon Services can turn it more friendly and safe.

[gfosco]

This would make a really neat example app... Hope to see someone put it together.

[flovilmart]

When I have time :)

[rafapetter]

Still dont know how to create it but I'll be glad to test it

[flovilmart]

Look at the AWS lambda documentation and use the Parse hooks (that are experimental) to register your URLs

[rafapetter]

Ok, thanks

[messagenius-admin]

Guys, i just read that Lambda now supports node 4.3.2
Is it possible to host the Parse server in Lambda? (I mean the full server, not just the cloud code)

[alexnaldo]

I don't think possible. Lambda is stateless and Parse Server is statefull.

[rucas]

@flovilmart I got Express, the parse-server, and the parse-dashboard on AWS Lambda using the aws-serverless-express and the serverless framework using this tutorial and the example of express with parse-server

[flovilmart]

Ah nice!

@alexnaldo

 Parse Server is statefull.

What do you mean by stateful? Besides the caches that warm up with time, otherwise nothing is stateful.

[abdulwahid24]

@rucas What about storage ?

[rucas]

@abdulwahid24 I didn't use the parse storage features, but I'm sure theres a way to integrate with AWS S3 or some other cloud storage

[abdulwahid24]

@rucas But in order to use parse we required to connect database such mongo db. That's what I wanted to know how did you manage that part. Thanks

[rucas]

@abdulwahid24 Just set up a mongolab db or a mongodb on AWS and add the connection string to the parse server config

[JacobJT]

Awesome to know you can throw a full blown parse-server onto lambda!

For me, the powerful part is combining with step functions to schedule tasks. I previously had a few features implemented by running very frequent background jobs to search a queue for tasks that need to be completed and to complete them. A large downside was that our job logs were filled with instances of this job that didn't even do anything because there was nothing to do, and we ended up missing some failures of a couple of our daily background jobs due to corrupt data.

AWS Step Functions charge per state transition, and are relatively cheap, and can trigger lambda functions. I'm beginning to instantiate a Step Function instance from cloud code on the standard AWS EB / EC2 hosted parse-server to add a delay that will then trigger a Lambda function that calls my actual cloud code function.

I have a couple features that require checking the status of data, and either they terminate or schedule another check later on after doing some action. So, I need to figure out if it'd be better to have one big Step Functions with Lambda steps that handles the whole flow (get the cloud code response and terminate / move to next step appropriately), or individual step functions that handle each step with that delay, and the cloud code function will schedule another Step function. Since Lambda charges for server usage, it would save a small amount by not having to wait for the response from cloud code and performing more actions, but I suppose it depends on how Step Functions charges. It's by state transition, and the flow shows a start and end transition, but I'm not 100% those count. If my flow is Start > Delay > lambda function > end, is that two counts for the delay and lambda function steps? Or is it three for each > transition?

[mfkenson]

@rucas i am very interested in what u've done - parse server -> lambda! is there any chance you could share the integrated example? great thanks!

[danepowell]

I'm really excited to try this as well, could we reopen this issue until we have a working public example? And maybe rename it to reflect that we want to run the entire server on lambda...

Also, can anyone comment on how the response time differs between standard hosting and lambda? It seems like lambda would take longer even on a "warm" start...

[flovilmart]

@danepowell the best is probably that you try it out for yourself, and understand the limitations, advantages of running the server in lambdas

[danepowell]

I just got a basic version of Parse running on Lambda following tips provided by @rucas . The only significant problem I've seen so far is that the filesystem is read-only, causing the Parse logger to throw a ton of errors:

Error: EROFS: read-only file system, open '/var/task/logs/parse-server.info.2018-07-23

Is there any way to disable the Parse logger, or switch it from a file logger to console output?

[flovilmart]

I’ll have a look at how to disable the logs

[flovilmart]

You can set logsFolder to ‘null’ to disable file based logging:
https://github.com/parse-community/parse-server/blob/master/src/Options/Definitions.js#L66

[rucas]

you can also set logs to /tmp you get 512mb of write space there

https://docs.aws.amazon.com/lambda/latest/dg/limits.html

[danepowell]

Thanks, setting it to /tmp seemed to work! So far, everything seems to be working pretty well on lambda. The only annoying thing I've found is that the logging in general is harder to use... doesn't stream as quickly as Heroku, it's harder to get request info from the API Gateway / router, etc... Maybe using some sort of Winston adapter to send Parse logs directly to CloudWatch would help, but that's a pretty low priority for me.

[danepowell]

@rucas were you ever able to get push notifications working reliably from a Parse Server running on Lambda? That's the last obstacle I haven't been able to overcome.

If I send pushes one at a time, usually they go through. But if I try to send them in batches, most don't get sent. My theory is that because pushes operate on a message queue / publisher / subscriber model, Lambda terminates the process before the queue finishes running.

[rucas]

@danepowell I have not tried out the push notifications 😿. I think it uses the DB for a message queue so you're good there as long as you got a MongoDB setup somewhere and is accessible.

However this sounds like maybe an error where your job takes too long. Have you tried increasing the execution time. I would also check the Cloudwatch logs to see if the logging for the push notifications clips out and ends too early.

Also make sure the client isn't using a websocket connection, I don't think websockets are supported for AWS API Gateway + Lambda

[danepowell]

If there's one thing you might have noticed from my pattern of support requests, it's that Node.js sync/async architectures are not my forte 😄 If there was enough guidance I might be able to work on a fix, but I'm guessing it would be a pretty big challenge. I'd love to help if I can though, even if only to test a fix.

[flovilmart]

The fix is quite architectural after all. The goal is to decouple the http endpoint from the expansion (querying all installation) to sending the push to the adapters.

I believe we can move the expansion phase that is done in the push queue to the push worker, after all, the push worker already has a pub sub configuration.

This way, this will hit the queue right away, and we’ll be able to guarantee that the push is being enqueued (not expanded).

Can you look through the code and let me know if what I describe make sense to you? Important files would be PushController, PushQueue, PushWorker.

[huksley]

Hi, is were any updates on this? Serverless are gaining traction and it would be good to see such a comprehensive system as parse-server to be available as AWS Lambda

[jcguarinpenaranda]

Hello, I'm also very interested on this :) serverless seems like a great way to have a Parse Server working

[jimcamut]

@rucas I know this is a really old thread, but if you see this question: did you have any issues with require('parse-server') on Lambda? I can get everything to work on local serverless, but require('parse-server') on an actual Lambda doesn't work for me.

[grosscorporation]

It wont work if you don't push the parse-server module, but are you sure it's the only thing not working because Lambda and parse make no sense to me at least just yet. and remember that evertime you call a lambda you are actually starting a parse-server meaning its stateless even with aws-serverless-express.

[jimcamut]

@GoGross Something is definitely amiss because I'm getting other errors even when I push up the parse-server module. Regarding restarts, a warmup plugin in serverless keeps it quick (in theory), but this has been more of an experiment than a practical solution. I'm about to abandon this approach if I keep hitting dead ends.

[grosscorporation]

by using cold starts plugin you are going against the main reason why you probably chose Lambda, pay for what you use. Cold starts will increase calls to the function, cloudwatch and I am not sure what else

[grosscorporation]

I really recommend back4app for now

[jimcamut]

@GoGross I've already got Parse Server running on in production in AWS (Nodejs, non-lambda). I'd agree that using Parse Server in an environment it was intended for (on AWS or something like back4app) is the best way to go, especially for the built in caching, efficient DB connections, etc. There is definitely still a case for Serverless/lambdas for some specific scenarios, but it's proving to be hacky, and a headache to configure properly. I'm putting my attempt on pause for now.

[rucas]

I know this is a really old thread, but if you see this question: did you have any issues with require('parse-server') on Lambda? I can get everything to work on local serverless, but require('parse-server') on an actual Lambda doesn't work for me.

@jimcamut sorry for the late reply, I'm catching up now. I would export your lambda function from AWS and make sure parse-server is in your node_modules directory.

[kevflynn]

You can set logsFolder to ‘null’ to disable file based logging:
https://github.com/parse-community/parse-server/blob/master/src/Options/Definitions.js#L66

This doesnt' seem to work. Running locally, setting to null, it looks like the default value overrides and still sets to ./logs. Trying to get it up and running on App Engine Standard Env, and getting this error

[dplewis]

silent:true, verbose false works for me

[kevflynn]

Thanks for the reply @dplewis

This is my config
{ cloud: ${ROOT}/parse/cloud/index.js, clientId: Config.parse.clientId, databaseURI: Config.parse.dbURI, // Connection string for your MongoDB database masterKey: Config.parse.masterKey, // Keep this key secret! serverURL: Config.parse.serverUrl, // Don't forget to change to https if needed javascriptKey: Config.parse.javascriptKey, // Don't forget to change to https if needed revokeSessionOnPasswordReset: false, appId: Config.parse.appId, verbose: false, logsFolder: null, silent: true }

These are my config settings and every time i start the server locally, i see this get created:

And if i console log from the winston logger module, i see this when start the server, two separate outputs:

First output:
Logs Folder: ./logs/ JSON LOGS: false LOG LEVEL: info VERBOSE: false SILENT: undefined

Then Immediately a follow-up output:
Logs Folder: null JSON LOGS: false LOG LEVEL: info VERBOSE: false SILENT: true

I assume, the problem is the first output seems to be called with _defaults and second is with my config.

[kevflynn]

The problem is - that first call on Google App Engine crashes the server because it doesn't have permission to make those file writes

[kevflynn]

So @dplewis - it looks like just importing the parse server module, instantiates configuration with default settings. Any way around this? So I think it's instantiating twice, once on important and once when you instantiate with your own config settings

[kevflynn]

I think I found the problem:
This function:

function defaultLogger() { const options = { logsFolder: _defaults.default.logsFolder, jsonLogs: _defaults.default.jsonLogs, verbose: _defaults.default.verbose, silent: _defaults.default.silent }; console.log('RETURNINGING A DEFAULT LOGGER: ', defaultLogger); const adapter = new _WinstonLoggerAdapter.WinstonLoggerAdapter(options); return new _LoggerController.LoggerController(adapter, null, options); }

gets called from within logger.js upon importing the parseserver library, which instantiates the logger before having a chance to set default config.

Update.... Ok - next line down:

let logger = defaultLogger();

instantiates itself globally, so as soon as this file is included, it gets instantiated. Is that necessary?

[Taylorsuk]

@kevflynn is it possible to shed a little more light on your experiences with running Parse Server in Lambda, recommended set-up, issues, performance etc.

I've most concerned about cold boot times - what are you getting on average?

[Simone-cogno]

Has anybody a working example to start integrating parse-server to AWS Lambda?

[Taylorsuk]

@Simone-cogno I didn’t manage to get it working. From a concept I think there are real challenges with running parse, especially code start times etc.

The last thing I looked at was https://apex.sh/up which looked promising. However, since then I’ve moved to Docker on numerous servers with a Cloudflare load balancer / health check etc.

To code isolated cloud code functions as isolated singular files to server less endpoints is still a pipe dream, unfortunately.

[mtrezza]

Interesting thread. Trying to run parse server in a serverless environment sounds like trying to drive a car on the train.

The issue is long closed but it seems to pop up in Google search results and attract people who intend to explore this alley.

Lambda is a building block from a different paradigm and seems more comparable to Cloud Code, which is what this thread was originally about, if I follow correctly, before it morphed into running the whole Parse Server on Lambda. Even if the experiment of running Parse Server on Lambda works technically, I am sure that the Lambda pricing structure works against it at scale, because spinning up Parse Server is contrary to what Lambda is intended for.

If a PaaS such as AWS Elastic Beanstalk is too much work, then I suggest to dockerize Parse Server and deploy it on AWS Fargate which abstracts the server environment and makes it more of a SaaS.