Skip to content

Using Imagemagick in Cloud Code #648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
oyvindvol opened this issue Feb 25, 2016 · 35 comments
Closed

Using Imagemagick in Cloud Code #648

oyvindvol opened this issue Feb 25, 2016 · 35 comments

Comments

@oyvindvol
Copy link

Hi,

This is not really an issue with Parse-Server, rather a cry for help with using Imagemagick instead of Parse-Image in my cloud code functions. Can anyone point me in the right direction to achieve the same as expressed in my old cloud code:

Parse.Cloud.beforeSave("TheClass", function(request, response) {
  var theObject= request.object;

  Parse.Cloud.httpRequest({
    url: theObject.get("img").url()
  }).then(function(response) {
    var image = new Image();
    return image.setData(response.buffer);
  }).then(function(image) {
    var currentWidth = image.width();
    var currentHeight = image.height();
    var ratio = currentHeight / currentWidth;
    if (currentWidth > 640) {
      return image.scale({
        width: 640,
        height: 640 * ratio
      });
    } else {
      return image;
    }
  }).then(function(image) {
    return image.setFormat("JPEG");
  }).then(function(image) {
    return image.data();
  }).then(function(buffer) {
    var base64 = buffer.toString("base64");
    var cropped = new Parse.File("thumb.jpeg", { base64: base64 });
    return cropped.save();
  }).then(function(cropped) {
    theObject.set("thumb", cropped);
  }).then(function(result) {
    response.success();
  }, function(error) {
    response.error(error);
  });
});

Any help will be much appreciated!

@oyvindvol
Copy link
Author

@christianmarth Thanks! E.g. in the first link example (EasyImage), how would you initialize an image from the response.buffer like I do in line 8 of my pasted code?

@flovilmart
Copy link
Contributor

@oyvindvol I wrote a wrapper for Parse.Image that has 100% compatibility with the original parse-image:

https://github.com/flovilmart/parse-image.

@nitrag
Copy link

nitrag commented Feb 25, 2016

I was just getting to the point where I needed this too. Thanks @flovilmart !

@flovilmart
Copy link
Contributor

@nitrag @oyvindvol the installation is automated on debian and OSX. Installation in Windows is not working yet. For heroku, you need to add a build pack (as explained in the README)

@oyvindvol
Copy link
Author

@flovilmart Nice! So, to use this on a heroku based install of parse-server, i just add imagemagick , graphicsmagick and your parse-image to my package.json, and add var Image = require("parse-image") to the top of my cloud code file?

@flovilmart
Copy link
Contributor

@oyvindvol follow the instructions here: https://github.com/flovilmart/parse-image#heroku-installation

then:

var Image = require("parse-image");
var image = new Image();
image.setData(buffer).then(function(){

});

Given the code you provided, it seems that it should work right away.

@oyvindvol
Copy link
Author

@flovilmart Ok, so now I did only the "Heroku installation" part, and after deploy the heroku logs say "Cannot find module 'parse-image'". Then I added 'parse-image' as a dependency in package.json, and I still get "cannot find module 'parse-image'" in the logs.

@flovilmart
Copy link
Contributor

Did you run npm install --save parse-image?

@oyvindvol
Copy link
Author

@flovilmart On Heroku server?

@flovilmart
Copy link
Contributor

does it work locally with foreman?

@oyvindvol
Copy link
Author

I haven`t tested locally, I am just trying to set it up on my parse-server-example that I deployed directly to Heroku. I configured the heroku-buildpack-graphicsmagick via git bash, and push to heroku master. I can see that the buildpack is installed, but am I also supposed to put something in my package.json for it to work in my cloud code?

@flovilmart
Copy link
Contributor

the build pack is installing the binaries required to parse-image to work correctly.
When you push to heroku, you you see parse-image being installed? Can you give me the heroku logs?

@oyvindvol
Copy link
Author

Ok, starting from fresh. I added the buildpack in git bash. I remove everything regarding parse-image from my package.json, then i do git push heroku master. This shows in the console:

$ git push heroku master
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 328 bytes | 0 bytes/s, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Fetching set buildpack https://github.com/mcollina/her
remote: -----> GraphicsMagick app detected
remote: -----> Installing zlib 1.2.8
remote: -----> Installing libpng 1.5.17
remote: -----> Installing nasm 2.10.09
remote: -----> Installing libjpeg-turbo 1.3.0
remote: -----> Installing graphicsmagick 1.3.23
remote: -----> Building runtime environment for graphicsmagick
remote: -----> Using set buildpack heroku/nodejs
remote: -----> Node.js app detected
remote:
remote: -----> Creating runtime environment
remote:
remote:        NPM_CONFIG_LOGLEVEL=error
remote:        NPM_CONFIG_PRODUCTION=true
remote:        NODE_ENV=production
remote:        NODE_MODULES_CACHE=true
remote:
remote: -----> Installing binaries
remote:        engines.node (package.json):  >=4.1
remote:        engines.npm (package.json):   unspecified (use default
remote:
remote:        Resolving node version >=4.1 via semver.io...
remote:        Downloading and installing node 5.6.0...
remote:        Using default npm version: 3.6.0
remote:
remote: -----> Restoring cache
remote:        Loading 2 from cacheDirectories (default):
remote:        - node_modules
remote:        - bower_components (not cached - skipping)
remote:
remote: -----> Building dependencies
remote:        Pruning any extraneous modules
remote:        Installing node modules (package.json)
remote:
remote: -----> Caching build
remote:        Clearing previous node cache
remote:        Saving 2 cacheDirectories (default):
remote:        - node_modules
remote:        - bower_components (nothing to cache)
remote:
remote: -----> Build succeeded!
remote:        ├── [email protected]
remote:        ├── [email protected]
remote:        ├── [email protected]
remote:        └── [email protected]
remote:
remote:
remote: -----> Discovering process types
remote:        Procfile declares types     -> (none)
remote:        Default types for buildpack -> web
remote:
remote: -----> Compressing...
remote:        Done: 33.8M
remote: -----> Launching...
remote:        Released v140
remote:        https://<appname>.herokuapp.com/ deployed to Heroku
remote:
remote: Verifying deploy... done.
To https://git.heroku.com/<appname>.git
   241bb07..94bc7b3  master -> master

After deploy finished, this is the heroku logs:

2016-02-25T13:44:26.959982+00:00 heroku[api]: Release v140 created by 
2016-02-25T13:44:27.098280+00:00 heroku[slug-compiler]: Slug compilation started
2016-02-25T13:44:27.098284+00:00 heroku[slug-compiler]: Slug compilation finished
2016-02-25T13:44:27.472830+00:00 heroku[web.1]: State changed from crashed to starting
2016-02-25T13:44:31.180828+00:00 heroku[web.1]: Starting process with command `npm start`
2016-02-25T13:44:34.223438+00:00 app[web.1]:
2016-02-25T13:44:34.223454+00:00 app[web.1]: > [email protected] start /app
2016-02-25T13:44:34.223455+00:00 app[web.1]: > node index.js
2016-02-25T13:44:34.223456+00:00 app[web.1]:
2016-02-25T13:44:35.351267+00:00 app[web.1]: npm ERR! Linux 3.13.0-77-generic
2016-02-25T13:44:35.351727+00:00 app[web.1]: npm ERR! argv "/app/.heroku/node/bin/node" "/app/.heroku/node/bin/npm" "start"
2016-02-25T13:44:35.352006+00:00 app[web.1]: npm ERR! node v5.6.0
2016-02-25T13:44:35.352641+00:00 app[web.1]: npm ERR! npm  v3.6.0
2016-02-25T13:44:35.352863+00:00 app[web.1]: npm ERR! code ELIFECYCLE
2016-02-25T13:44:35.353074+00:00 app[web.1]: npm ERR! [email protected] start: `node index.js`
2016-02-25T13:44:35.353260+00:00 app[web.1]: npm ERR! Exit status 1
2016-02-25T13:44:35.353469+00:00 app[web.1]: npm ERR!
2016-02-25T13:44:35.353663+00:00 app[web.1]: npm ERR! Failed at the [email protected] start script 'node index.js'.
2016-02-25T13:44:35.353868+00:00 app[web.1]: npm ERR! Make sure you have the latest version of node.js and npm installed.
2016-02-25T13:44:35.354073+00:00 app[web.1]: npm ERR! If you do, this is most likely a problem with the parse-server-example package,
2016-02-25T13:44:35.354285+00:00 app[web.1]: npm ERR! not with npm itself.
2016-02-25T13:44:35.354466+00:00 app[web.1]: npm ERR! Tell the author that this fails on your system:
2016-02-25T13:44:35.354682+00:00 app[web.1]: npm ERR!     node index.js
2016-02-25T13:44:35.354864+00:00 app[web.1]: npm ERR! You can get information on how to open an issue for this project with:
2016-02-25T13:44:35.355304+00:00 app[web.1]: npm ERR! Or if that isn't available, you can get their info via:
2016-02-25T13:44:35.361157+00:00 app[web.1]: npm ERR! Please include the following file with any support request:
2016-02-25T13:44:35.361327+00:00 app[web.1]: npm ERR!     /app/npm-debug.log
2016-02-25T13:44:35.360891+00:00 app[web.1]:
2016-02-25T13:44:35.355700+00:00 app[web.1]: npm ERR! There is likely additional logging output above.
2016-02-25T13:44:35.355092+00:00 app[web.1]: npm ERR!     npm bugs parse-server-example
2016-02-25T13:44:35.355515+00:00 app[web.1]: npm ERR!     npm owner ls parse-server-example
2016-02-25T13:44:35.329120+00:00 app[web.1]: module.js:341
2016-02-25T13:44:35.329124+00:00 app[web.1]:     throw err;
2016-02-25T13:44:35.329125+00:00 app[web.1]:     ^
2016-02-25T13:44:35.329125+00:00 app[web.1]:
2016-02-25T13:44:35.329126+00:00 app[web.1]: Error: Cannot find module 'parse-image'
2016-02-25T13:44:35.329128+00:00 app[web.1]:     at Function.Module._load (module.js:290:25)
2016-02-25T13:44:35.329127+00:00 app[web.1]:     at Function.Module._resolveFilename (module.js:339:15)
2016-02-25T13:44:35.329129+00:00 app[web.1]:     at Module.require (module.js:367:17)
2016-02-25T13:44:35.329129+00:00 app[web.1]:     at require (internal/module.js:16:19)
2016-02-25T13:44:35.329131+00:00 app[web.1]:     at Module._compile (module.js:413:34)
2016-02-25T13:44:35.329130+00:00 app[web.1]:     at Object.<anonymous> (/app/cloud/main.js:1:75)
2016-02-25T13:44:35.329131+00:00 app[web.1]:     at Object.Module._extensions..js (module.js:422:10)
2016-02-25T13:44:35.329132+00:00 app[web.1]:     at Module.load (module.js:357:32)
2016-02-25T13:44:35.329132+00:00 app[web.1]:     at Function.Module._load (module.js:314:12)
2016-02-25T13:44:35.329133+00:00 app[web.1]:     at Module.require (module.js:367:17)
2016-02-25T13:44:35.340090+00:00 app[web.1]:
2016-02-25T13:44:36.035081+00:00 heroku[web.1]: State changed from crashed to starting
2016-02-25T13:44:36.033934+00:00 heroku[web.1]: State changed from starting to crashed
2016-02-25T13:44:36.030078+00:00 heroku[web.1]: Process exited with status 1
2016-02-25T13:45:58.111674+00:00 heroku[web.1]: Starting process with command `npm start`
2016-02-25T13:46:00.447983+00:00 app[web.1]: > node index.js
2016-02-25T13:46:00.447984+00:00 app[web.1]:
2016-02-25T13:46:00.447981+00:00 app[web.1]: > [email protected] start /app
2016-02-25T13:46:00.447964+00:00 app[web.1]:
2016-02-25T13:46:01.919098+00:00 app[web.1]: npm ERR! Linux 3.13.0-77-generic
2016-02-25T13:46:01.901180+00:00 app[web.1]: module.js:341
2016-02-25T13:46:01.901184+00:00 app[web.1]:
2016-02-25T13:46:01.901186+00:00 app[web.1]:     at Function.Module._load (module.js:290:25)
2016-02-25T13:46:01.901190+00:00 app[web.1]:     at Module.require (module.js:367:17)
2016-02-25T13:46:01.919662+00:00 app[web.1]: npm ERR! node v5.6.0
2016-02-25T13:46:01.920572+00:00 app[web.1]: npm ERR!
2016-02-25T13:46:01.901189+00:00 app[web.1]:     at Module.load (module.js:357:32)
2016-02-25T13:46:01.920799+00:00 app[web.1]: npm ERR! Make sure you have the latest version of node.js and npm installed.
2016-02-25T13:46:01.921630+00:00 app[web.1]: npm ERR! Or if that isn't available, you can get their info via:
2016-02-25T13:46:01.920445+00:00 app[web.1]: npm ERR! Exit status 1
2016-02-25T13:46:01.901183+00:00 app[web.1]:     ^
2016-02-25T13:46:01.901186+00:00 app[web.1]:     at Module.require (module.js:367:17)
2016-02-25T13:46:01.921130+00:00 app[web.1]: npm ERR! Tell the author that this fails on your system:
2016-02-25T13:46:01.901188+00:00 app[web.1]:     at Module._compile (module.js:413:34)
2016-02-25T13:46:01.901185+00:00 app[web.1]:     at Function.Module._resolveFilename (module.js:339:15)
2016-02-25T13:46:01.920340+00:00 app[web.1]: npm ERR! [email protected] start: `node index.js`
2016-02-25T13:46:01.901187+00:00 app[web.1]:     at require (internal/module.js:16:19)
2016-02-25T13:46:01.921412+00:00 app[web.1]: npm ERR! You can get information on how to open an issue for this project with:
2016-02-25T13:46:01.901185+00:00 app[web.1]: Error: Cannot find module 'parse-image'
2016-02-25T13:46:01.920061+00:00 app[web.1]: npm ERR! npm  v3.6.0
2016-02-25T13:46:01.920218+00:00 app[web.1]: npm ERR! code ELIFECYCLE
2016-02-25T13:46:01.901182+00:00 app[web.1]:     throw err;
2016-02-25T13:46:01.921287+00:00 app[web.1]: npm ERR!     node index.js
2016-02-25T13:46:01.901190+00:00 app[web.1]:     at Function.Module._load (module.js:314:12)
2016-02-25T13:46:01.901188+00:00 app[web.1]:     at Object.<anonymous> (/app/cloud/main.js:1:75)
2016-02-25T13:46:01.921847+00:00 app[web.1]: npm ERR! There is likely additional logging output above.
2016-02-25T13:46:01.901189+00:00 app[web.1]:     at Object.Module._extensions..js (module.js:422:10)
2016-02-25T13:46:01.919493+00:00 app[web.1]: npm ERR! argv "/app/.heroku/node/bin/node" "/app/.heroku/node/bin/npm" "start"
2016-02-25T13:46:01.911119+00:00 app[web.1]:
2016-02-25T13:46:01.921026+00:00 app[web.1]: npm ERR! not with npm itself.
2016-02-25T13:46:01.921739+00:00 app[web.1]: npm ERR!     npm owner ls parse-server-example
2016-02-25T13:46:01.920685+00:00 app[web.1]: npm ERR! Failed at the [email protected] start script 'node index.js'.
2016-02-25T13:46:01.921524+00:00 app[web.1]: npm ERR!     npm bugs parse-server-example
2016-02-25T13:46:01.925610+00:00 app[web.1]: npm ERR!     /app/npm-debug.log
2016-02-25T13:46:01.920921+00:00 app[web.1]: npm ERR! If you do, this is most likely a problem with the parse-server-example package,
2016-02-25T13:46:01.925509+00:00 app[web.1]: npm ERR! Please include the following file with any support request:
2016-02-25T13:46:01.925339+00:00 app[web.1]:
2016-02-25T13:46:02.554694+00:00 heroku[web.1]: Process exited with status 1
2016-02-25T13:46:02.564724+00:00 heroku[web.1]: State changed from starting to crashed

@flovilmart
Copy link
Contributor

it seems that your package.json is not properly updated, or that heroku don't like it..

 Build succeeded!
remote:        ├── [email protected]
remote:        ├── [email protected]
remote:        ├── [email protected]
remote:        └── [email protected]

Try to run on your machine: npm install --save parse-image and then deploy to heroku

@oyvindvol
Copy link
Author

After adding parse-image to my package.json the module is found.

But now I get this error:

[Error: Command failed: identify.im6: no decode delegate for this image format `/tmp/magick-x4qqyfba'

@oyvindvol
Copy link
Author

Ok, thanks a lot so far @flovilmart!

I tried console.logging the url to my uploaded ParseFile, console.log(theObject.get("img").url());. The url was http, not https. Then i ran this command in the heroku bash identify -verbose <URL-TO-FILE>, and there the identify seemed to work fine at least.

identify -verbose http://<app>/parse/files/sv78uRNzD4RPUKW9

Image: 5ef1d74591c83c189609957ee1f18860_portal.PNG
  Base filename: 5ef1d74591c83c189609957ee1f18860_portal.PNG
  Format: PNG (Portable Network Graphics)
  Class: DirectClass
  Geometry: 2438x1400+0+0
  Resolution: 75.59x75.59
...
...
and so on

@flovilmart
Copy link
Contributor

Is your response.buffer set?

@oyvindvol
Copy link
Author

Yes

@flovilmart
Copy link
Contributor

Uhm... that's very very odd indeed. Did you try on you local machine, see if it's heroku related or something else?

@flovilmart
Copy link
Contributor

You should probably open an issue on parse-image instead of there. Closing for now.

@oyvindvol
Copy link
Author

Well, initially this issue was about using the recommended Imagemagick in parse server cloud code. I am still interested in examples on that though..

@flovilmart
Copy link
Contributor

OK, can you post an issue on parse-image I'll re-open in that case.

@flovilmart flovilmart reopened this Feb 25, 2016
@simonbengtsson
Copy link
Contributor

I used jimp to replace parse-image when I migrated our app. It is written entirely in JavaScript for Node so no external dependencies needed (no special install on heroku or windows). It probably has worse performance than imagemagick, but I have yet to run any benchmarks. We are scaling a lot of images in our app however and so far it looks promising.

@oyvindvol
Copy link
Author

@simonbengtsson That looks lightweight and good. Are you using jimp to e.g. save thumbnails via cloud code? I am having trouble getting that work. With the code provided below, everything seems OK, and the thumb is saved on the "Message" object, but viewing the thumb in the browser with the file URL is not working.

var Jimp = require("jimp");

Parse.Cloud.beforeSave("Message", function(request, response) {
  var obj = request.object;

  if (!obj .get("img")) {
    response.success();
    return;
  }

  Jimp.read(obj.get("img").url())
  .then(function(image) {
    var currentWidth = image.bitmap.width;
    var currentHeight = image.bitmap.height;
    var ratio = currentHeight / currentWidth;
    if (currentWidth > 640) {
      return image.resize(640, 640 * ratio)
    } else {
      return image;
    }
  })
  .then(function(image) {
    console.log("### 1");
    var base64 = image.hash();
    var cropped = new Parse.File("thumb.jpeg", { base64: base64 });
    return cropped.save();
  })
  .then(function(cropped) {
    console.log("### 2");
    obj.set("thumb", cropped);
  })
  .then(function(result) {
    console.log("### 3");
    response.success();
  }, function(error) {
    console.log("### 4 ERROR: " + error);
    response.error(error);
  })
  .catch(function (err) {
    console.log("### 5 ERROR: " + err);
  });
});

@simonbengtsson
Copy link
Contributor

Yes we are generating thumbnails in cloud code using jimp. Your issue is probably that the string from image.hash() cannot be used as data for parse files. Try something like this instead:

return new Parse.Promise((resolve, reject) => {
    image.resize(100, 100).getBuffer(Jimp.MIME_JPEG, (err, buffer) => {
        if (err) return reject(err);
        var file = new Parse.File('picture.jpg', {base64: buffer});
        resolve(file.save());
    });
});

@oyvindvol
Copy link
Author

@simonbengtsson Thanks! :) That worked..

@nitrag
Copy link

nitrag commented Mar 3, 2016

Since it took me several hours to get this correct I'm posting here so the next guy has an easier time:

Parse.Cloud.beforeSave("Photos", function(request, response) {
    Parse.Cloud.useMasterKey();

    createThumbnail(request.object.get("Image").url(), 360, 360).then(function(shrunk) {
        request.object.set("ImageThumbnail", shrunk);
        console.log("Thumbnail generated/added: " + JSON.stringify(shrunk));
    }).then(function(result) {
            response.success("thumbnail saved");
        }, function(error) {
            response.error("thumbnail creation error: " + error.code + "("+ error.message + ")");
    });
});
function createThumbnail(url, maxWidth, maxHeight){
    var Jimp = require("jimp");
    Parse.Cloud.useMasterKey();
    if(url === ""){
        console.log("url empty");
        return;
    }

    console.log("Generating Thumbnail...");
    return Jimp.read(url).then(function(image) {
        console.log("Buffer read, resizing...");
        var ratioX = maxWidth / image.bitmap.width; 
        var ratioY = maxHeight / image.bitmap.height;
        var ratio = Math.min(ratioX, ratioY);

        var w = image.bitmap.width * ratio;
        var h = image.bitmap.height * ratio;

        return image.resize(w, h );
    }).then(function(image) {
        console.log("Resized, setting quality...");
        return image.quality(90);
    }).then(function(image) {
        console.log("Quality set, return buffer");
        var promise = new Parse.Promise();
            image.getBuffer(Jimp.MIME_JPEG, function(err, buffer){
                if (err) return promise.reject(err);
                promise.resolve(buffer);
            });
        return promise;
    }).then(function(buffer) {
        console.log("Buffer returned, creating Parse File");
        var shrunk = new Parse.File("thumbnail.jpg", { base64: buffer }, "image/jpeg");
        return shrunk.save();
    }).then(function(shrunk) {
            return shrunk;
        }, function(error) {
            console.log("thumbnail generation error: " + error.code + "("+ error.message + ")");
            return;
    });

}

@6thfdwp
Copy link
Contributor

6thfdwp commented May 14, 2016

Thanks for sharing this pure js image module, here is the benchmarks I found for jimp in comparison with other libraries.
lovell/sharp#275 (comment)

It seems enough for common image processing cases.

@majidhassan
Copy link

@flovilmart I tried your parse-image module on Heroku for a simple thumbnail generation. I did not need to change a single line of code, only added "parse-image": "~0.1.1" in my package.json dependencies, and it works fine so far.

Great job!

@rickkrauland
Copy link

@nitrag Thanks for your post. Fwiw, I had to add a buffer.toString conversion before I could get everything to work.

var base64 = buffer.toString("base64");
var shrunk = new Parse.File("thumbnail.jpg", { base64: base64 }, "image/jpeg");         
return shrunk.save({useMasterKey: true});

@parkej60
Copy link

parkej60 commented Sep 23, 2016

This all seems to be working great for me using jimp until the part where I try to set it to the object in the beforeSave hook. I see the images showing up in S3 and it comes back correctly. I have other attributes that are getting set correctly as well. It is just the 'pictureThumbnail' that isn't. For some reason it just seems as if it's not updating the request.object. @rickkrauland Was there something you had to do different then what is documented in @nitrag 's example?

createThumbnail(url, 300, 300).then(function(shrunk) {
            restaurant.set("pictureThumbnail", shrunk);
            console.log("Thumbnail generated/added: " + JSON.stringify(shrunk));
        }).then(function(result) {
                response.success("thumbnail saved");
            }, function(error) {
                response.error("thumbnail creation error: " + error.code + "("+ error.message + ")");
        });

@parkej60
Copy link

So... turns out that response.success doesn't want a string sent along with it. I removed "thumbnail saved" from response.success() and it worked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants