feat: Add options to disable case insensitivity and force email and username to lower case #8042

Moumouls · 2022-06-13T13:46:06Z

New Pull Request Checklist

I am not disclosing a vulnerability.
I am creating this PR in reference to an issue.

Issue Description

Add options to be able to support MongoDB Serverless (no collation support)

Related issue: #7937

Approach

Add 2 abstract options disableCaseInsensitivity and forceEmailAndUsernameToLowerCase to allow a developer to disable default collations of Parse Server and forceEmailAndUsernameToLowerCase to keep clean usernames and emails in the database.

TODOs before merging

Add tests
Add changes to documentation (guides, repository pages, in-code descriptions)
Add security check
A changelog entry is created automatically using the pull request title (do not manually add a changelog entry)

parse-github-assistant · 2022-06-13T13:46:08Z

Thanks for opening this pull request!

🎉 We are excited about your hands-on contribution!

codecov · 2022-06-13T13:53:19Z

Codecov Report

Patch coverage: 73.33% and project coverage change: -0.13 ⚠️

Comparison is base (65e5879) 94.33% compared to head (15710ee) 94.20%.

❗ Current head 15710ee differs from pull request most recent head 24ab89b. Consider uploading reports for the commit 24ab89b to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##            alpha    #8042      +/-   ##
==========================================
- Coverage   94.33%   94.20%   -0.13%     
==========================================
  Files         183      182       -1     
  Lines       14515    13718     -797     
==========================================
- Hits        13692    12923     -769     
+ Misses        823      795      -28

Impacted Files	Coverage Δ
src/Options/Definitions.js	`100.00% <ø> (ø)`
src/Options/index.js	`100.00% <ø> (ø)`
src/Controllers/DatabaseController.js	`94.13% <73.33%> (+0.19%)`	⬆️

... and 132 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

mtrezza · 2022-06-13T20:40:29Z

Can we find something that reads easier than "disable case insensitivity" - avoiding the double negation. It could also be more descriptive - "case sensitivity for what"?

Moumouls · 2022-06-13T20:52:06Z

Currently, case-insensitivity is only used for indexes/queries on email/username fields on User class.

I can suggest:

preventCollation
disableCaseInsensitiveIndexAndQuery
preventCollationOnIndexAndQuery

Collation is also the error name returned by the Mongo driver if you try to use "case insensitive" index/query on Mongo Serverless. Using the same wording could help developers to fix the error.

I do not have other ideas @mtrezza 🙂

mtrezza

Is the collation applied for all actions regarding username / email, e.g. also password reset, email verification?

mtrezza · 2022-06-13T20:42:54Z

src/Controllers/DatabaseController.js

@@ -360,6 +360,15 @@ const relationSchema = {
  fields: { relatedId: { type: 'String' }, owningId: { type: 'String' } },
 };

+const maybeTransformUsernameAndEmailToLowerCase = (object, className, options) => {


remove maybe

use one term across all code, currently we have "force" and "transform", let's use "convert" for all

we can remove maybe but i'll keep transform. It's the term used in this file

update = transformObjectACL(update); transformUsernameAndEmailToLowerCase(update, className, this.options); transformAuthData(className, update, schema);

mtrezza · 2022-06-13T20:48:48Z

src/Options/index.js

@@ -94,6 +94,12 @@ export interface ParseServerOptions {
  databaseOptions: ?DatabaseOptions;
  /* Adapter module for the database */
  databaseAdapter: ?Adapter<StorageAdapter>;
+  /* Disable case insensitivity (collation) on queries and indexes, needed if the you use mongodb serverless
+  :ENV: PARSE_SERVER_DISABLE_DEFAULT_CASE_INSENSITIVITY */


Is it necessary to specify the ENV parameter? The options script should create that automatically from the param name. That also prevents the mistake, that the env var name is different than the param name, as is here.

Please use correct spelling ("MongoDB") and notation, this should be spelled like full sentences, as it goes into the API docs as prose text.

Can we find a param name that is easier to read and more descriptive, for example avoiding the double negation, for example enable... with a default value of true?

mtrezza · 2022-06-13T20:53:51Z

src/Options/index.js

+  disableCaseInsensitivity: ?boolean;
+  /* Force username and email to lowercase on create/update/login/signup. On queries client needs to ensure to send lowercase email/username
+  :ENV: PARSE_SERVER_FORCE_EMAIL_AND_USERNAME_TO_LOWER_CASE */
+  forceEmailAndUsernameToLowerCase: ?boolean;


Instead of "force" let call it "convert to".

Notation as if these were full sentences, this goes into the API docs.

what about ? transformUsernameToLowerCase and transformEmailToLowerCase

Sounds good

mtrezza · 2022-06-13T21:01:32Z

Collation is also the error name returned by the Mongo driver

Maybe lets call it enableCollation.... It may be useful to split up the collation for email / username into 2 separate params to make this more useful, also outside of a server-less usage. Email collation is also somewhat sensitive, because by specification [email protected] is not the same as [email protected], these are technically 2 different mailboxes.

Moumouls · 2022-06-14T07:34:14Z

Yes enableCollation, transformEmailToLowerCase and transformUsernameToLowerCase makes sense.

Do you have a suggestion about "default injection" of enableCollation to true. Or i just use a === false check in the code. It means that if enableCollation does not exists in the config, it's enabled @mtrezza

mtrezza · 2022-06-14T20:15:51Z

Let's simply use convert, it's a simple case conversion.

I don't understand the 2nd part.

dblythy

Some nice refactoring! Minor suggestions below.

src/Controllers/DatabaseController.js

src/Options/Definitions.js

Co-authored-by: dblythy <[email protected]>

Moumouls · 2022-07-03T10:32:28Z

Real usage feedback: With this PR and options activated I currently succeed in running Parse Server in a fully Serverless env using Google Cloud Run (CPU only reserved during requests) and MongoDB Atlas Serverless and it works pretty well.

parse-github-assistant · 2022-11-14T23:25:51Z

I will reformat the title to use the proper commit message syntax.

Moumouls · 2022-11-14T23:27:15Z

okay @mtrezza @dblythy I'll refresh this one with your feedback

mtrezza · 2023-04-09T20:44:49Z

To go ahead and merge this, could you address any open comments?

lilonpro · 2023-05-19T03:04:48Z

Does this resovle issue #8492 ? I just tried to run this PR and still hitting the same error message.

mattia1208 · 2023-08-18T13:20:31Z

Real usage feedback:
With the PR of @Moumouls and options activated I currently succeed in running Parse Server in a fully Serverless env using Azure Web App + Azure Cosmos DB for MongoDB and it works pretty well.

I think by now it is necessary to make Parse Server compatible with serverless environments.

mtrezza · 2023-08-18T18:48:06Z

@mattia1208 As you can see in the conversation this PR has open issues that need to be resolved to get merged. Maybe @Moumouls could review them to see which ones can be closed. And there is a merge conflict to resolve.

Moumouls · 2023-08-29T16:48:11Z

Hi @mattia1208 , happy to here that adjustments that i made allow to run parse server with Azure cosmos DB.

It's because Azure Cosmos like Mongo Serverless doesn't support insensitive indexes ?

A special not @mattia1208 , Parse Server is quite slow to start form a cold start, what is your serverless setup on Azure to prevent cold start. On my side on GCP Cloud Run i've some minute cron task needed for special events, it helps to keep atleast one instance active wihtout paying for a full hot docker instance.

mattia1208 · 2023-09-15T08:05:46Z

Hi @Moumouls ,

Yes, you're correct. Azure Cosmos, like Mongo Serverless, does not support case insensitive indexes.

For my serverless setup on Azure, I use the following:

Azure Web App + Database + Redis Cache
The Web App is hosted on Linux using a Premium V2 SKU, specifically a small size with a total of 210 ACU and 3.5GB of memory.
I also utilize Azure Cosmos DB with the API for MongoDB.

With this setup, I haven't experienced any cold start issues.

I hope this provides clarity!

b10f · 2023-09-21T14:33:42Z

@Moumouls, @mtrezza,

I recently came across this PR while searching for a solution for a particular problem.

Long story short, AWS’ DocumentDB also suffers from the same issue that this PR is intended to solve.

To have this PR merged would be a GIGANTIC help for our team, if we could utilize AWS DocumentDB Elastic Clusters to scale our database horizontally. I tried to install @Moumouls version from this exact branch from this PR and it works absolutely beautifully.

Is there any chance that you gentlemen can make this happen in the foreseeable future? Is there anything I can help with to make this happen?

Thanks.

mtrezza · 2023-09-21T16:25:36Z

@b10f see #8042 (comment); if you want to help, you could either suggest the changes via PR review (use code suggestion), or use this PR as a start for a new PR and incorporate the changes.

b10f · 2023-09-26T13:39:49Z

@Moumouls, any thoughts?

Moumouls · 2023-09-26T14:28:26Z

Hi @mtrezza @b10f @mattia1208 @lilonpro , if someone want to recycle my code and finish last details feel free to continue this one.

My team is moving out of Mongo Serverless for some specific reason ( but Mongo Serverless works really well with this PR), if it helps your teams now feel free to finish last details of this PR 🚀

We are close, but i will dedicate some time to parse-server on other PRs like GraphQL.

b10f · 2023-09-26T14:32:59Z

@Moumouls, since you are the original author of this PR, and considering that it only needs some merge conflicts to be sorted out, and addressing some review comments, I'm assuming it would not take too much effort.

Any chance that you could finish it? Even though you're moving out from Mongo Serverless, this PR would make it possible to use AWS DocumentDB and other providers that still have the same limitation.

Moumouls · 2023-09-26T14:52:12Z

It only needs some merge conflicts to be sorted out and addressing some review comments.

Actually, it's not difficult to finish the PR. I'm a volunteer contributor to Parse Server, and most of the time, I try to send PRs to parse-server when I discover an improvement. However, there is no magic, unfortunately. I work and dedicate my time to parse-server on features that are really used by my team. Now we are moving away from Mongo Serverless, and we do not use DocumentDB, so we are not "motivated" to ship this ASAP 😉. I keep my open-source time for other PRs/projects.

Here, the job is almost done. Since we are in an open-source environment, you can easily fork, copy the branch, and provide suggested changes by @mtrezza, and it's done. If it's your first contribution to parse-server, it might take some additional time, but contributing to parse-server is feasible. Even better, you will be able to easily contribute in the future 🚀.

I'll be happy to help you start contributing 💯. @b10f

mtrezza · 2023-09-26T17:18:18Z

I have also simplified the comments and removed some of the less important suggested changes (like naming), so it will hopefully be easier to get this PR finished.

b10f · 2023-09-29T17:14:07Z

@mtrezza #8767

mtrezza · 2023-11-14T00:49:54Z

Continued in #8805

feat: disableCaseInsensitivity and forceEmailAndUsernameToLowerCase

6bcd29c

Merge branch 'alpha' into moumouls/supportMongoServerless

7f46c6a

Moumouls requested a review from a team June 13, 2022 13:46

fix: test to mongo

4fead08

mtrezza changed the title ~~feat: disableCaseInsensitivity and forceEmailAndUsernameToLowerCase~~ feat: add options to disable case insensitivity and force email and username to lower case Jun 13, 2022

mtrezza requested changes Jun 13, 2022

View reviewed changes

mtrezza linked an issue Jun 13, 2022 that may be closed by this pull request

Parser Server not fully compatible with Mongo Atlas Serverless #7937

Closed

4 tasks

Moumouls added 2 commits June 26, 2022 13:37

Merge branch 'alpha' into moumouls/supportMongoServerless

495a4db

fix: ensure to lower case on string

a53fbba

dblythy suggested changes Jul 1, 2022

View reviewed changes

src/Controllers/DatabaseController.js Outdated Show resolved Hide resolved

src/Options/Definitions.js Outdated Show resolved Hide resolved

mtrezza and others added 3 commits July 2, 2022 11:35

Merge branch 'alpha' into moumouls/supportMongoServerless

d6bd27f

Update src/Controllers/DatabaseController.js

0080bca

Co-authored-by: dblythy <[email protected]>

Update src/Options/Definitions.js

15710ee

Co-authored-by: dblythy <[email protected]>

Moumouls mentioned this pull request Nov 14, 2022

Parse Server compatibility with Serverless environment #7938

Open

3 tasks

Merge branch 'alpha' into moumouls/supportMongoServerless

30ef7e7

parse-github-assistant bot changed the title ~~feat: add options to disable case insensitivity and force email and username to lower case~~ feat: Add options to disable case insensitivity and force email and username to lower case Nov 14, 2022

Merge branch 'alpha' into moumouls/supportMongoServerless

24ab89b

This was referenced Apr 8, 2023

feat: Skip indexes for Azure Cosmos DB for MongoDB #8494

Open

Disable case-insensitive index on _User #8492

Open

mtrezza requested a review from a team April 9, 2023 20:42

b10f mentioned this pull request Sep 29, 2023

feat: Add options to disable collation and transform email and username to lower case #8767

Closed

3 tasks

mattia1208 mentioned this pull request Nov 10, 2023

feat: Add compatibility for MongoDB Atlas Serverless and AWS Amazon DocumentDB with collation options enableCollationCaseComparison, convertEmailToLowercase, convertUsernameToLowercase #8805

Merged

3 tasks

mtrezza closed this Nov 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add options to disable case insensitivity and force email and username to lower case #8042

feat: Add options to disable case insensitivity and force email and username to lower case #8042

Moumouls commented Jun 13, 2022 •

edited

Loading

parse-github-assistant bot commented Jun 13, 2022 •

edited

Loading

codecov bot commented Jun 13, 2022 •

edited

Loading

mtrezza commented Jun 13, 2022 •

edited

Loading

Moumouls commented Jun 13, 2022

mtrezza left a comment

mtrezza Jun 13, 2022

Moumouls Jun 14, 2022

mtrezza Jun 13, 2022

mtrezza Jun 13, 2022

Moumouls Jun 14, 2022

mtrezza Sep 21, 2023

mtrezza commented Jun 13, 2022 •

edited

Loading

Moumouls commented Jun 14, 2022 •

edited

Loading

mtrezza commented Jun 14, 2022 •

edited

Loading

dblythy left a comment •

edited

Loading

Moumouls commented Jul 3, 2022

parse-github-assistant bot commented Nov 14, 2022

Moumouls commented Nov 14, 2022

mtrezza commented Apr 9, 2023

lilonpro commented May 19, 2023

mattia1208 commented Aug 18, 2023

mtrezza commented Aug 18, 2023 •

edited

Loading

Moumouls commented Aug 29, 2023

mattia1208 commented Sep 15, 2023

b10f commented Sep 21, 2023

mtrezza commented Sep 21, 2023 •

edited

Loading

b10f commented Sep 26, 2023

Moumouls commented Sep 26, 2023

b10f commented Sep 26, 2023 •

edited

Loading

Moumouls commented Sep 26, 2023

mtrezza commented Sep 26, 2023 •

edited

Loading

b10f commented Sep 29, 2023

mtrezza commented Nov 14, 2023

feat: Add options to disable case insensitivity and force email and username to lower case #8042

feat: Add options to disable case insensitivity and force email and username to lower case #8042

Conversation

Moumouls commented Jun 13, 2022 • edited Loading

New Pull Request Checklist

Issue Description

Approach

TODOs before merging

parse-github-assistant bot commented Jun 13, 2022 • edited Loading

Thanks for opening this pull request!

codecov bot commented Jun 13, 2022 • edited Loading

Codecov Report

mtrezza commented Jun 13, 2022 • edited Loading

Moumouls commented Jun 13, 2022

mtrezza left a comment

Choose a reason for hiding this comment

mtrezza Jun 13, 2022

Choose a reason for hiding this comment

Moumouls Jun 14, 2022

Choose a reason for hiding this comment

mtrezza Jun 13, 2022

Choose a reason for hiding this comment

mtrezza Jun 13, 2022

Choose a reason for hiding this comment

Moumouls Jun 14, 2022

Choose a reason for hiding this comment

mtrezza Sep 21, 2023

Choose a reason for hiding this comment

mtrezza commented Jun 13, 2022 • edited Loading

Moumouls commented Jun 14, 2022 • edited Loading

mtrezza commented Jun 14, 2022 • edited Loading

dblythy left a comment • edited Loading

Choose a reason for hiding this comment

Moumouls commented Jul 3, 2022

parse-github-assistant bot commented Nov 14, 2022

Moumouls commented Nov 14, 2022

mtrezza commented Apr 9, 2023

lilonpro commented May 19, 2023

mattia1208 commented Aug 18, 2023

mtrezza commented Aug 18, 2023 • edited Loading

Moumouls commented Aug 29, 2023

mattia1208 commented Sep 15, 2023

b10f commented Sep 21, 2023

mtrezza commented Sep 21, 2023 • edited Loading

b10f commented Sep 26, 2023

Moumouls commented Sep 26, 2023

b10f commented Sep 26, 2023 • edited Loading

Moumouls commented Sep 26, 2023

mtrezza commented Sep 26, 2023 • edited Loading

b10f commented Sep 29, 2023

mtrezza commented Nov 14, 2023

Moumouls commented Jun 13, 2022 •

edited

Loading

parse-github-assistant bot commented Jun 13, 2022 •

edited

Loading

codecov bot commented Jun 13, 2022 •

edited

Loading

mtrezza commented Jun 13, 2022 •

edited

Loading

mtrezza commented Jun 13, 2022 •

edited

Loading

Moumouls commented Jun 14, 2022 •

edited

Loading

mtrezza commented Jun 14, 2022 •

edited

Loading

dblythy left a comment •

edited

Loading

mtrezza commented Aug 18, 2023 •

edited

Loading

mtrezza commented Sep 21, 2023 •

edited

Loading

b10f commented Sep 26, 2023 •

edited

Loading

mtrezza commented Sep 26, 2023 •

edited

Loading