Parse Server performance loss #7036
Comments
|
Maybe related to #6543. |
|
Thanks for reporting. To eliminate the possibility that this is caused by any custom configuration, can you please redo the tests with:
Then it would be interesting if you could share your Parse Server configuration and a full description of the Parse Objects that you queried. You could also try a different or external benchmarking tool to get a second opinion. |
|
Hey @mtrezza Thank you for taking time on this. I can do what you said. But I want to say a thing. I was doing tests. I changed my code. Previously my cloud code was like this: And I changed it to this: Both cloud code returns same thing(well, second cloud code returns status code and http headers etc as extra. but also returns query result so basicly same thing). But second code Doesnt decrease perormance. It was stable. In fact I get 30% performance increase with second code. So With this way performace doesnt decrease and its stable. What do you think about this? |
|
Very interesting indeed. Can you share your Parse Server configuration? |
|
@mtrezza Of course |
|
|
|
@davimacedo hey. Unfortunatly setting directAccess to false didnt work. Here is my test history with my original code(directAccess is false): And This is the test results with my second code. DirectAccess is false: |
|
I have one more questions. I see that in your second code you are not passing the master key. Is this query really working and returning the same results of the first code (because in the first one you are using the master key)? |
|
Using masterkey is habit. I used to. I disable all class level permissions for every class. So I put master key in every query. but for testing porpouses, I enabled public permissions for that class. Here is what second code returns: |
I see that there are still some options in your configuration, that are not the default Parse Server configuration. As I mentioned earlier:
To clarify, by that I mean:
|
Yes. Thats my current setup. I tried with fresh install parse server(cloned parse-server-example repo). But I used my existing database.And Problem still exists. Even on fresh install parse server, performace decreases. About fresh mongoDb, I dont make the connection why this can be a problem. I mean when I restart parse server ıts give full performance or when we do query |
|
@mtrezza Ok. You were right. I made a fresh install MongoDb database. Creates class named "Test". I added one integer field called "int".(It has other default fields like id and createdAt etc..) Set value to 12. Now In my database I have one class and one object. I do query without any parameter like this: Performace doesnt decrease. Only diference this object and my other objects is pointer fields and one string field. I will add a string field to object first. And if performace was stable, I will Create another class and create a pointer field on my Test class. |
|
Great investigative process, we are getting somewhere. I see 4 dimensions you can experiment with:
|
|
@mtrezza Ok. I replicated all fields. Created pointer and string fields. Performace didnt decrease. Then I connect my previous parse server to fresh mongodb database. Performace decrease. So I think problem is related to parse server configuration |
|
It would be interesting now to find out which configuration specifically is related to the performance decrease. Let us know when you found the culprit. |
|
@mtrezza I will stop investigating for now. I have finals in mid january(18th) I have to study my exams. And I will refractor my currect cloud code. I will change parse queries to http requests. When my exams finished I can start re investigate in february. |
|
If you could just do one last test and take the current deployment that works fine but set the full Parse Server config that you had issues with, then we could get some important insight. I'm afraid that in Feb things will not look the same and we'll loose track of this, so if you could it would be great, if not then we'll pick this up in Feb. |
|
@mtrezza I'm afraid this cloud take longer than we tought. I said ıts related to parse config. But I made another test yesterday(my first test with fresh parse server and existing database). But performance was still decreases. see:
I dont know why I skipped this. ıt was midnight and I was sleepy. So This might be or might not be related to parse config. Yes I can do few more tests. But This looks like it will take longer than we think |
|
As I understand it now: - 1. old parse server + old database = performance decrease
- 2. new parse server + old database = performance decrease
+ 3. new parse server + new database = no performance decrease
# 4. old parse server + new database = ?
# 5. new parse server with old configuration + new database = ?I don't want to ask for too much, but the test case 5 is what would be interesting to see. |
|
@mtrezza I will format my server. And re install both parse-server and mongodb. And will test all 5 scenario. |
|
Let's just make sure we don't loose the old DB and old Parse Server, so that we can reproduce the issue. |
|
@mtrezza Ok. I made some tests. I found the problem. Its not related to parse config. Its related to what I pass into query constructor. For example: This decreases performance. |
|
Ah right, well done. I think it's not supported to pass a Parse Object. Interesting that that worked at all. Did you see this anywhere in the docs as an example? |
|
@mtrezza here is an example: https://docs.parseplatform.org/js/guide/#basic-queries Also JS sdk also check if ıts a stiring or parse object: https://github.com/parse-community/Parse-SDK-JS/blob/master/src/ParseQuery.js#L251 |
|
Very interesting and well researched from your side - now that we circled in the issue, the next step would be to look into the Parse Server / JS SDK code. It just a simple property setting, so it's not obvious at this point why this would cause a performance impact:
I also don't see the object being retained beyond the method call. It could be an issue with If the code above also decreases performance, then the issue may not be related to |
I'm suspecting that too. Edit: Ok. I will do that |
|
@mtrezza |
|
Glad to hear that, we are getting closer to solving this. Can you try again without extending the Parse Object, but just creating a new one? Maybe can you also remove the query and still measure a performance decrease? Also, you may want to try to remove the |
Can you share your |
This doesnt decrease performance This also doesnt decrease performance Here is my lua script: |
|
I have been trying to recreate the issue of decreasing performance using The results are: Cloud Code functionsThe Cloud Code function wrk log-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 164.05ms 60.53ms 713.18ms 92.50%
Req/Sec 63.53 18.24 101.00 77.62%
1843 requests in 30.09s, 3.54MB read
Requests/sec: 61.25
Transfer/sec: 120.53KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 191.24ms 152.52ms 1.82s 96.86%
Req/Sec 56.15 22.58 100.00 64.64%
1521 requests in 30.02s, 2.92MB read
Socket errors: connect 0, read 0, write 0, timeout 7
Requests/sec: 50.66
Transfer/sec: 99.69KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 179.16ms 56.93ms 880.74ms 91.60%
Req/Sec 56.54 16.22 99.00 69.47%
1639 requests in 30.04s, 3.15MB read
Requests/sec: 54.56
Transfer/sec: 107.37KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 220.54ms 146.75ms 1.78s 94.95%
Req/Sec 48.26 18.17 90.00 64.89%
1393 requests in 30.07s, 2.68MB read
Socket errors: connect 0, read 0, write 0, timeout 4
Requests/sec: 46.33
Transfer/sec: 91.16KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 264.34ms 209.52ms 1.97s 93.56%
Req/Sec 43.93 15.84 89.00 67.29%
1227 requests in 30.05s, 2.36MB read
Socket errors: connect 0, read 0, write 0, timeout 2
Requests/sec: 40.83
Transfer/sec: 80.34KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 248.77ms 52.89ms 473.62ms 72.47%
Req/Sec 40.08 11.98 70.00 79.79%
1188 requests in 30.05s, 2.28MB read
Requests/sec: 39.53
Transfer/sec: 77.78KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 310.53ms 170.92ms 1.99s 94.99%
Req/Sec 32.92 12.34 68.00 61.15%
950 requests in 30.05s, 1.83MB read
Socket errors: connect 0, read 0, write 0, timeout 5
Requests/sec: 31.62
Transfer/sec: 62.21KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 315.16ms 70.92ms 846.97ms 77.49%
Req/Sec 31.61 11.44 70.00 62.89%
938 requests in 30.04s, 1.80MB read
Requests/sec: 31.22
Transfer/sec: 61.44KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 353.37ms 141.22ms 1.80s 88.39%
Req/Sec 28.22 11.09 59.00 71.64%
810 requests in 30.05s, 1.56MB read
Socket errors: connect 0, read 0, write 0, timeout 7
Requests/sec: 26.96
Transfer/sec: 53.05KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 389.79ms 129.77ms 1.65s 88.36%
Req/Sec 25.68 9.72 60.00 70.22%
747 requests in 30.08s, 1.44MB read
Socket errors: connect 0, read 0, write 0, timeout 2
Requests/sec: 24.83
Transfer/sec: 48.86KB
-----------------------------------------------------However, after a while the performance seems to stabilize in the lower range: wrk log-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.50s 233.25ms 1.98s 66.67%
Req/Sec 7.10 4.55 29.00 92.41%
191 requests in 30.11s, 375.84KB read
Socket errors: connect 0, read 0, write 0, timeout 5
Requests/sec: 6.34
Transfer/sec: 12.48KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.48s 238.73ms 1.95s 72.82%
Req/Sec 6.88 3.82 20.00 89.02%
195 requests in 30.01s, 383.72KB read
Requests/sec: 6.50
Transfer/sec: 12.79KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.42s 177.94ms 1.85s 85.85%
Req/Sec 7.26 3.88 20.00 91.53%
205 requests in 30.02s, 403.39KB read
Requests/sec: 6.83
Transfer/sec: 13.44KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.51s 255.28ms 1.97s 69.63%
Req/Sec 6.62 3.76 20.00 94.84%
191 requests in 30.02s, 375.84KB read
Requests/sec: 6.36
Transfer/sec: 12.52KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.58s 244.29ms 1.99s 70.06%
Req/Sec 6.23 3.40 20.00 90.00%
181 requests in 30.09s, 356.17KB read
Socket errors: connect 0, read 0, write 0, timeout 4
Requests/sec: 6.02
Transfer/sec: 11.84KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.44s 191.72ms 1.95s 81.09%
Req/Sec 7.30 4.42 20.00 91.28%
201 requests in 30.05s, 395.52KB read
Requests/sec: 6.69
Transfer/sec: 13.16KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.61s 244.69ms 1.92s 68.18%
Req/Sec 5.85 3.22 20.00 91.56%
173 requests in 30.04s, 340.42KB read
Socket errors: connect 0, read 0, write 0, timeout 19
Requests/sec: 5.76
Transfer/sec: 11.33KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.41s 195.47ms 1.88s 76.70%
Req/Sec 7.33 4.07 20.00 90.61%
206 requests in 30.08s, 405.36KB read
Requests/sec: 6.85
Transfer/sec: 13.48KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.48s 216.69ms 1.92s 73.77%
Req/Sec 6.92 4.32 20.00 91.93%
191 requests in 30.08s, 375.84KB read
Socket errors: connect 0, read 0, write 0, timeout 8
Requests/sec: 6.35
Transfer/sec: 12.49KB
-----------------------------------------------------
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.39s 241.53ms 1.90s 78.26%
Req/Sec 7.51 4.27 20.00 87.57%
207 requests in 30.08s, 407.33KB read
Requests/sec: 6.88
Transfer/sec: 13.54KB
----------------------------------------------------- |
|
@mtrezza Did you look for And yeah I can confirm it stabilize on 5-6 req/sec. I've made a test a month ago. It was set to one hour. And gave me 5 req/sec. It was on 4 core machine. And Today I refractored my cloud code. Changed parse objects to className strings. Performance was stable. So changing parseobject to string definetly solves the issue. Edit: I think easiest workaround for now would be disabling passing parse object to query constructor. İf user passed a parseobject to query constructor, Throw an error. But this requires developer to refreactor the code. But I've made it in 10 minutes on 8k lines of code. Thanks to Ctrl+F |
|
I did not reach a conclusive result on whether The decreasing performance and eventual stabilization could have to do with the NodeJS heap allocation or garbage collection algorithm. Maybe a NodeJS expert could give more insight here. |
|
Here the JS SDK code /**
* @param {(String|Parse.Object)} objectClass An instance of a subclass of Parse.Object, or a Parse className string.
*/
constructor(objectClass: string | ParseObject) {
if (typeof objectClass === 'string') {
if (objectClass === 'User' && CoreManager.get('PERFORM_USER_REWRITE')) {
this.className = '_User';
} else {
// No reference
this.className = objectClass;
}
} else if (objectClass instanceof ParseObject) {
// Can lead to some JS ref issue may be
this.className = objectClass.className;
} else if (typeof objectClass === 'function') {
if (typeof objectClass.className === 'string') {
this.className = objectClass.className;
} else {
var obj = new objectClass();
this.className = obj.className;
}
} else {
throw new TypeError(
'A ParseQuery must be constructed with a ParseObject or class name.'
);
}it would be interesting if we try to break some JS object reference. |
I would be surprised if string interpolation would make any difference because However, I think we need some more analysis to be sure that there is actually a problem that needs to be fixed and how we would measure any improvements we attempt to make. I'll do some more tests. |
|
@mtrezza me too ahaha, i think here it could be interesting to run a |
|
I did the tests above while profiling, couldn't interpret any causes, just saw the execution times rising, but for all JS ops, hence I assumed that it may just be a natural V8 characteristic. But feel free to give it a go. |
|
Hey guys. I need to mention that, Performance only decreases when query returns a parse objects. When you do test with count query, performance is stable. Maybe you should try with count method. So This is not only related to extend method(I guess) See this: https://community.parseplatform.org/t/parse-server-performance-loss-after-time/1070/10?u=uzaysan Can you please test this code: This code was stable for me |
|
Maybe related: #7065 describes neg. performance impact on queries with pointers. |
|
Maybe related: #7080 describes slow ops on WT engine |
|
Maybe related: #6636 (comment) describes possible memory leak with |
|
Maybe related: #6405 describes slow queries with certain Parse Server / Node versions |
|
Closing via #7214 |
|
@dplewis I will try it when I have a free time. |
|
Just would like to know if this issue has been resolved? Thanks. |
New Issue Checklist
Issue Description
When I do performance test with parse server, performance decreases. And When I say decreases, I don't mean it decrease a little then stop. I mean It always decrease and doesn't stop decreasing. But This only happens when we use parse query in cloud code and query (in cloud code) returns parse object(s). But queries that doesn't return parse object(count query) or queries that should return array of parse objects but returns empty array(for example collection is empty), performance doesn't decrease. They are fine. Problem only happens when query return parse object. You can see more information on forum thread that I have created
Steps to reproduce
Create a cloud code and put a simple query in it(query must return parse object(s)). Then do countinues benchmark test against this cloud code.
I used wrk library to do benchmark tests. This is the code I used when I do benchmarking:
wrk -t1 -c400 -d30s -H "Content-Type: application/json" -H "X-Parse-Application-Id: YOUR_APP_ID" -s post.lua http://parse_server_ip:1337/parse/functions/codeWithQueryHere is an example cloud code:
Actual Outcome
Requests per second decreases. You can see my benchmark history on this message.
And You can see information on that thread.
Expected Outcome
Performance shouldn't decrease. Even if it does, It should be stable after some time.
Environment
Server
4.3.0Ubuntu 18.04Remote hostDatabase
MongoDB4.4.1RemoteClient
Cloud CodeCloud CodeLogs
The text was updated successfully, but these errors were encountered: