feat: fallback cache and lock driver

[ibrunotome]

Is your feature request related to a problem? Please describe.

Recently, I got almost 100 concurrent requests to debit balance from one wallet via api, and I got a redis exception "could not acquire lock"

Describe the solution you'd like

Read data directly from database when could not acquire lock (and update the state in redis in the next request)

Describe alternatives you've considered

Tried other cache drivers, but the only centralized is redis, array driver for example would give two different results if you access balance in a http container vs a container for queues for example.

P.S: I would put a warning about that non centralized cache drivers in the docs.

[rez1dent3]

Hello.
This is not a package issue, but an infrastructure issue. Redis did its job, you didn't get a race condition.
Consider a cluster solution, for example
https://redis.io/docs/manual/sentinel/

[rez1dent3]

And here is the framework layer.
laravel/framework#18850
https://laracasts.com/discuss/channels/laravel/redis-sentinel-in-laravel-nodesetname-back-off-strategym-mastersetc-config
@ibrunotome FYI^

[ibrunotome]

Hi @rez1dent3, thank you for the quick support.

But, I didn't said it's a package issue, I made a feature request.

Would it possible to make a change to achieve this behavior?

[rez1dent3]

Would it possible to make a change to achieve this behavior?

No. This is the infrastructure layer. The code shouldn't work in that case.

Your cache should be behind a balancer (for example, haproxy) and monitor it. In case of problems, change the master.

I would consider sentinel if I were you.

[rez1dent3]

For example, if you solve this with code, then you need to maintain data consistency in the failback storage. And this is already overhead.

[rez1dent3]

If you are going to solve a problem with code, you will most likely have a race condition.

Sample packages can be found in packagist. Someone solved a similar problem: https://github.com/mathieu-bour/laravel-cache-fallback

PS, I highly recommend using cluster solutions.

[ibrunotome]

Thank you for all the links/recommendations.

[rez1dent3]

@ibrunotome You are welcome.
If you want to pull the balance directly, then interfaces will help you:

laravel-wallet/src/Services/BookkeeperServiceInterface.php

Line 11 in c6b8a22

interface BookkeeperServiceInterface

laravel-wallet/src/Services/RegulatorServiceInterface.php

Line 9 in c6b8a22

interface RegulatorServiceInterface

Don't overuse interfaces too much, you need to have a deep understanding of working with the state. Read #412

[rez1dent3]

laravel-wallet/tests/Units/Domain/StateTest.php

Lines 80 to 98 in 7a30789

	$wallet->deposit(1000);
	self::assertSame(0, (int) $regulator->diff($wallet));
	self::assertSame(1000, (int) $regulator->amount($wallet));
	self::assertSame(1000, (int) $bookkeeper->amount($wallet));
	self::assertSame(1000, $wallet->balanceInt);
	app(DatabaseServiceInterface::class)->transaction(function () use ($wallet, $regulator, $bookkeeper) {
	$wallet->deposit(10000);
	self::assertSame(10000, (int) $regulator->diff($wallet));
	self::assertSame(11000, (int) $regulator->amount($wallet));
	self::assertSame(1000, (int) $bookkeeper->amount($wallet));
	return false; // rollback
	});
	self::assertSame(0, (int) $regulator->diff($wallet));
	self::assertSame(1000, (int) $regulator->amount($wallet));
	self::assertSame(1000, (int) $bookkeeper->amount($wallet));
	self::assertSame(1000, $wallet->balanceInt);

[rez1dent3]

@ibrunotome The last thing I remembered, you can pull the balance from the database directly. It is clear that there may be data lagging a bit (race condition).

laravel-wallet/src/Models/Wallet.php

Line 122 in c4e5d69

public function getOriginalBalanceAttribute(): string

[ibrunotome]

@rez1dent3 sometimes I'm getting exceptions with no messages, have you seem that before?

the code is a transaction with 1 to 3 possible transactions, ex:

app(DatabaseServiceInterface::class)->transaction(function () use ($workContract) {
    $this->transaction($workContract); // transaction that always occurs
    $this->distributeAffiliateEarnings($workContract); // transaction that conditionally occurs
    $this->distributeBonus($workContract);  // transaction that conditionally occurs
});

[rez1dent3]

@ibrunotome
I put the error in $previous, you can get it from there.
At first glance, this is some kind of error in the framework or your code. I ran through the code and I see that all of my exceptions have a message.

Example:

try {
    app(DatabaseServiceInterface::class)->transaction(function () use ($workContract) {
        $this->transaction($workContract); // transaction that always occurs
        $this->distributeAffiliateEarnings($workContract); // transaction that conditionally occurs
        $this->distributeBonus($workContract);  // transaction that conditionally occurs
    });
} catch (\Throwable $t) {
    dd($throwable->getPrevious());
}

[ibrunotome]

It was a LockTimeoutException, who doesn't has messages. (so even with ->getPrevious() there wasn't messages)

I switched the lock driver to database, by now these are my settings:

Facing issues in production that I can't simulate in local or testing environment, maybe because i'm using redis in https://cloud.google.com/memorystore, and, in all other environments a simple redis:7.0.2-alpine container in the same network.

[rez1dent3]

There are two options here:

1. Somewhere in the transaction you block and wait (until you get a lock timeout);
2. Somewhere in parallel, methods are twitching with the preservation of atomicity and blocking the transaction, so the current one cannot be executed;

here you need to profile.