Leak of PB Socket proccess from pbc_pool_master

[shino]

Connection pool management uses two layer of poolboy pools.

In default settings, the relation of pool and thier size are as follows:

request_pool (128) -------+
                          |
                          +----> pbc_pool_master (133)
                          |
bucket_list_pool (5) -----+

UPDATE As descripbed in the comment below #985 (comment), this steps may not be essential.

Steps to reproduce (or emulate) the leak at pbc_pool_master.

(rcs-dev1@127.0.0.1)1> [{P, poolboy:status(P)} || P <- [request_pool, bucket_list_pool, pbc_pool_master]].
[{request_pool,{ready,128,0,0}},
 {bucket_list_pool,{ready,5,0,0}},
 {pbc_pool_master,{ready,133,0,0}}]
(rcs-dev1@127.0.0.1)2> f(), spawn(fun() ->
     {ok, RcPid} = riak_cs_riak_client:start_link([]),
     riak_cs_riak_client:manifest_pbc(RcPid),
     spawn_link(fun() -> exit(dummY)
  end) end).
<0.504.0>
(rcs-dev1@127.0.0.1)3> [{P, poolboy:status(P)} || P <- [request_pool, bucket_list_pool, pbc_pool_master]].
[{request_pool,{ready,128,0,0}},
 {bucket_list_pool,{ready,5,0,0}},
 {pbc_pool_master,{ready,132,0,1}}]   %% <==== LEAK!!

The process which calls riak_cs_riak_client:start_link/1 emulates riak_cs_gc_d. The one which callsexit(dummY)` emulates other GC
related process, for example riak_cs_put_fsm or riak_cs_delete_server.

[kuenishi]

Nice catch Shino!!!

[shino]

The steps in the description may not be essential.
Initially spawned process (A), which mimics gc daemon, may dies before another spawn_linked process (B) dies with dummY signal.
So the chain of links from (B) to RcPid is broken.

But, in actual case, gc daemon is long running process and does not exit normal.

[shino]

Inversely, pbc_pool_master leak pattern.

f(),
{ok, RcPid} = riak_cs_riak_client:checkout(),
{ok, Pbc} = riak_cs_riak_client:manifest_pbc(RcPid),
exit(Pbc, dummY).

[shino]

Branch to fix the above comment https://github.com/basho/riak_cs/compare/release/1.5...bugfix/pool-leak?expand=1

[shino]

Other (much more simpler) cases

(1) Just call /riak-cs/users with the admin user

$ s3curl.pl --id cs -- -s -x localhost:15018 http://riak-cs.s3.amazonaws.com/users

(2) Call list object of non-existent bucket by anonymous user (kudos @kuenishi )

$ curl http://127.0.0.1:15018/ping

(ping is not to ping the cs node, just a bucket name, a non-existent one)