Detailed storage stats

[shino]

This PR adds several classes for storage calculation.

• user: user accessible, this is the sum of Active and WritingMultipart.
• Active: active manifests
• ActiveInvisible: active but not user accessible
• WritingMultipart: writing Multipart
• WritingNew: writing non-MP, within leeway (maybe actively uploading)
• WritingOld: writing non-MP, out of leeway (probably orphan)
• PendingDeleteNew: pending_delete, within leeway (maybe being moved to GC bucket right now)
• PendingDeleteOld: pending_delete, out of leeway (probably failed to be moved)
• ScheduledDeleteNew: scheduled_delete, within leeway (normal, just staying as is)
• ScheduledDeleteOld: scheduled_delete, out of leeway (GC can collect but not catches up)

Each cass has three counters for object count (Objects),
bytes (Bytes) and blocks (Blocks). One exception is for user,
no prefix and only Objects, Bytes and Blocks.
The key in JSON/XML usage response looks like WritingMultipartBytes.

Sample JSON output: https://gist.github.com/shino/1be474a0f5b949f47c91

Future tasks:

• Calculate deleted buckets. Deleted buckets can hold non-active manifests and corresponding blocks reside on disks. This PR does not calculate stats for them.
• Cleanup lingering objects like old writing or pending_delete manifests.

[shino]

Ready for review 👀

[kuenishi]

Also addresses #1054 (RCS-107)

[kuenishi]

I'd rather prefer non-two-char-acronym like this:

• Active : user accessible
• ActiveInvisible : active but not user accessible
• WritingMultiPart : writing MP, items in this class are doubly counted with user
• WritingNew : writing non-MP, within leeway (maybe actively uploading)
• WritingOld : writing non-MP, out of leeway (probably orphan)
• PendingDeleteNew : pending_delete, within leeway (maybe being moved to GC bucket right now)
• PendingDeleteOld: pending_delete, out of leeway (probably failed to be moved)
• ScheduledDeleteNew : scheduled_delete, within leeway (normal, just staying as is)
• ScheduledDeleteOld : scheduled_delete, out of leeway (GC can collect but not catches up)

With all of them having suffix Bytes Counts and Blocks

[shino]

Nice naming!

I will change only one item Active: user accessible because we use active for another meaning, i.e. one of the states of manifests.

[kuenishi]

Let's use latest year for new files.

[kuenishi]

Just for code readability, I'd also prefer not using two-chars-acronym. Maybe codes get ugly?

[kuenishi]

16:26:04.514 [info] ./s3_conf.sh: 1: ./s3_conf.sh: uuidgen: not found

Fmm, accidentally I've found this is not documented...

[shino]

Where to write it? ceph_test README?

[shino]

Please give me your environment information and install command for the dep.

[kuenishi]

I think riak_test/README.md is suitable.

[kuenishi]

For mine, apt-get install uuid-runtime was required.

[kuenishi]

Mine was

$ uname -a
Linux nausicaa 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

[shino]

I hope this became ready for review again.

[kuenishi]

So, user = active + writing_multipart ? Or MP is just for number of parts?

[shino]

So, user = active + writing_multipart ?

Yes

[shino]

pushed

[kuenishi]

Looks nice. Will run r_t and eventually +1.

[kuenishi]

@borshop merge

[kuenishi]

Even it's after merge, I tested, data stored in storage bucket has some strange form:

$ curl -s http://localhost:8098/buckets/moss.storage/keys/20150414T000000Z.admin-key  | jq '.'                                                
{
  "StartTime": "20150414T004958Z",
  "EndTime": "20150414T005000Z",
  "test": {
    "WrOldCt": 0,
    "WrOldBy": 0,
    "WrOldBl": 0,
    "WrNewCt": 0,
    "WrNewBy": 0,
    "WrNewBl": 0,
    "WrMpCt": 14,
    "WrMpBy": 220200960,
    "WrMpBl": 210,
    "Ct": 32436,
    "By": 1438887763,
    "Bl": 33156,
    "SdOldCt": 0,
    "SdOldBy": 0,
    "SdOldBl": 0,
    "SdNewCt": 71,
    "SdNewBy": 838860822,
    "SdNewBl": 811,
    "PdOldCt": 0,
    "PdOldBy": 0,
    "PdOldBl": 0,
    "PdNewCt": 0,
    "PdNewBy": 0,
    "PdNewBl": 0,
    "AcDeCt": 0,
    "AcDeBy": 0,
    "AcDeBl": 0
  }
}

[kuenishi]

Maybe this might be simpler?

bucket_summary_fold(Sums) ->
   lists:foldl(fun(Sum, Acc) -> orddict:merge(fun(K, V1, V2) -> V1+V2 end, Sum, Acc) end,
                     orddict:new(), Sums).

[kuenishi]

Make clean and stage again made it working. Something was wrong...