Collect and store logs with full diagnostics for Benchmarking

[tangerinous]

Design aproach to run Benchmark tool on VMs

• Backgraound.
• Proposed Solution.

Backgraund

The goal of this feature is to make the process of collecting diagnostic data for requests exceeds specified by input parameter threshold and store them in storage

Proposed Solution

• Extend the deployment of the ARM template to create an Azure Blob Storage account resource.

• Include an input parameter to specify the latency threshold.

• Capture diagnostics data for requests that exceed the specified threshold.

  • Collect the data in a file on the VM disk.
  • Add a background task to check the size of the diagnostics data file every 5 seconds.
  • If the file size exceeds 100MB, rename the current file and start collecting logs in a new file.

• Upload all the files with diagnostics data to the Azure Blob Storage in the VM's base directory.

• Validate if any resources have already been created.

  • If they have, cancel the deployment.

• Add a postfix to the VM name.

• Run for 30 minutes and monitor live metrics.

• Fix the bug with VM file upload.

sequenceDiagram
    participant A as Arm Template
    participant B as Azure VM
    participant C as Benchmark Tool
    participant D as CosmosDB
    participant E as Blob Storage

    A->>E: Create Blob Storage Account
    A->>B: Create VM
    B->>C: Launch Benchmark Tool
        
        C->>D: Request
        activate D
        
        D->>C: Response 
        deactivate D
        C->> B: Add diagnostincs data to file <br> if latence above threshold
        activate C
        
        par Every 5 sec
            C->>B: Check file size < 100mb
            Note right of B: if file size too big, move the current file <br> and start coolect data in a new one
        end
        deactivate C
        C->>C: Benchmark completed
        C->>E: Export files with captured diagnostics to Blob Storage

Loading