[ET-VK] Allocate memory for weight and activation tensors lazily (#13501)

Summary:
* Allocate memory for weight tensors right before the prepacking shader is dispatched, rather than while building the graph
* Move allocation of shared objects (i.e. memory for intermediate tensors) to occur after prepacking

## Motivation

Prevent screen blackout (Llama 3.2 1B) / device crash (Llama 3.2 3B) when running Llama 3.2 models on Samsung Galaxy S24. This behaviour is related to high peak memory usage when loading the model.

## Full Context

During model loading, Vulkan delegate needs to store 3 copies of constant data in memory at various points:

* source data obtained from loading the model
* staging buffer
* GPU texture/buffer

The general rationale of this change is to allocate memory for each copy only when necessary to minimize the "overlap" when all 3 exist at once.

### Current Order of operations

Legend:
* `W` represents total weight nbytes
* `w` represents weight nbytes for one tensor
* `A` represents total activations nbytes
* `M` represents approximation of total memory footprint

First, model file is loaded

Then, when building compute graph, for each weight tensor:
1. Weight data is loaded from NamedDataMap (`M = W`)
2. GPU texture/buffer for weight is initialized + memory allocated (`M = 2W`)
3. After building the graph, `graph->prepare()` is called which currently allocates memory for the activation tensors as well (`M = 2W + A`)

Then, during the prepacking stage for each weight tensor, each weight tensor is copied individually:
1. Staging buffer initialized (`M = 2W + A + w`)
2. Copy CPU weight data to staging + CPU Weight data is freed (`M = 2W + A`)
3. Compute shader dispatch to copy staging to GPU texture/buffer + free staging buffer (`M = 2W + A - w`)

The peak usage in mainline will be `M = 2W + A + w`

### Revised order of operations

This change revises the order of operations:

1. Weight data is loaded from NamedDataMap (`M = W`)
2. GPU texture/buffer for weight is initialized, but **memory is not allocated** (`M = W`)

Then, during the prepacking stage for each weight tensor, each weight tensor is copied individually:
1. Staging buffer initialized (`M = W + w`)
2. **Memory allocated for GPU texture/buffer** (`M = W + 2w`)
3. Copy CPU weight data to staging + CPU Weight data is freed (`M = W + w`)
4. Compute shader dispatch to copy staging to GPU texture/buffer + free staging buffer (`M = W`)

**Then, after all prepacking operations complete, only then is Activation memory allocated** (`M = W + A`)

Under this scheme, peak memory is reduced to `M = W + A` (or alternatively `M = W + 2w` if `2w > A`) which is (or at least very close to) the theoretical minimum.

Test Plan:
## Logging Memory Usage

Using

```
uint64_t getVmRssInKB() {
  std::ifstream statusFile("/proc/self/status");
  std::string l, num;
  while (std::getline(statusFile, l)) {
    if (l.substr(0, 5) == "VmRSS") {
      size_t pos = l.find_first_of("0123456789");
      num = l.substr(pos);
      break;
    }
  }
  uint64_t vmRssInKB = std::stoi(num);
  return vmRssInKB;
}

uint64_t getVmaStatsInKB() {
  auto stats =
      vkcompute::api::context()->adapter_ptr()->vma().get_memory_statistics();
  uint64_t vmaBlockInKB = stats.total.statistics.blockBytes >> 10;
  return vmaBlockInKB;
}
```

to log memory footprint at various points of inference when running the llama_runner binary with Llama 3.2 1B, we can compare the memory footprint with and without these changes.

With changes: P1908051860 (Meta only)

```
Memory usage before model compilation: 1115760 KB (VmRSS), 0 KB (VMA)
Memory usage after graph building: 1924832 KB (VmRSS), 17920 KB (VMA)
Memory usage after graph preparation: 1935312 KB (VmRSS), 17920 KB (VMA)
Memory usage prepack start: 1935312 KB, VMA Block: 17920 KB
Memory usage after prepack operations: 1372376 KB (VmRSS), 2330528 KB (VMA)

Memory usage before execute: 1372804 KB (VmRSS), 2330528 KB (VMA)
Memory usage at end of execute: 1376916 KB (VmRSS), 2330528 KB (VMA)
```

WIthout changes: P1908054759 (Meta only)

```
Memory usage before model compilation: 1114784 KB (VmRSS), 0 KB (VMA)
Memory usage after graph building: 1924432 KB (VmRSS), 962464 KB (VMA)
Memory usage after graph preparation: 1922916 KB (VmRSS), 2326432 KB (VMA)
Memory usage prepack start: 1922916 KB, VMA Block: 2326432 KB
Memory usage after prepack operations: 1359180 KB (VmRSS), 2330528 KB (VMA)

Memory usage before execute: 1359492 KB (VmRSS), 2330528 KB (VMA)
Memory usage at end of execute: 1363636 KB (VmRSS), 2330528 KB (VMA)
```

It is evident how peak memory can be reduced with these changes, as VMA footprint gradually increases while loading the model while VmRss gradually decreases. Without these changes, VMA footprint will reach its peak after initializing the graph.

Visually, it can also be verified that Samsung Galaxy S24's screen no longer blacks out while loading the model.

Differential Revision: [D80460033](https://our.internmc.facebook.com/intern/diff/D80460033)

[ghstack-poisoned]

Author: pytorchbot

backends/vulkan/runtime/api/containers/Tensor.cpp

@@ -897,6 +897,16 @@ VkMemoryRequirements vTensor::get_memory_requirements() const {
   return {};
 }
 
+bool vTensor::memory_is_bound() const {
+  switch (storage_type()) {
+    case utils::kBuffer:
+      return storage_->buffer_.has_memory();
+    case utils::kTexture2D:
+    case utils::kTexture3D:
+      return storage_->image_.has_memory();
+  }
+}
+
 void vTensor::bind_allocation(const vkapi::Allocation& allocation) {
   switch (storage_type()) {
     case utils::kBuffer:
@@ -909,6 +919,18 @@ void vTensor::bind_allocation(const vkapi::Allocation& allocation) {
   }
 }
 
+void vTensor::acquire_allocation(vkapi::Allocation&& allocation) {
+  switch (storage_type()) {
+    case utils::kBuffer:
+      storage_->buffer_.acquire_allocation(std::move(allocation));
+      break;
+    case utils::kTexture2D:
+    case utils::kTexture3D:
+      storage_->image_.acquire_allocation(std::move(allocation));
+      break;
+  }
+}
+
 void vTensor::update_metadata() {
   numel_ = utils::multiply_integers(sizes_);
   strides_ = calculate_strides(sizes_, dim_order_);

backends/vulkan/runtime/api/containers/Tensor.h

@@ -560,6 +560,12 @@ class vTensor final {
    */
   VmaAllocationCreateInfo get_allocation_create_info() const;
 
+  /*
+   * Checks if the tensor's underlying buffer or image resource is bound to a
+   * memory allocation.
+   */
+  bool memory_is_bound() const;
+
   /*
    * Return the VkMemoryRequirements of the underlying resource
    */
@@ -570,6 +576,11 @@ class vTensor final {
    */
   void bind_allocation(const vkapi::Allocation& allocation);
 
+  /*
+   * Binds and acquires a rvalue memory allocation
+   */
+  void acquire_allocation(vkapi::Allocation&& allocation);
+
  private:
   /*
    * Assuming sizes, dim order, or axis mapping was modified, recompute all

backends/vulkan/runtime/graph/ComputeGraph.cpp

@@ -356,8 +356,6 @@ ValueRef ComputeGraph::add_tensor(
     const utils::GPUMemoryLayout memory_layout,
     const int64_t shared_object_idx,
     const utils::AxisMapLayout axis_map_layout) {
-  bool allocate_memory = shared_object_idx < 0;
-
   ValueRef idx(static_cast<int>(values_.size()));
   check_no_active_value_ptrs();
   values_.emplace_back(api::vTensor(
@@ -366,10 +364,10 @@ ValueRef ComputeGraph::add_tensor(
       dtype,
       storage_type,
       memory_layout,
-      allocate_memory,
+      false,
       axis_map_layout));
 
-  if (!allocate_memory) {
+  if (shared_object_idx >= 0) {
     get_shared_object(shared_object_idx).add_user(this, idx);
   }
   return idx;
@@ -626,6 +624,17 @@ SharedObject& ComputeGraph::get_shared_object(const int64_t idx) {
   return shared_objects_.at(idx);
 }
 
+void ComputeGraph::create_dedicated_allocation_for(const ValueRef idx) {
+  vTensorPtr tensor = get_tensor(idx);
+  if (!tensor->memory_is_bound()) {
+    VmaAllocationCreateInfo alloc_create_info =
+        context()->adapter_ptr()->vma().gpuonly_resource_create_info();
+    tensor->acquire_allocation(
+        context()->adapter_ptr()->vma().create_allocation(
+            tensor->get_memory_requirements(), alloc_create_info));
+  }
+}
+
 void ComputeGraph::update_descriptor_counts(
     const vkapi::ShaderInfo& shader_info,
     bool execute) {
@@ -873,6 +882,20 @@ void ComputeGraph::prepare_pipelines() {
       vkapi::ComputePipelineCache::Hasher>();
 }
 
+void ComputeGraph::prepare_pipelines() {
+  for (std::unique_ptr<PrepackNode>& node : prepack_nodes_) {
+    node->prepare_pipelines(this);
+  }
+  for (std::unique_ptr<ExecuteNode>& node : execute_nodes_) {
+    node->prepare_pipelines(this);
+  }
+  context_->pipeline_cache().create_pipelines(pipeline_descriptors_);
+
+  pipeline_descriptors_ = std::unordered_set<
+      vkapi::ComputePipelineCache::Key,
+      vkapi::ComputePipelineCache::Hasher>();
+}
+
 void ComputeGraph::submit_current_cmd(const bool final_use) {
   context_->submit_cmd_to_gpu(VK_NULL_HANDLE, final_use);
 }
@@ -952,6 +975,18 @@ void ComputeGraph::prepack() {
   submit_current_cmd_and_wait(/*final_use=*/true);
   context_->flush();
   staging_nbytes_in_cmd_ = 0;
+
+  // Initialize allocations for intermediate tensors
+  for (SharedObject& shared_object : shared_objects_) {
+    shared_object.allocate(this);
+    shared_object.bind_users(this);
+  }
+  // Make sure all remaining tensors have allocations
+  for (int i = 0; i < values_.size(); i++) {
+    if (values_.at(i).isTensor()) {
+      create_dedicated_allocation_for(i);
+    }
+  }
 }
 
 void ComputeGraph::execute() {

backends/vulkan/runtime/graph/ComputeGraph.h

@@ -827,6 +827,13 @@ class ComputeGraph final {
 
   SharedObject& get_shared_object(const int64_t idx);
 
+  /*
+   * Creates a dedicated memory allocation for a vTensor value, and have the
+   * tensor acquire the allocation object. If the tensor is already bound to a
+   * memory allocation, this function will be a no-op.
+   */
+  void create_dedicated_allocation_for(const ValueRef idx);
+
   //
   // Graph Preparation
   //

backends/vulkan/runtime/graph/ops/PrepackNode.cpp

@@ -97,6 +97,10 @@ void PrepackNode::encode(ComputeGraph* graph) {
   }
 
   {
+    // If the vTensor is not yet bound to a memory allocation, create a new one
+    // and aquire it.
+    graph->create_dedicated_allocation_for(packed_);
+
     vkapi::PipelineBarrier pipeline_barrier{};
     vkapi::DescriptorSet descriptor_set = context->get_descriptor_set(
         shader_, local_workgroup_size_, spec_vars_, push_constants_offset);

backends/vulkan/runtime/vk_api/memory/Buffer.cpp

@@ -20,6 +20,7 @@ VulkanBuffer::VulkanBuffer()
       allocator_(VK_NULL_HANDLE),
       memory_{},
       owns_memory_(false),
+      memory_bundled_(false),
       is_copy_(false),
       handle_(VK_NULL_HANDLE) {}
 
@@ -33,6 +34,7 @@ VulkanBuffer::VulkanBuffer(
       allocator_(vma_allocator),
       memory_{},
       owns_memory_(allocate_memory),
+      memory_bundled_(allocate_memory),
       is_copy_(false),
       handle_(VK_NULL_HANDLE) {
   // If the buffer size is 0, allocate a buffer with a size of 1 byte. This is
@@ -77,6 +79,7 @@ VulkanBuffer::VulkanBuffer(
       allocator_(other.allocator_),
       memory_(other.memory_),
       owns_memory_(false),
+      memory_bundled_(false),
       is_copy_(true),
       handle_(other.handle_) {
   // TODO: set the offset and range appropriately
@@ -91,6 +94,7 @@ VulkanBuffer::VulkanBuffer(VulkanBuffer&& other) noexcept
       allocator_(other.allocator_),
       memory_(std::move(other.memory_)),
       owns_memory_(other.owns_memory_),
+      memory_bundled_(other.memory_bundled_),
       is_copy_(other.is_copy_),
       handle_(other.handle_) {
   other.handle_ = VK_NULL_HANDLE;
@@ -99,16 +103,19 @@ VulkanBuffer::VulkanBuffer(VulkanBuffer&& other) noexcept
 VulkanBuffer& VulkanBuffer::operator=(VulkanBuffer&& other) noexcept {
   VkBuffer tmp_buffer = handle_;
   bool tmp_owns_memory = owns_memory_;
+  bool tmp_memory_bundled = memory_bundled_;
 
   buffer_properties_ = other.buffer_properties_;
   allocator_ = other.allocator_;
   memory_ = std::move(other.memory_);
   owns_memory_ = other.owns_memory_;
+  memory_bundled_ = other.memory_bundled_;
   is_copy_ = other.is_copy_;
   handle_ = other.handle_;
 
   other.handle_ = tmp_buffer;
   other.owns_memory_ = tmp_owns_memory;
+  other.memory_bundled_ = tmp_memory_bundled;
 
   return *this;
 }
@@ -119,14 +126,22 @@ VulkanBuffer::~VulkanBuffer() {
   // ownership of the underlying resource.
   if (handle_ != VK_NULL_HANDLE && !is_copy_) {
     if (owns_memory_) {
-      vmaDestroyBuffer(allocator_, handle_, memory_.allocation);
+      if (memory_bundled_) {
+        vmaDestroyBuffer(allocator_, handle_, memory_.allocation);
+        // Prevent the underlying memory allocation from being freed; it was
+        // freed by vmaDestroyImage
+        memory_.allocation = VK_NULL_HANDLE;
+      } else {
+        vkDestroyBuffer(this->device(), handle_, nullptr);
+        // Allow underlying memory allocation to be freed by the destructor of
+        // Allocation class
+      }
     } else {
       vkDestroyBuffer(this->device(), handle_, nullptr);
+      // Prevent the underlying memory allocation from being freed since this
+      // object doesn't own it
+      memory_.allocation = VK_NULL_HANDLE;
     }
-    // Prevent the underlying memory allocation from being freed; it was either
-    // freed by vmaDestroyBuffer, or this resource does not own the underlying
-    // memory
-    memory_.allocation = VK_NULL_HANDLE;
   }
 }
 
@@ -136,6 +151,24 @@ VmaAllocationInfo VulkanBuffer::allocation_info() const {
   return info;
 }
 
+void VulkanBuffer::bind_allocation_impl(const Allocation& memory) {
+  VK_CHECK_COND(!memory_, "Cannot bind an already bound allocation!");
+  if (!is_copy_) {
+    VK_CHECK(vmaBindBufferMemory(allocator_, memory.allocation, handle_));
+  }
+}
+
+void VulkanBuffer::bind_allocation(const Allocation& memory) {
+  bind_allocation_impl(memory);
+  memory_.allocation = memory.allocation;
+}
+
+void VulkanBuffer::acquire_allocation(Allocation&& memory) {
+  bind_allocation_impl(memory);
+  memory_ = std::move(memory);
+  owns_memory_ = true;
+}
+
 VkMemoryRequirements VulkanBuffer::get_memory_requirements() const {
   VkMemoryRequirements memory_requirements;
   vkGetBufferMemoryRequirements(this->device(), handle_, &memory_requirements);

backends/vulkan/runtime/vk_api/memory/Buffer.h

@@ -100,6 +100,10 @@ class VulkanBuffer final {
   Allocation memory_;
   // Indicates whether the underlying memory is owned by this resource
   bool owns_memory_;
+  // Indicates whether the allocation for the buffer was created with the buffer
+  // via vmaCreateBuffer; if this is false, the memory is owned but was bound
+  // separately via vmaBindBufferMemory
+  bool memory_bundled_;
   // Indicates whether this VulkanBuffer was copied from another VulkanBuffer,
   // thus it does not have ownership of the underlying VKBuffer
   bool is_copy_;
@@ -162,13 +166,21 @@ class VulkanBuffer final {
     return (handle_ == other.handle_) && is_copy_;
   }
 
-  inline void bind_allocation(const Allocation& memory) {
-    VK_CHECK_COND(!memory_, "Cannot bind an already bound allocation!");
-    if (!is_copy_) {
-      VK_CHECK(vmaBindBufferMemory(allocator_, memory.allocation, handle_));
-    }
-    memory_.allocation = memory.allocation;
-  }
+ private:
+  void bind_allocation_impl(const Allocation& memory);
+
+ public:
+  /*
+   * Given a memory allocation, bind it to the underlying VkImage. The lifetime
+   * of the memory allocation is assumed to be managed externally.
+   */
+  void bind_allocation(const Allocation& memory);
+
+  /*
+   * Given a rvalue memory allocation, bind it to the underlying VkImage and
+   * also acquire ownership of the memory allocation.
+   */
+  void acquire_allocation(Allocation&& memory);
 
   VkMemoryRequirements get_memory_requirements() const;