Difference to skypilot?

[TomExMachina]

I'm now to the point where I need to optimize for cost/scale as much as possible. I've just started getting to know skypilot and I came across openmodelz. How does it differ from skypilot?

[gaocegege]

Thanks for raising this, I will write a doc for this in 1 hour.

[gaocegege]

Ongoing PR: tensorchord/openmodelz-docs#13 (review)

[gaocegege]

Here's a breakdown of how the two platforms differ:

OpenModelZ:

• Autoscaling: OpenModelZ supports autoscaling, allowing your deployment to dynamically adjust the number of replicas based on the inflow of requests. This helps optimize resource allocation and ensures efficient handling of varying workloads.
• Flexibility: Deployments on OpenModelZ can be dynamically scheduled on any machine, providing flexibility in resource allocation. In contrast, Skypilot requires a fixed set of machines and doesn't allow for changes to the application once it's deployed without recreating the cluster from scratch.
• Cloud Native: OpenModelZ is a container-based solution, leveraging technologies like Docker to package applications into lightweight and portable containers. This approach enables easier deployment and management across different environments.
• Traffic Management: OpenModelZ provides public endpoints for your deployments, allowing external access to your applications. This simplifies the process of serving API requests and enables interaction with your application from external systems.
• Observability: OpenModelZ offers built-in observability features, with metrics of your deployments stored in Prometheus. You can visualize these metrics using tools like Grafana or the Prometheus dashboard. This includes metrics such as CPU, memory, GPU usage, as well as the number of requests and replicas.

Skypilot:

• Virtual Machines (VMs): Skypilot focuses on provisioning and managing VMs in a cloud environment. It provides a streamlined process for deploying and configuring VMs declaratively.
• Setup Bash Scripts: Skypilot allows you to run setup bash scripts as part of the VM provisioning process, enabling you to customize the configuration of your VMs based on specific requirements.

When comparing OpenModelZ and Skypilot, consider factors such as autoscaling support, flexibility in resource allocation, containerization, traffic management options, and built-in observability features. These considerations will help determine which platform aligns better with your specific needs and use cases.