|
| 1 | +# OADP Performance Testing |
| 2 | + |
| 3 | +This document provides guidance on performance testing OADP (OpenShift API for Data Protection) with Velero using comprehensive testing tools and methodologies. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +OADP performance testing is critical for understanding backup and restore behavior at scale. The OADP team has developed a comprehensive performance testing toolkit that uses industry-standard tools to simulate realistic workloads and measure Velero's performance characteristics. |
| 8 | + |
| 9 | +## Performance Testing Toolkit |
| 10 | + |
| 11 | +The OADP team maintains a dedicated performance testing repository that provides: |
| 12 | + |
| 13 | +- **Automated test scripts** for creating large-scale Kubernetes objects (30k-300k) |
| 14 | +- **Velero backup/restore performance testing** with detailed analysis |
| 15 | +- **Industry-standard tooling** using [kube-burner](https://github.com/kube-burner/kube-burner) for efficient object creation |
| 16 | +- **Comprehensive documentation** and usage guides |
| 17 | +- **Performance analysis scripts** for identifying bottlenecks |
| 18 | + |
| 19 | +### Repository Access |
| 20 | + |
| 21 | +**GitHub Repository**: [https://github.com/shubham-pampattiwar/velero-performance-testing](https://github.com/shubham-pampattiwar/velero-performance-testing) |
| 22 | + |
| 23 | +The repository contains everything needed for comprehensive OADP performance testing, including: |
| 24 | +- Pre-configured test scenarios (30k and 300k objects) |
| 25 | +- Velero installation and setup scripts |
| 26 | +- Performance analysis tools |
| 27 | +- Detailed documentation |
| 28 | + |
| 29 | +## Quick Start Guide |
| 30 | + |
| 31 | +### Prerequisites |
| 32 | + |
| 33 | +- OpenShift/Kubernetes cluster with sufficient resources |
| 34 | +- [kube-burner](https://github.com/kube-burner/kube-burner) installed |
| 35 | +- Cluster-admin privileges |
| 36 | +- OADP operator installed and configured |
| 37 | + |
| 38 | +### Basic Performance Test Workflow |
| 39 | + |
| 40 | +1. **Clone the performance testing repository**: |
| 41 | + ```bash |
| 42 | + git clone https://github.com/shubham-pampattiwar/velero-performance-testing.git |
| 43 | + cd velero-performance-testing |
| 44 | + ``` |
| 45 | + |
| 46 | +2. **Run a simple test** (30k objects): |
| 47 | + ```bash |
| 48 | + ./scripts/run-simple-test.sh |
| 49 | + ``` |
| 50 | + |
| 51 | +3. **Test OADP backup performance**: |
| 52 | + ```bash |
| 53 | + ./velero/backup-performance-test.sh |
| 54 | + ``` |
| 55 | + |
| 56 | +4. **Analyze results**: |
| 57 | + ```bash |
| 58 | + ./velero/analyze-performance.sh <backup-name> |
| 59 | + ``` |
| 60 | + |
| 61 | +5. **Clean up**: |
| 62 | + ```bash |
| 63 | + ./scripts/cleanup-simple.sh |
| 64 | + ``` |
| 65 | + |
| 66 | +### Large-Scale Testing |
| 67 | + |
| 68 | +For enterprise-scale testing with 300k objects: |
| 69 | + |
| 70 | +```bash |
| 71 | +# Create large-scale test objects |
| 72 | +./scripts/run-large-scale-test.sh |
| 73 | + |
| 74 | +# Test backup performance |
| 75 | +./velero/backup-performance-test.sh |
| 76 | + |
| 77 | +# Analyze and cleanup |
| 78 | +./velero/analyze-performance.sh <backup-name> |
| 79 | +./scripts/cleanup-large-scale.sh |
| 80 | +``` |
| 81 | + |
| 82 | +### Performance Analysis |
| 83 | + |
| 84 | +Use the toolkit's analysis scripts to identify bottlenecks: |
| 85 | +```bash |
| 86 | +# Detailed performance analysis |
| 87 | +./velero/analyze-performance.sh <backup-name> |
| 88 | + |
| 89 | +# Check resource utilization |
| 90 | +kubectl top nodes |
| 91 | +kubectl top pods -n openshift-adp-operator |
| 92 | +``` |
| 93 | + |
| 94 | +## Best Practices |
| 95 | + |
| 96 | +### Testing Guidelines |
| 97 | + |
| 98 | +1. **Start small**: Begin with 30k object tests before attempting large-scale tests |
| 99 | +2. **Monitor resources**: Keep an eye on cluster resource utilization |
| 100 | +3. **Test incrementally**: Gradually increase object counts to find limits |
| 101 | +4. **Document results**: Track performance metrics across different configurations |
| 102 | + |
| 103 | +### Production Considerations |
| 104 | + |
| 105 | +1. **Test in staging**: Never run large-scale performance tests in production |
| 106 | +2. **Resource planning**: Ensure sufficient cluster resources before testing |
| 107 | +3. **Backup windows**: Plan backup windows based on performance test results |
| 108 | +4. **Monitoring**: Implement monitoring based on performance testing insights |
| 109 | + |
| 110 | +## Support and Contributing |
| 111 | + |
| 112 | +For questions about performance testing: |
| 113 | +1. Review the [performance testing repository documentation](https://github.com/shubham-pampattiwar/velero-performance-testing) |
| 114 | +2. Check existing [OADP issues](https://github.com/openshift/oadp-operator/issues) |
| 115 | +3. Contribute improvements to the performance testing toolkit |
| 116 | + |
| 117 | +## Performance Testing Repository Structure |
| 118 | + |
| 119 | +The external repository includes: |
| 120 | +- **Automated scripts** for object creation and cleanup |
| 121 | +- **Velero integration** scripts for backup/restore testing |
| 122 | +- **Performance analysis** tools and reports |
| 123 | +- **Multiple test scenarios** (30k, 300k objects) |
| 124 | +- **Comprehensive documentation** with troubleshooting guides |
| 125 | + |
| 126 | +For complete usage instructions, refer to the [Velero Performance Testing Repository](https://github.com/shubham-pampattiwar/velero-performance-testing). |
0 commit comments