Optimizing Your Optimization: A Holistic Approach