VIDEO | Neural Magic CEO: We Shrink Machine Learning Models by 80-90%

(US and Canada) Neural Magic’s CEO Brian Stevens and VP of Business Development Jay Marshall speak with Kirk Ball, EVP and CIO of Giant Eagle and Editorial Board Member of CDO Magazine, in a video interview about how the AI startup helps customers simplify their machine learning deployment to help them use compute heavy models on existing infrastructure.

Stevens says that Neural Magic built a software engine that can take state-of-the-art machine learning models and run them at amazing speeds on ordinary computers. He adds that the company was founded on the principles of removing friction.

Sharing his POV on the timing of Neural Magic, Marshall says that data science and machine learning terms and technologies have been around for a while. He however notes that topics like deep learning models and neural networks have surfaced in recent years making way for things like computer vision with Edge computing. He adds that Neural Magic optimizes and runs those models on existing infrastructure that teams already know how to manage.

Sharing the secret sauce of Neural Magic and what it enables, Stevens says that it is introducing the notion of sparsity or pruning into ML models, shrinking their size by 80-90% while preserving accuracy. The compressed model is not only as accurate but also reduces the computation needed to process the model.

Stevens explains that even the biggest models are very tiny from a computer architecture perspective and can fit inside the average memory on a desktop.

Speaking of customer success stories and real-world use cases, Marshall mentions how a large retail outlet has been able to run twice as many cameras on the same computing capacity. He also discusses use cases like large-scale batch processing, natural language processing, and ingesting large quantities of documents or text. Instead of large GPU clusters, Neural Magic helps run these models on a regular CPU infrastructure reducing the cost footprint.

CDO Magazine appreciates Brian Stevens and Jay Marshall for sharing their insights and data success stories with our global community.