AI4OPT Seminar Series
Date: Thursday, April 13, 2023
Time: Noon – 1:00 pm
Location: Instructional Center 115 (Scale Up Room) - (759 Ferst Dr, Atlanta, GA 30318)
Join Virtually: https://gatech.zoom.us/j/99381428980
Speaker: Thiago Serra
Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regions
Abstract: One surprising trait of neural networks is the extent to which their connections can be pruned with little to no effect on accuracy. But when we cross a critical level of parameter sparsity, pruning any further leads to a sudden drop in accuracy. This drop plausibly reflects a loss in model complexity, which we aim to avoid. In this work, we explore how sparsity also affects the geometry of the linear regions defined by a neural network. We observe that pruning affects accuracy similarly to how sparsity affects the number of linear regions. Conversely, we find out that selecting the sparsity across layers to maximize the expected maximum number of linear regions very often improves accuracy in comparison to pruning as much but with the same sparsity in all layers, thereby providing us guidance on where to prune from each neural network architecture.
This is joint work with Junyang Cai, Khai-Nguyen Nguyen, Nishant Shrestha, Aidan Good, and Ruisen Tu from Bucknell University, as well as Xin Yu and Shandian Zhe from University of Utah. To be published in the proceedings of CPAIOR 2023.
Bio: Thiago Serra is an assistant professor of analytics and operations management at Bucknell University's Freeman College of Management. Previously, he was a visiting research scientist at Mitsubishi Electric Research Labs from 2018 to 2019, and an operations research analyst at Petrobras from 2009 to 2013. He has a Ph.D. in operations research from Carnegie Mellon University's Tepper School of Business, and received the Gerald L. Thompson Doctoral Dissertation Award in Management Science in 2018. During his PhD., he was also awarded the INFORMS Judith Liebman Award and a best poster award at the INFORMS Annual Meeting. His research at the intersection of discrete optimization and machine learning is currently supported by the National Science Foundation (NSF) award IIS 2104583.
Lunch will be served at the seminar. So, please stop by 15 minutes before the seminar to pick up lunch.
To receive AI4OPT seminar announcements, please sign up to our mailing list. at https://lists.isye.gatech.edu/mailman/listinfo/ai4opt-seminars.
Past seminars can be found at https://www.ai4opt.org/seminars/past-seminars.