Accelerating Deep Neural Network Learning for Speech
Recognition on a Cluster of GPUs
Author/Presenter
Event Type
Workshop
Deep Learning
Machine Learning
SIGHPC Workshop
TimeMonday, November 13th11:30am -
12pm
Location502-503-504
DescriptionWe train deep neural networks to solve the acoustic
modeling problem for large-vocabulary continuous speech
recognition. We employ distributed processing using a
cluster of GPUs. On modern GPUs, the sequential
implementation takes over a day to train, and efficient
parallelization without losing accuracy is notoriously
hard. We show that ASGD methods for parallelization are
not efficient for this application. Even with 4 GPUs,
the overhead is significant, and the accuracies achieved
are poor. We adapt a P-learner K-step model averaging
algorithm that with 4 GPUs achieves accuracies
comparable to that achieved by the sequential
implementation. We further introduce adaptive measures
that make our parallel implementation scale to the full
cluster of 20 GPUs. Ultimately our parallel
implementation achieves better accuracies than the
sequential implementation with a 6.1 times
speedup.
Author/Presenter
