Figure Overview of RaNNCDistributed Deep Learning Framework for Extremely Large-scale Neural NetworksTANAKA Masahiroeep learning technology has been drawing a great deal of attention in recent years. It generally uses hardware accelerators, such as graphics processing units (GPUs). However, articial neural net-works used in deep learning are rapidly ex-panding in scale. Some recently proposed neural networks exceed the memory capac-ity of GPUs. For example, a network named BERT, proposed in 2018, which achieved a breakthrough in natural language process-ing, had 340 million parameters—among the largest number of parameters ever at the time. The number of parameters has continued to grow: T5, proposed in 2019, has a staggering 11 billion parameters.We have been developing RaNNC (Rapid Neural Network Connector), a framework that automatically partitions an extreme-ly large-scale neural network into smaller units and distributes them onto multiple GPUs. RaNNC can determine how to opti-mally partition a large neural network by estimating the computational loads and re-quired GPU memory. We have already suc-ceeded in training a neural network with several billion parameters more eciently using RaNNC than is possible using existing deep learning frameworks in which a net-work partitioning has to be tuned manually. In the future, we plan to evaluate the appli-cability of RaNNC to a wide variety of neu-ral networks and release it as open source software. We also plan to integrate trained large-scale neural networks into WISDOM X (a large-scale web information analytical system), WEKDA (a next-generation spoken dialogue system), MICSUS (a multimodal interactive care support system) and other Senior ResearcherData-driven Intelligent System Research CenterUniversal Communication Research InstitutePh.D. (Informatics)systems to enhance their performance. We then intend to make trained networks avail-able for use by private companies and other organizations.I have been working mostly from home for the past three years for family reasons. Although it took me a while to become ac-customed to this new working style, I even-tually adapted to it with the support of the people around me. This experience is now helping me cope with the work-from-home practices widely adopted in response to the COVID-19 pandemic.What do you like the most about being a researcher?My current research position allows me to tackle unsolved technical problems in my own way. This is a very exciting aspect of my work.What are you currently interested in outside of your research?I recently refurbished my viola, which hadn’t been used for several years, and resumed playing it in my spare time. My daughter also recently started playing the piano, so we sometimes play together. I nd playing music, even for a brief period of time, to be very refreshing.What advice would you like to pass on to people aspiring to be researchers?Because graduate students can spend only a limited amount of time on research, they often feel rushed to complete their degrees. However, I recommend that they think carefully and pursue research that would be truly valuable in their elds. I believe this will be more rewarding for them over the long run.QAQAQA●BiographyBorn in Nara Prefecture in 1981.Graduated from Kyoto University Undergraduate School of Informatics and Mathematical Science in 2004.Earned a doctorate from Kyoto University Graduate School of Informatics in 2009Jointed NICT in 2009.Assumed his current position in 2016.●Awards, etc.Co-recipient of the DOCOMO Mobile Science Award (2015) and the Maejima Hisoka Award (2016)DQ&AsNICT NEWS 2020 No.612File 13
元のページ ../index.html#14