Sheng Z.

Deep learning community has largely evolved independently from the prior community of data science and machine learning community in NumPy. While most deep learning frameworks now provide NumPy-like math and array library, they differ in the definition of the operations which creates a steeper learning curve of deep learning for machine learning practitioners and data scientists. This creates a chasm not only in the skillsets of the two different communities, but also hinders the exchange of… Show more

Deep learning community has largely evolved independently from the prior community of data science and machine learning community in NumPy. While most deep learning frameworks now provide NumPy-like math and array library, they differ in the definition of the operations which creates a steeper learning curve of deep learning for machine learning practitioners and data scientists. This creates a chasm not only in the skillsets of the two different communities, but also hinders the exchange of knowledge. The next major version, 2.0, of Apache MXNet (incubating) seeks to bridge the fragmented deep learning and machine learning ecosystem. It provides NumPy-compatible programming experiences and simple enhancements to NumPy for deep learning with the new Gluon 2.0 interface. The NumPy-compatible array API also brings the advances in GPU acceleration, auto-differentiation, and high-performance one-click deployment to the NumPy ecosystem. Show less

Natural language processing (NLP) is at the core of the pursuit for artificial intelligence, with deep learning as the main powerhouse of recent advances. Most NLP problems remain unsolved. The compositional nature of language enables us to express complex ideas, but at the same time making it intractable to spoon-feed enough labels to the data-hungry algorithms for all situations. Recent progress on unsupervised language representation techniques brings new hope. In this hands-on tutorial, we… Show more

Natural language processing (NLP) is at the core of the pursuit for artificial intelligence, with deep learning as the main powerhouse of recent advances. Most NLP problems remain unsolved. The compositional nature of language enables us to express complex ideas, but at the same time making it intractable to spoon-feed enough labels to the data-hungry algorithms for all situations. Recent progress on unsupervised language representation techniques brings new hope. In this hands-on tutorial, we walk through these techniques and see how NLP learning can be drastically improved based on pre-training and fine-tuning language representations on unlabelled text. Specifically, we consider shallow representations in word embeddings such as word2vec, fastText, and GloVe, and deep representations with attention mechanisms such as BERT. We demonstrate detailed procedures and best practices on how to pre-train such models and fine-tune them in downstream NLP tasks as diverse as finding synonyms and analogies, sentiment analysis, question answering, and machine translation. All the hands-on implementations are with Apache (incubating) MXNet and GluonNLP, and part of the implementations are available on Dive into Deep Learning. Show less

Deep learning has become the dominant approach to NLP problems, especially when applied on large scale corpora. Recent progress on unsupervised pre-training techniques such as BERT, ELMo, GPT-2, and language modeling in general, when applied on large corpora, is shown to be effective in improving a wide variety of downstream tasks. These techniques push the limits of available hardware, requiring specialized frameworks optimized for GPU, ASIC, and distributed cloud-based training. A few… Show more

Deep learning has become the dominant approach to NLP problems, especially when applied on large scale corpora. Recent progress on unsupervised pre-training techniques such as BERT, ELMo, GPT-2, and language modeling in general, when applied on large corpora, is shown to be effective in improving a wide variety of downstream tasks. These techniques push the limits of available hardware, requiring specialized frameworks optimized for GPU, ASIC, and distributed cloud-based training. A few complexities pose challenges to scale these models and algorithms effectively. Compared to other areas where deep learning is applied, these NLP models contain a variety of moving parts: text normalization and tokenization, word representation at subword-level and word-level, variable-length models such as RNN and attention, and sequential decoder based on beam search, among others. In this hands-on tutorial, we take a closer look at the challenges from these complexities and see how with proper tooling with Apache MXNet and GluonNLP, we can overcome these challenges and achieve state-of-the-art results for real-world problems. GluonNLP is a powerful new toolkit that combines MXNet’s speed, the flexibility of Gluon, and an extensive new library automating the most laborious aspects of deep learning for NLP. Show less

Deep Learning lectures for 2018 JHU Summer School on Human Language Technology.