syllabus

DATASCI W266: Natural Language Processing with Deep Learning

Course Overview
Grading
Final Project
Course Resources
Schedule and Readings

• Piazza - we'll use this for Q&A, and this will be the fastest way to reach the course staff. Note that you can post anonymously, and/or make posts visible only to instructors for private questions.

Course Overview

Understanding language is fundamental to human interaction. Our brains have evolved language-specific circuitry that helps us learn it very quickly; however, this also means that we have great difficulty explaining how exactly meaning arises from sounds and symbols. This course is a broad introduction to linguistic phenomena and our attempts to analyze them with machine learning. We will cover a wide range of concepts with a focus on practical applications such as information extraction, machine translation, sentiment analysis, and summarization.

Prerequisites:

• Language: All assignments will be in Python using Jupyter notebooks, NumPy, and TensorFlow.
• Time: There are 6-7 substantial assignments in this course as well as a term project. Make sure you give yourself enough time to be successful! In particular, you may be in for a rough semester if you have other significant commitments at work or home, or take both this and any of 210 (Capstone), 261, or 271 :)
• MIDS 207 (Machine Learning): We assume you know what gradient descent is. We'll review simple linear classifiers and softmax at a high level, but make sure you've at least heard of these! You should also be comfortable with linear algebra, which we'll use for vector representations and when we discuss deep learning.

Contacts and resources:

• Course website: GitHub datasci-w266/2021-summer-main
• Piazza - we'll use this for Q&A, and this will be the fastest way to reach the course staff. Note that you can post anonymously, and/or make posts visible only to instructors for private questions.
• Email list for course staff (expect a somewhat slower response here): mids-nlp-instructors@googlegroups.com

Live Sessions:

• (Section 1) Monday 6:30 - 8p Pacific (Joachim Rahmfeld)
• (Section 2) Tuesday 2 - 3:30p Pacific (Peter Grabowski)
• (Section 3) Tuesday 4 - 5:30p Pacific (Daniel Cer)
• (Section 4) Wednesday 6:30 - 8p Pacific (Mike Tamir, Paul Spiegelhalter)
• (Section 5) Thursday 4 - 5:30p Pacific (Zack Alexander)
• (Section 6) Friday 4 - 5:30p Pacific (Mark Butler)

Teaching Staff Office Hours:

• Zack Alexander: Thursday immediately after his live session
• Mark Butler: Monday at 1pm PST and Friday immediately after his live session
• Daniel Cer: Friday at noon PST
• Peter Grabowski: Tuesday at 1pm PST
• Joachim Rahmfeld: Wednesday at noon PST
• Mike Tamir/Paul Spiegelhalter: Wednesday immediately after his live session
• Drew Plant: Monday at 6pm PST
• Gurdit Chahal: Tuesday at 3pm PST

Office hours are for the whole class; students from any section are welcome to attend any of the times above.

Async Instructors:

• Dan Gillick
• James Kunz
• Kuzman Ganchev

Grading

Breakdown

Your grade report can be found at https://w266grades.appspot.com.

Your grade will be determined as follows:

• Weekly Assignments: 40%
• Final Project: 60%
• Participation: Up to 10% bonus

There will be a number of smaller assignments throughout the term for you to exercise what you learned in async and live sessions. Some assignments may be more difficult than others, and may be weighted accordingly.

Participation will be graded holistically, based on live session participation as well as participation on Piazza (or other activities that improve the course this semester or into the future). Do not stress about this part.

Letter Grades

We curve the numerical grade to a letter grade. While we don't release the curve, it usually results in about a quarter of the class each receiving A, A-, B+, and B. Exceptional cases receive A+, C, or F, as appropriate.

A word of warning: Given that we (effectively) release solutions to assignments in the form of unit tests, it shouldn't be surprising that most students earn near perfect scores. Since the variance is so low, assignment scores aren't the primary driver of the final letter grade for most students. A good assignment score is necessary, but not sufficient, for a strong grade in the class. A well structured, novel project with good analysis is what makes the difference between a high B/B+ and an A-/A.

As mentioned above: this course is a lot of work. Give it the time it deserves and you'll be rewarded intellectually and on your transcript.

Late Day Policy

We recognize that sometimes things happen in life outside the course, especially in MIDS where we all have full time jobs and family responsibilities to attend to. To help with these situations, we are giving you 5 "late days" to use throughout the term as you see fit. Each late day gives you a 24 hour (or any part thereof) extension to any deliverable in the course except the final project presentation or report. (UC Berkeley needs grades submitted very shortly after the end of classes.)

Once you run out of late days, each 24 hour period (or any part thereof) results in a 10 percentage point deduction on that deliverable's grade.

You can use a maximum of 2 late days on any single deliverable. We will not be accepting any submissions more than 48 hours past the original due-date, even if you have late days. (We want to be more flexible here, but your fellow students also want their graded assignments back promptly!)

We don't anticipate granting extensions beyond these policies. Plan your time accordingly!

More serious issues

If you run into a more serious issue that will affect your ability to complete the course, please email the instructors mailing list and cc MIDS student services. A word of warning though: in previous sections, we have had students ask for INC grades because their lives were otherwise busy. Mostly we have declined, opting instead for the student to complete the course to the best of their ability and have a grade assigned based on that work. (MIDS prefers to avoid giving INCs, as they have been abused in the past.) The sooner you start this process, the more options we (and the department) have to help. Don't wait until you're suffering from the consequences to tell us what's going on!

Final Project

See the Final Project Guidelines

Requirements for in-class participation

We believe in the importance of the social aspects of learning: between students, and between students and instructors, and we recognize that knowledge-building is not solely occurring on an individual level, but that it is built by social activity involving people and by members engaged in the activity. Participation and communication are key aspects of this course that are vital to the learning experiences of you and your classmates.

Therefore, we like to remind all students of the following requirements for live class sessions:

• Students are required to join live class sessions from a study environment with video turned on and with a headset for clear audio, without background movement or background noise, and with an internet connection suitable for video streaming.

• You are expected to engage in class discussions, breakout room discussions and exercises, and to be present and attentive for your and other teams’ in-class presentations.

• Keep your microphone on mute when not talking to avoid background noise. Do your best to minimize distractions in the background video, and ensure that your camera is on while you are engaged in discussions.

That said, in exceptional circumstances, if you are unable to meet in a space with no background movement, or if your connection is poor, make arrangements with your instructor (beforehand if possible) to explain your situation. Sometimes connections and circumstances make turning off video the best option. If this is a recurring issue in your study environment, you are responsible for finding a different environment that will allow you to fully participate in classes, without distraction to your classmates.

Failure to adhere to these requirements will result in an initial warning from your instructor(s), followed by a possible reduction in grades or a failing grade in the course.

Course Resources

We are not using any particular textbook for this course. We’ll list some relevant readings each week. Here are some general resources:

• Speech and Language Processing (2nd edition) (Jurafsky and Martin)
• Speech and Language Processing (3rd edition draft) (Jurafsky and Martin) - free online!
• NLTK Book - Accompanies NLTK (Natural Language ToolKit) and includes useful, practical descriptions (with python code) of basic concepts.
• Deep Learning (Goodfellow, Bengio, and Courville)

We’ll be posting materials to the course GitHub repo.

Note: the syllabus below might be subject to change. We'll be sure to announce anything major on Piazza.

Code References

The course will be taught in Python, and we'll be making heavy use of NumPy, TensorFlow, Keras, and Jupyter (IPython) notebooks. We'll also be using Git for distributing and submitting materials. If you want to brush up on any of these, we recommend:

• Git tutorials: Introduction / Cheat Sheet, or interactive tutorial
• Python / NumPy: Stanford's CS231n has an excellent tutorial.
• TensorFlow: We'll go over the basics of TensorFlow and Keras in Assignment 2.
  Effective TensorFlow is a great reference, ranging from the absolute basics through advanced topics like multi-GPU training, tf.learn, and debugging.
  You can also check out the tutorials on the TensorFlow website, but these can be somewhat confusing if you're not familiar with the underlying models. Also, look at the Keras Guide as we will be using Keras in this class.

Misc. Deep Learning and NLP References

A few useful papers that don’t fit under a particular week. All optional, but interesting!

• (optional) Chris Olah’s blog and Distill
• (optional) GloVe: Global Vectors for Word Representation (Pennington, Socher, and Manning, 2014)

---

Tentative Schedule and Readings

We'll update the table below with assignments as they become available, as well as additional materials throughout the semester. Keep an eye on GitHub for updates!

Dates are tentative: assignments in particular may change topics and dates. (Updated slides for each week will be posted during the live session week.)

Live Session Slides: [available with @berkeley.edu address]

Deliverables

Note: we will update this table as we release (approximately weekly) assignments. Each assignment will be released around the last live session of the week and due approximately one week later.

	Topic	Release	Deadline
Assignment 0	Course Set-up GitHub Piazza Google Cloud		May 16
Assignment 1	Assignment 1 Information Theory Probability		May 23
Assignment 2	Assignment 2 Neural Networks		May 30
Project Proposal	Final Project Guidelines		Jun 5
Assignment 3	Assignment 3 Embeddings ML Fairness		Jun 6
Assignment 4	Assignment 4 Classification Convolutional Neural Nets		Jun 13
Assignment 5	Assignment 5 Language Models		Jun 20
Assignment 6	Assignment 6 Machine Translation		Jun 27
Assignment 7	Assignment 7 Various Topics		Jul 6
Project Reports			due July 31 (hard deadline)
Project Presentations			in-class August 2-6

Course Schedule

	Async to Watch	Topics	Materials
Week 1 (May 3)	Introduction	Overview of NLP applications NLP tasks, model structures and neural architectures Ambiguity and grounding in language Information theory review ML models: Logistic regression and feed forward networks	Skim: NLTK book chapter 1 (python and basics) Skim: NLTK book chapter 2 (data resources) Read: AI’s Language Problem (Technology Review) Read: The Rise and Fall of the English Sentence Optional: The Interpreter (New Yorker) Optional: Introduction to Linguistic Typology Optional / fun: Tensorflow Playground
Week 2 (May 10)	5.2 Softmax Classification 5.4 Neural network recap 5.6 Neural network training loss	ML models: Logistic regression and feed forward networks Learning and back propagation Initialization	Skim: Original backprop paper Optional / fun: Tensorflow Playground
Week 3 (May 17)	Classification and Sentiment (up to 2.6), 4.2, 4.12 - 4.17, 6.10, 6.12	Sentiment lexicons Aggregated sentiment applications Bag-of-words models Introduction to word embeddings	Skim: Opinion Mining and Sentiment Analysis (Pang and Lee 2008) - focus on Chapters 1-4 Optional: Natural Language Processing (almost) from Scratch (Collobert et al., 2011)
Week 4 (May 24)	Classification and Sentiment (2.7 onwards) Note: you should review Async 5.3, 5.4, and 5.5.	Convolutional neural networks for NLP	Read: Understanding Convolutional Neural Networks for NLP Read: Convolutional Neural Networks for Sentence Classification (Yoon Kim, 2014)
Week 5 (May 31)	Language Modeling I, 4.1-4.4, 5.8, 5.11	LM applications N-gram models Smoothing methods Representations of meaning Distributed representations Neural Net LMs Word embeddings Softmax variants	Language model introduction: Skim: Chen and Goodman Survey Skim: 1 Billion Word Benchmark Optional: Natural Language Corpus Data (Peter Norvig) [Language Modeling Notebook] Distributed representations: Optional: Brown Clustering (Brown et al. 1992) Read: CBOW and SkipGram (Mikolov et al. 2013) Optional: Deep Learning, NLP, and Representations (Chris Olah's blog) Read: A Neural Probabilistic Language Model (Bengio et al. 2003) [Word Embeddings Notebook] [NPLM Notebook]
Interlude (Extra Material)	Units of Meaning: Words, Morphology, Sentences	Edit distance for strings Tokenization Sentence splitting	Skim: NLTK book chapter 3 (processing raw text) Skim: Natural Language Corpus Data (Peter Norvig) (if you didn't read in Week 2) Read: Sentence Boundary Detection and the Problem with the U.S.
Week 6 (June 7)	Language Modeling II	Recurrent Neural Nets State of the art: Advanced Embeddings and Transfer Learning	Read or skim: How the backpropagation algorithm works Read: Understanding LSTM Networks (Chris Olah's blog) Optional: ”Deep contextualized word representations”, (Peters et al, 2018) Optional (skim): Tensorflow LSTM Language Model Tutorial
Week 7 (June 14)	Machine Translation I Machine Translation II	Word- and phrase-based MT IBM alignment models Evaluation Neural MT with sequence-to-sequence models and attention	Read: Sequence to Sequence Learning with Neural Networks Read: Neural Machine Translation by Jointly Learning to Align and Translate Optional: Google’s Neural Machine Translation System Optional: Attention and Augmented Recurrent Neural Networks (section on “Attentional Interfaces” has an awesome visualization of an MT example, showing alignments)
Week 8 (June 21)	No Async	Self-Attention Transformers Transfer Learning	Read (skim): Attention is All You Need (Vaswani et. al., 2017) Read (skim): BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, (Devlin et al, 2018) Read (skim): The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)”, (Alammar, 2018)
Week 9 (June 28)	Entities	From syntax to semantics Named Entity Recognition Coreference Resolution Entity Linking Information Extraction	Read: NLTK Book Chapter 7 (Extracting Information from Text) Optional: Simple Coreference Resolution with Rich Syntactic and Semantic Features (Haghighi and Klein 2009, rule-based coreference) Optional: Improving Coreference Resolution by Learning Entity-Level Distributed Representations (Clark and Manning 2016, neural coreference)
Week 10 (July 5)	Summarization	Single- vs. multi-document summarization Extractive and abstractive summarization Classical summarization algorithms Evaluating generated summaries Neural summarization architectures	Skim: A Survey on Automatic Text Summarization (Das and Martins, 2007) Read: A Neural Attention Model for Abstractive Sentence Summarization (Rush et al. 2015) Skim: Get To The Point: Summarization with Pointer-Generator Networks (See et al. 2017)
Week 11 (July 12)	No Async	Document Representation Document Term Matrix Document Vectors Topic Modeling and LDA Document Classification	Read (skim): DocBERT: BERT for Document Classification (Adhikari et. al., 2019) Read: Hierarchical Attention Networks for Document Classification (Yang et. al., 2019) Read (skim): Multi-passage BERT: A globally normalized BERT model for open-domain question answering (Wang et. al., 2019)
Week 12 (July 19)	Part of Speech Supplementary Videos Dependency Parsing (Parsing I) Constituency Parsing (Parsing II)	Tag sets Most frequent tag baseline HMM/CRF models Dependency trees Transition-based parsing: Arc‑standard, Arc‑eager Context-free grammars (CFGs) CKY algorithm Probabilistic CFGs Note: Section 7.6 this week in the async is optional.	Read: NLTK book chapter 5: Categorizing and Tagging Words Read: A Universal Part-of-Speech Tagset Optional: Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? Read: SyntaxNet (Parsey McParseface) Skim: Dependency Parsing (J&M Chapter 14) Skim: Constituency Parsing (J&M Chapter 13) Optional: A Fast and Accurate Dependency Parser using Neural Networks (Chen & Manning 2014) Skim: NLTK book chapter 8 (analyzing sentence structure) Skim: Accurate Unlexicalized Parsing (Klein & Manning 2003) Play: Stanford parser (online demo) Optional / reference: Penn Treebank Constituent Tags [Optional: Interactive HMM Demo] [Optional: Interactive CKY Demo]
Week 13 (July 26)	Information Retrieval	Building a Search Index Ranking TF-IDF Click signals Neural IR models	Read: Web Search for a Planet (Google) Read: The Anatomy of a Large-Scale Hypertextual Web Search Engine Skim: "An Introduction to Information Retrieval", sections 6.2 and 6.3 Optional: PageRank (Page, et al. 1999)
Week 14 (Aug 2)		In class project presentations

Thanks for a great semester!