Discriminative models take an instance as input, and predict a label

• Real-valued labels are regressions

• Finite-set labels are classifications

• Training comes from a training set with true labels (supervised learning)

• Then searches set of possible models optimizing for some objective (minimizing prediction error, etc.)

• Many model types

  • Decision trees

  • ANNs (discussed later)

  • SVMs

  • nearest-neighbor

Generative models define probability distributions

• Generate new instances resembling training data

• HMMs, Bayesian networks, probabilistic models, na\"ive Bayes

• Trained via unsupervised learning, that is, lacking labels; note, labels mean generative model can exist for each class

• Can be used to discriminate estimating probability that a given instance is generated

discriminative models classify text into finite set of categories

Subject classification, spam filtering, author ident,

Given training set output a classifier $\gamma: \mathcal{D} \rightarrow \mathcal{C}$ predicting labels of new documents

Representing each document is important

• Option: each document is a bag of words with counts associated with them

  • Simple model, looses position information

  • May need to drop some things

Recall Bayes Theorem: $P(c|d) = (P(d|C)P(c))/P(d)$

• The probability of the class given the document

• $P(c)$ is the prior probability of label

• $P(d)$ is the probability of the document

• $P(c|d)$ is probability of $c$ given $d$

• $P(d|c)$ is probability of $d$ given $c$

Product Rule $P(A \land B) = P(A | B)P(B) = P(B | A)P(A)$

Sum Rule $P(A \lor B) = P(A) + P(B) - P(A \land B)$

Total probability

• If events are mutually exclusive, and sum of probabilities = 1,

• $P(B) = \sum P(B|A_{i})P(A_{i})$

MAP/Maximum a Posteriori classification

• Given document, our goal is to find the most likely label

• Done using argmax

• $argmax_{c \in C} P(c|d)$, or, assuming feature representation, of $P(x_{1}, \ldots x_{n} | c)P(c)$

• $P(c)$ is easily estimated, fraction of documents that are class $c$

• Estimating $P(x1...xN|c)$ is harder, as the number of combinations is exponential, and there won't be sufficient data

Make the Naive Bayes Assumption!

• $P(x_{1}, x_{2}, \ldots, x_{n} | c) = \prod_{i} P(x_{i}|c)$

• Assumes, of course, that feature values are conditionally independent given the class

• Classifier is $c_{NB }= \mathrm{argmax}_{c \in C} \prod_{i} P(x_{i}|c)$

• Values needed are polynomial!

• Standard method, and incredibly practical

• Works well when you have moderate/large training sets

• when attributes that describe instances are conditionally indepent given classes

• Learn Algorithm (See slide 15)

  • Estimate $P(c)$, call $\hat{P}(c)$

  • For each word, $w_{i}$ in vocabulary, estimate, $\hat{P}(w_{i}|c) = \frac{count(w_{i},c)}{\sum_{w\in V} count(w,c)}$

• Classify:

  • Compute above $c_{NB}$ using $\hat{P}$

• Smoothing done using Laplace smoothing with Pseudocounts!

Replace count-based bag-of-words with binary indicators

Negations have to be handled

Goal: generalization, and estimating performance

But, other goals possible

• Estimate generalization performance of model/hypothesis

• Tuning parameters

• Comparing two algorithms on specific task

Can't answer with perfect certainty

• Estimator must be chosen, determining the distribution thereof, and bounding it

• Must work regardless of distribution of training/test data

Statistical variation related to specific application is significant

Plan experiments well, don't test on training data!!

• It biases the performance estimator

• And don't let duplicates into training/testing sets if using data augmentation

• Holds for validation set used for early stopping/tuning parameters!

Confidence intervals are used

• $error_{\mathcal{D}}(h)$ is the 0-1 loss hypothesis on instances drawn according to $\mathcal{D}$. If test set has $N$ examples drawn independently (of $h$)

• 0-1 loss is the probability that $h$ makes mistake

• and $N \geq 30$

• Then, with 95% probability, the error likes in $error_{\mathcal{V}}(h) \pm 1.96\sqrt{\frac{error_{v}(h)(1 - error_{v}(h))}{N}}$

• So, if the hypothesis misclassifies 12 of the 40 examples, with 95% confidence, the error is in that range.

• Confidence interval can be generalized by changing real-number coefficient

  • Based on binomial distributions, mean error of an unbiased estimator

  • Can approximate binomials by normal distributions

  • And so, then, based on standard deviations in normal distribution

  • Can be justified due to the Central Limit Theorem

• Steps

  1. Pick Parameter to estimate

  2. Choose Estimator

  3. Determine Probability Distribution

  4. Find appropriate interval

Can't just compare error rates on single test set

Use instead $K$-fold cross validation and paired $t$-test

Each instance is a vector $x = [x_{1}, x_{2}, \ldots, x_{n}]$ of features

Supervised training, each instance has label, $y \in \{0, 1\}$

Output of model on input is $P(y = 1 | x)$

Weighted sum $z = wx+b$, where $w_{i}$ is feature weight, $b$ is the bias term

Final output is $P(y = 1 | x) = \sigma(z) = 1/(1 + e^{{-z}})$

Discrete valued prediction thresholded at 1/2

Called logistic/sigmoid function

Squashes $z$ into $[0, 1]$! (sometimes in older stuff called a "squashing function")

Assume classifier is trained with given values and biases. (5.1.1 from textbook)

• Compute weighted sum, run through logistic function, and compare to threshold

$w$ and $b$ must be discovered

Given labeled training corpus, map documents to feature vectors, and attach label

Initial $w$ and $b$ randomly!

Evaluate each training vector predicting $P(1|x)$ and use loss function $L$ to compare prediction to true label

stochastic gradient descent used to update $w$ and $b$, reducing $L$ on $x$

Repeat

Is perfect fit the best model?

Recall, the goal is to generalize well, there may be noise in data afterall

Perfect fits tend not to generalize well

Training accuracy vs simplicity

Want moth accuracy and simplicity

$h$ is a model, training error on $\mathcal{X}$, $error_{{\mathcal{X}}}(h)$ is sum of loss function over size of dataset

Generalization error, $error_{{\mathcal{D}}}(h) = \mathbb{E}_{{x \tilde \mathcal{D}}}[L(h(x), y)]$

Model overfits if there's an alternative model such that $error_{{\mathcal{x}}}(h) < error_{{\mathcal{X}}}(h^{{\prime}})$ and $error_{{\mathcal{D}}}(h) > error_{{\mathcal{D}}}(h^{{\prime}})$

Done with "Softmax", when $> 2$, one set of weights and one set bias for each class

Then, $z_{c} = w_{c} x + b_{c}$

Probability distribution calculated using softmax, not $\sigma$

• $\mathrm{softmax}(z_{c}) = \frac{exp(z_{c})}{\sum_{{c^{{\prime}} \in C}} exp(z_{c^{\prime}})}$

• Generalize cross-entropy to handle multiclass, y is generalized

• Labeled with one-hot vector

• When training, $\delta(-\log \hat{y}_{c^{*}})/\delta w_{i}$ for each weight, update with SGD

How do words relate?

Synonymy – similar meanings

Antonymy – opposing meanings

Similarity – based on what is similar, "car" and "bicycle"

Relatedness – not similar, relating to each other, "coffee", "cup"

Semantic field – covering some sort of semantic domain

Want to numerically represent words such that those that share some sort of lexical semantics cluster

Because regression models (and NNs) require numeric inputs!

Trained model that predicts positive when vanish appears should respond similarly to disappear

Representations grouping words by context

Two-dimensional projection of a 60-dim embedding (t-SNE)

Term-Document matrix, $|V| \times |D|$ matrix. Column $d$ is $d$'s BOW representation

Even bag of words representation clusters!

Rows of TD matrices can represent words

• Context then is the set of documents that it appears in and how often

• Similar words will have similar representation

Word-Word or Term-Context matrix can be used ($|V| \times |V|$)

• $(i,j)$ hold the number of times $i$ and $j$ appear in the same context

  • Context can be document, fixed window, etc.

Compute similarity of words/documents

Often, cosign of the angle \[\frac{v \cdot w}{\Vert v\Vert \Vert w \Vert} = \frac{\sum_{{i = 1}}^{N} v_{i} w_{i}}{\sqrt{\sum v_{i}^{2}}\sqrt{\sum w_{i}^{2}}}\]

Normalize vectors by dividing by length and take dot product of unit vectors

• 1 if identical, 0 if orthogonal

Must be careful about context definition

Some words appear frequently in any corpus

Can inflate similarity measures!

Weights terms by frequency, reducing weight if appearing in many documents

Term Frequency is the count of $t$ in $d$ +1 to avoid log zero \[\mathrm{tf}_{t,d} = \log_{10}(\mathrm{count}(t, d) + 1)\]

Log used to differentiate between orders of magnitude \[\mathrm{idf}_{t} = \log_{10}\left( \frac{N}{\mathrm{df_{}}_{t}}\right)\]

• $N$ is documents

• $\mathrm{df}_{t}$ nuber containing $t$

Weight of $t, d$ is $w_{{t,d}} = (\mathrm{tf}_{{t,d}})(\mathrm{idf}_{t})$

Word-word matrix then weight by tf-idf

Vector for term computed as normal

normalize/compute cosine similarity

From Mikolov et al.

vectors from tf-idf long, sparse

Dense, short vectors prefered

• Fewer params to learn!

• Captures synonymy better!

• Turns out, it works better in practice!

Movie Critic Analysis

• A form of sentiment analysis, but perhaps on a continuum

• Training a model to discriminate between professional and amateur critics

• IMDb and Stanford Sentiment Treebank datasets

Music Analysis

• Why are songs more popular than others

• Why things related to ranking happen

• Sentiment analysis on lyric

  • predictive from sentiment, etc.

• Datasets from Billboard charts

Professor Ratings

• Trends, correlation with language

• RateMyProfessor datasets, but most focus on correlation between numerical scores

Intent Classification

• Assigns document intent

• Useful for businesses to provide service, chatbots, automated assistants

Toxic Comment Classification

• More interactions online, but people are kinda toxic

• Understanding interaction is beneficial in making the internet a safer place

• Filtering social networking sites

• Understand how hate speech etc., spread through network sites

Sentiment Analysis

• Marketing use

• Employee satisfaction

• real-time information

• public opinion

• spam detection

Dialog Generation

• Goal-oriented

• Conversational systems

• Often a mix of the two

• Customer Service

• Chatbots

• Blended approach

• Pattern recognition, state machines, Markov processes, vector space, neural networks

Grammar Correction

• Spelling, grammar, punctuation errors

• Sequence tagging

• CNNs

• transformer-models

Taxonomy Translation

• Experts use special, domain ontologies for efficient communication, jargon

• Developing models that can be used to translate from experts to non-expert

• For a domain, learn an ontology for experts; then a general, for laymen; find connections between the two

Common Sense

• Reasoning and Validation in language

• Uses understanding and generation

• High accuracy, only text-to-text

• Applicable in many areas: taxonomic reasoning, recommendation systems, chatbots, MT

Speech processing

• Recognizing spoken language

Semantic Analysis

• Understanding meaning

Lemmatization

• De-inflecting words, returning to original form

Text Classification

• Categorization

Summarization

• Simplify document while communicating core message

• Not always generative

• Removing redundant information

• Abstracting/extracting

Sentiment Analysis

• Exactly what's been previously covered

• Aspect-based analysis, looks for multiple indicators/sentiments

Data to text generation

• auto-correct/text completion

• Various sorts of neural models, BOW, RNNs w/ Attention, etc

• Text to SQL, etc

Dialogue State Tracking

• Component of dialogue system

• Various ways of handling state

• Belief Tracking

POS tagging

• Solves other NLP problems

• Helps with language learning

• Meaning between homophones

• Lexical

• Rule-based

• Probabilistic

• Deep Learning

Missing Elements

• Filling things in

• Missing elements in foreign languages

• Fused head ident (find phrase missing noun; rule based or ML)

• Fused head resolution (finding in context)

• Bridging anaphora

• Bridging anaphora resolution

Takes weighted sum of input, $z$

Passes through non-linear activation function $f$

If $f$ is logistic, it's the same as logistic regression function

Single nodes are limited in what they represent

Linear Threshold Unit, the perceptron

• Activation thresholds weighted sum at 0

• Represents things like $\land$ well

• Other functions aren't representable!

• Because they're not linearly separable, XOR

• Would be a network of units

XOR, intersection of two separators

• $g_{1}(x) = 1x_{1} + 1x_{2} - 1/2$

• $g_{2}(x) = 1x_{1} + 1x_{2} - 3/2$

• So, $z__{}i = 0, g_{i} < 0$, $1$ otherwise

• Now, $g(z) = 1z_{1} - 2z_{2} - 1/2$

Remaps vectors $x$ to $z$ so that separable in new space

• Input layer, output layer

• hidden layer is $g_{1}$, $g_{2}$

• Two-layer perceptron/feed-forward neural network

• Non-linear activation in hidden output

• Many dozens of hidden layers, several output nodes

Activation functions: $z$ is weighted sum

• Linear unit

• Sigmoid/Logistic

• Softmax (generalizes sigmoid)

• Tanh

Hidden nodes

• Rectified Linear Unit, max 0, z

  • Good default

  • Variants to handle $<0$ differently

Training multiple layers means tuning parameters in all

Uses gradient descent minimizing a loss function

Cross-entropy is a common loss function

• $L_{{CE}}(\hat{y},y) = -\sum_{{c \in C}}y_{{c}}\log\hat{y}_{{c}} = -\log\hat{y}_{{c^{*}}}$

Must compute gradient of loss function for each param, even several layers from output

Backprop feeds forward inputs to outputs, propagates back error via chain rule for derivatives

• Simply, modularly decomposed

Recall language model can take input sequence, and predict probability over next words

Feed-forward does the same, takes input sequence, outputs probabilities

word2vec embeddings used

Pre-trained (faster) or trained with network (may be better)

Can fine-tune a pre-trained embedding to save time

Fixed word sequence input

Softmax output compared to target word

Trained with cross entropy loss

Fixed word sequence input

• Embedding learned with model

• Softmax compared to output

• trained with cross-entropy loss

Neural units take weighted sum, and applies non-linear activation

Multilayer networks used because single unit is limited

hidden layers allow learning of own representations – nonlinear feature extraction

Stochastic GD used for training

• Calculated using back prop

Neural models predict word after sequence of words

• Can use pre-trained embeddings or learn their own.

Recurrent cells have connections going backward as well as forward

• At $t$, neuron receives input $x_{t}$ and its output $y_{{t - 1}}$

Recurrent layer can be built, where each node gets the vector $x_{t}$ and the vector $y_{{t - 1}}$ (that is, the whole layer's output)

Output depends on past, that is, memory/state

• State at $t$ is $h_{t} = g(h_{{t - 1}}, x_{{t}})$, and output is $y_{t} = f(h_{{t - 1}}, x_{t})$

• $h_{t}$ then stores important information about input

• Forward pass – evaluate forward by unrolling in time

Can stack RNN layers on each other

bidirectional – process separately from end towards start

• run in opposite directions, outputs combine in some way (concat, add, etc)

• Can be used for sequence classification

Long-term memory – vanishing & exploding gradients can be problematic

Can suffer from long times with long sequences

First inputs have diminishing impact

Add "long-term" memory

Accumulates information, and can decide to forget

Character level modeling sometimes used

• Can be helpful to have more detailed information

• Can use other RNNs to process at finer level and combine with word-level embeddings

Can be done using a birnn, and concatenating input, giving a character-level word-embedding

NLM can generate text sequence

Must feed start symbol

• $y_{{t}} = f(h_{t})$, $h_{t} = g(h_{t - 1}, x_{t})$, $x_{t} = y_{t - 1}$ (this is an autoregressive model)

Can generalize from a prefix (NLM completes existing sequence)

• Prefix input triggers hidden state vectors, outputs from prefix ignored

• hidden state captures context

Can be used for MT

• NLM trained on bitexts – sentence concatenated with translation

• Feed sentence 1 then generate translation

Previous model generalizes to encoder-decoder

Encoder input generates states

Context vector is a function of hidden states, the embedded representations

Decoder takes $c$ and creates new states and outputs

Encoder can be any RNN, stacked biLSTM, etc (same with decoder)

• c is final step, top layer encoder concatenated

• Initialise decoder's state = c

• Decoder state is function depending on output, previous state, context

Context vectors passed from encoder to decoder are embedded representation of input sequence

• $c$ represents the entire input

• Can't ensure context vector is sufficiently large to represent context!

• Can use attention mechanism

  • context becomes sequence of vectors

  • Decoder focuses on relevant subset

Static context vector becomes sequence of context fectors

• Uses score function, $score(h_{{i - 1}}^{d}, h_{j}^{e})$ gives relevance of encoder state to decoder state, likely uses some weight function (bilinear interpolation, can be learned!)

• Concatenate scores over j into relevance vector

• Normalize with softmax, and sum over j by encoder states

Architecture

• Compute context vector at each step

Attention $\alpha_{{ij}}$ shows probability that output $y_{i}$ is aligned to a given input $x_{j}$

Named entities are anything refered to by a proper name

Requires identifying what text is the entity (using a span of text), and then classifying it

• Can classify as people, organizations, locations, geo-political entities, facilities, etc.

IO/IOB tagging

• Inside, outside, inside/outside/beginning tags

Comes after NER

Relations depend on domain

Classes exist for each entity

Then must learn to classify and connect relations

Ordered tuples, many different forms of notation

Relations depend on domain, and many have specific models

Similar to NER, can differentiate between absolute, relative and durations

Rule-based, sequence labeling

Normalized with respect to specific date for calculation/comparison

ISO standard provides XML tags

Ident events in text, assignable to points in time

IOB tagging via supervised learning

Tends to be built on verbs

Features depend on many things

Timing information can provide temporal ordering, using normal relation extraction

Time-bank corpus can be used

Some situations/events have standard information

Templates represent these, with slots to fill

Recognize templates, then extract and fill slots.

Given question, find short snippet from a collection

Process question for IR to analyze answers

• Identify relevant documents

• Extract passages

• Filter to identify relevant passages

Must also do answer-type detection, decides what the answer will look like in the end

Question Processing

• Identifies a query, and other information

• Identifies focus

• Identifies question type

• Extracts tokens to submit

  • query is tf-idf and cosine similarity, can be enhanced with bigrams

  • If documents are limited, query expansion can be used (adding terms with a thesaurus, varied suffixes)

• Answer types look at Named Entity types, filter IR on that

  • Can use taxonomy of types (WordNet)

  • Classifier trained, uses embeddings of words, etc

Passages come from ranked set

• identify relevant paragraphs/sentences

• partition documents into passages

• NER used for filtering

• Rank with learned model

Answers extracted with Span labeling

• Return matching named entity if answer typue is specified

• Problem hen answer type is unavailable/non-existent (consider definition questions)

  • Use question keywords matching

  • words not in query

  • Length of sequence of keywords in candidate

  • hand-crafted patters matching candidate answers

• Neural Methods

  • DNN, uses SQuAD; computes embeddings for questions and embeddings for tokens

  • Compare embeddings, uses reading cqmprehension tasks

  • SQuAD dataset

  • Question $q$, passage $p$, $p_{s}(i)$ is probability that $p_{i}$ starts answer $p_{e}(i)$ is probability that $p_{i}$ ends answer span

  • Similarity scores used to predict start/end of answer span

  • Bi-LSTMs with attention, for question and passage; similarity scores used

  • $E$ is glove embedding

  • Input sequence of embeddings to get states

  • Combine into a query vector, being a weighted softmax sum, with $b$ and $w$ being learned weights

  • Passage embedded to vector sequence compared to $Q$

    • Computed similarly,using GloVE and PoS/NER tokens

  • Align attention between question and passage

Answers in structured DB, no need for IR

Must be able to create a query

Requires Semantic Parsing to form SQL or predicate Calculus

Can be done using hand-written features or learned

Supervised

• Dependency parsing, has things like tmod, nsubj

• Uses labeled examples (question-query pairs)

• learn general rules from inductive logic programming

Semi-suprevised

• distant supervision/open information extraction

• REVERB is an Open extractor

• Extracts triples of strings and aligns to things like FreeBase (or other structured relations)

• Match subject and argument to relations

• Then learn mapping between natural language and FreeBase

IBM Watson!

Clinical Decision Support

Uses both methods

Many candidates proposed and scored by evidence

Stages

• Question Processing

• Answer generation

• Answer Scoring

• Confidence merging/ranking

Parsing, NER, relation extraction

Focus, answer type, question classification

Generate answers from structure/text sources

Queries using extracted relations

Eliminates stopwords, upweights terms from focus

Generates score vector from type, time/space relations, supporting evidence

Merge based on equivalence/combine scoring features

Mean Reciprocal Rank

• Assume questions answers provided

• If system under test returns list of answers, score is reciprocal of rank of first correct

• Rank is rank of highest correct answer

Exact Match – fraction of predicted correct answers

$F_{1}$ score – harmonic mean of precision and recall

Datasets like SQuAD, TREX QA, TriviaQA, WikiQA

Single Document

Headline/outline brief extract

A group of related documents

News stories on an event, etc.

Summary in response to particular queries

QA but for non-factoid QA

Many documents potentially

Overlap with query, cosine similarity

Specialized systems for query type

Information Extraction to identify important information

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) N-gram overlap between generated summaries and ground-truths

Gross equation (slide 20)

For each bigram, how many times did generated summary use it?

Normalized with total bigrams in reference

A recall measure

Rouge-L measures longest-common subsequence

Rouge-SU measures skip bigrams (additional words between pairs)

Pyramid method/summary content units focus on units of meaning more relevant

Direct – word-by-word through input text

Transfer – parse, translate structure, generate output

Interlingua – parse input, translate to intermediate representation, generate output

Vauquois Triangle captures approaches

A "noise channel model"

• Probabilistic, outputs most probable translation

• Given sentence $F$, $P(E|F)$ is probability output sentence is correct translation

• Search for argmax $E$, by Bayes's rule

• Probability of sentence $P(E)$ is from language model

• $P(F|E)$ is from translation model

• A decoder is used to output $E$

• $F$ is a noisy translated $E$, and the goal is to recover $E$

Human-based: people evaluate based on fluency, fidelity, how much is necessary to "fix" output

BLEU

• Given output and reference, measure closeness

• Weighted average of $N$-gram overlaps with reference translations

• Modified precision, appearance based on max appearance in reference

• $N$ grams for $N$ in 1–4

• Brevity Penalty also used, based on length for shorter outputs

• Easy to calculate, widely used

• Doesn't consider meaning

• Doesn't handle syntax

• Not as good as human judgement

Can refer

• indefinite noun phrases

• definite noun phrases

• pronouns

• demonstratives

• non-stated pronouns

• names

Information status

• new/old

• New could be new or old to hearer (if part of common knowledge)

• Old were introduced previously

• Some are inferred, based on known properties

Non referring expressions exist

• Appositives

• Predictives

• Expletives

• Generics

Detect and cluster mentions

OntoNotes and ARRAU are common datasets

IDentify spans of text for each mention, retrieves candidates and filters, all N grams <= 10

Filter extraction based on noun phrases, posessives, named entities, etc

Entity-based represents each entity in a discourse model

Mention-based consider mentions independently

Models can also use ranking to compare antecedents

Mention-pair takes pair of mentions as input, outputs binary classification of coreference

• hand-coded, or neural network

• Each pair used for training

• Filter negative, create pair of coreferants, create positive, create negative training for each for some k

• Cluster mentions by pair top, and take transitive closure

Mention rank requires integration of resolution with anaphoricity by adding null

• Estimate probabilities

• At test time, pair top and take TC

Entity-based

• Links mentions to discourse entities/mention clusters

• Can use mention ranking approach, adding features

• Or trains RNN to process clusters and combine hidden states

Handcoded features

• Words/embeddings

• Role

• Length

• Matching strings, etc

Span is contiguous sequence of $\leq 10$ words

For a span, estimate probability that each previous span is it's antecedent

• $P(Y_{i}) = \frac{exp(s(i,y_{i}))}{\sum_{{y' \in Y(i)}}exp(s(i,y^{{\prime}}))}$

• $s(i,j) = m(i) + m(j) + c(i,j)$

• $m$ measures span is mention, $c$ measures if span is antecedent

  • $m$ is a feed-forward neural network

  • similar for $c$

Feed-forward NNs take input of vector representation (bi-LSTM w/ attention output)

Train my maximizing sum of probabilities of pairs in ground-truth

• Minimizing cross-entropy

Precision and recall and combination, $F$-measure

$H$ is hypotheses, $R$ is references

• Precision: $|H \cap R|/|H|$

• Recall: $|H \cap R|/|R|$

• $F$-measure: $2(precison)(recall)/(precision+recall)

• MUC $F$-measure, H, R are links, pairs of mentions

• $B^{3}$ are individual mentions

Connected to coreference resolution, connects real-world to ontology

Detect mentions, disambiguate with supervised learning

They can help each other

Kind of a sentence completion

Want to connect ambiguous references

Integrating world knowledge can assist in these issues

Winograd Schema Challenge include coreference problems solved with world knowledge but no automated methods

Somewhat solved with huge models

Biases exist

Which pronoun connects more frequently with certain professions

Biased datasets

Gender-swapped dataset combined with original can help reduce

WinoBias, GAP datasets available

Discrete representation of aspect of meaning

Discrete valued senses distinct from continuous embedding

Relationships between senses exist and can enhance semantics

Independent truth conditions, different syntactic behavior, and independent relationships? Do they exhibit antagonistic meanings

Zeugma is used for testing: "serve"

Synonyms

Antonyms

Taxonomic, IS-A

Meronymous, PART-OF

Structured relationships, author/works-of-author

150k+ words, nouns, verbs, adjectives, adverbs

Each has definition, synonyms, usage examples, etc

Synonym sets (synsets), of senses, based on senses, not words

Supersenses, categories of senses

Sense relations

• Hypernyms, hyponyms

• Instance hyper-/hyponyms

• Meronyms/part meronyms

Can form a graph structure with labels on edges

Chooses correct sense based on context

WordNet, application-specific dictionary, translations, etc.

Semantic Concordance – corpus with words labeled with sense

SemCor labels Brown corpus with WordNet

Can map from wikipedia/wikidata/wiktionary

Evaluate with $F_{1} =2PR/(P+R)$ precision/recall based

Baseline is most-frequent sense

Unsupervised

Induce set rather than using pre-defined

For each word in vocab

• For each instance in corpus

• Compute context vector

• Cluster these vectors (each is a sense of $w$)

• Centroid/median vector is a sense vector for each sense $j$ of $w$

• When processing, use nearest sense vector