naacl 2019 https://github.com/wabyking/qnn https://mp.weixin.qq.com/s/NbLnGQD4TjlbMGOXbfa8pg
TREC-CAR task:
query is wikipedia title concatenated with section titles, answer is the wikipedia page text
query_0 -> multi reformulator -> search results -> aggregator
use RL to train reformulator agents (state is documents, reward r is )
aggragator as meta-agent
multiple word senses reside in linear superposition within the wordembedding and simple sparse coding can re-cover vectors that approximately capture the senses.
polysemy vector is linear combination of its monosemous vectors
linearity Assertion linear structure appears out of a highly nonlinearembedding
Word Sense Induction via sparse coding
random walk on discourses
random walk across micro-topics(discourse)
there exists a linear relationship between the vector of aword and the vectors of the words in its contexts
tree-to-tree transduction method for sentence compression
entity link: https://spaces.ac.cn/archives/6919
https://github.com/facebookresearch/MLDoc
DRCD https://arxiv.org/pdf/1806.00920.pdf
fever Fact Extraction and VERification
LAMBADA dataset long dependency, predict last word
XNLI
SNLI, MNLI
babi
wikitext-2 ‘train’: “https://s3.amazonaws.com/datasets.huggingface.co/wikitext-2/train.txt” ‘valid’: “https://s3.amazonaws.com/datasets.huggingface.co/wikitext-2/valid.txt”
wikitext-103 ‘train’: “https://s3.amazonaws.com/datasets.huggingface.co/wikitext-103/wiki.train.tokens” ‘valid’: “https://s3.amazonaws.com/datasets.huggingface.co/wikitext-103/wiki.valid.tokens”
conala, Code/Natural Language Challenge
IMDB
Yelp-2, Yelp-5
DBpedia
AG
Amazon-2, Amazon-5
common QA
300 average length
cloze-style
span based
SQUAD unanswerable questions
NEWSQA
MSMARCO
exam based
[AI2 Elementary SchoolScience Questions] 1080 questions
[NTCIR QA Lab] university entrance
[CLEF QA Track]
WSCR
Pairwise Relations
- arc classification
- arc prediction(binary)
Preposition supersense disambiguation
STREUSLE 4.0 corpus
semantic tagging
event factuality
Universal Decompositional Semantics It Happened v2
Grammatical error detection
https://ilexir.co.uk/datasets/index.html
conjunct identification
https://github.com/Jess1ca/CoordinationExtPTB
Document-level Generation and Translation
Dependency parsing
head and dependent
the structure must be connected, have a designated root node, and be acyclic or planar.
each word has a single head.
there is a single root node from which one can follow a unique directed path to each of the words in the sentence.
projective, if there is a path from the head to every word that lies between the head and the dependent in the sentence.
a tree is projective, if it can be drawn with no crossing edges.
non-projective, seen in flexible langauages.
Translation, text, audio
Graph based dependency parsing
score the tree with edge weights
using maximum spanning tree, but might have cycles
to eliminate cycles, Chu, Liu, Edmonds(1967):
for each vertex, incoming edge with max weight is chosen
subtract each incoming edges with max incoming weight.
select a cycle and collapse it into a single new node.
higher order dependency parsing
Zhang and McDonald 2012
https://pair-code.github.io/interpretability/bert-tree/
𝑛 vectors completely at random from a unit Gaussian distribution in ℝ𝑚. If 𝑚≫𝑛, with high probability the result would be an approximate Pythagorean embedding.
The reason is that in high dimensions, (1) vectors drawn from a unit Gaussian distribution have length very close to 1 with high probability; and (2) when 𝑚≫𝑛, a set of 𝑛 unit Gaussian vectors will likely be close to mutually orthogonal.
Initialize with a completely random tree embedding, and in addition pick a special random vector for each vertex; then at each step, move each child node so that it is closer to its parent's location plus the child's special vector
> [A Structural Probe for Finding Syntax in Word Representations](https://nlp.stanford.edu/pubs/hewitt2019structural.pdf)
https://github.com/john-hewitt/structural-probes
https://github.com/rsennrich/subword-nmt
https://github.com/deepmind/sonnet/blob/master/docs/README.md
The idea in Gibbs sampling is to generate posterior samplesby sweeping through each variable (or block of variables) to sample from its conditionaldistribution with the remaining variables fixed to their current values.
dot product
bilinear
multi-layer perceptron
namesake (Markov 1906)
Sutskever et el. 2011
Mikolov el al. 2012, Sennrich el al. 2015
char and word level (Kang el al. 2011) (Ling el al. 2015) (Kim el al. 2016)
unsupervised morphology with log-bilinear model (Botha and Blunsom 2014)(Jozefowicz el al. 2016)
Brown el al. 1992
Chung el al. 2016
Luong and Manning 2016
combine count based and neural lm (Neubig Dyer 2016)
char based and word based to translate text and code (Ling el al 2016)
copy when UNK (Merity et al. 2016)
Phonetics and phonology
Morphology
Syntax
Lexical semantics
Compositional semantics
Pragmatics
Discourse
andreas
dataset: room to room
parsing technique
type 1
type 2
type 3
type 4
heavy NP Shift
verb + NP + PP(temporal adjunct), if NP is heavy enough, PP will be shifted after verb
hrasal verbs and particle shift
give the habit up / give up the habit
dative alternation
give the woman the object / give the object to the woman
double object / propositional object
Genitive altenation
the woman’s house / the house of the woman
one change to the rule: choose rightarc if it produces a correct relation given reference parsing, and, all dependents of the top word in stack have been assigned.
combine simples features, use feature template: location.property
clausal relation, that describe syntatic roles with respect to predicate(often a verb)
modifier relations, that categorize the ways that words can modify theirs heads.
| Clausal Argument Relations | Description |
|---|---|
| NSUBJ | Nominal subject |
| DOBJ | Direct Object |
| IOBJ | Indirect Object |
| CCOMP | clausal complement |
| XCOMP | open clausal complement |
| Nominal Modifier Relations | Description |
|---|---|
| NMOD | Nominal Modifier |
| AMOD | Adjective modifier |
| NUMMOD | Numeric modifier |
| APPOS | appositional modifier |
| DET | determiner |
| CASE | Prepositions, postpositions and other case markers |
| others | Description |
|---|---|
| CONJ | Conjunct |
| CC | Coordinationg conjunction |
https://cds.nyu.edu/text-data-speaker-series/
https://levyomer.wordpress.com/publications/
https://kevincrawfordknight.github.io/
https://scholar.google.com/citations?user=QW_JZnsAAAAJ&hl=en
https://arxiv.org/pdf/1608.00778.pdf
http://tisjune.github.io/research/
https://www.cs.washington.edu/people/faculty/lsz/publications-by-year
http://www.phontron.com/
http://benjaminlauderdale.net/
Institute for the Study of the Ancient World the ISAW Library NYU Center for Humanities NYU Division of Libraries NYU Center for Ancient Studies and NYU Department of Classics
https://diyclassics.github.io/future-philologies/
http://kyle-p-johnson.com/
https://mimno.infosci.cornell.edu/
http://www.ccs.neu.edu/home/dasmith/
Neural Network Acceptability Judgments
includes generative grammars of the type described by Chomsky (1957)
isolate a violation, so tend to be unacceptable for a single identifiable reason.
the artificial learner must not be ex-posed to any knowledge of language that could not plausibly be part of the input to a human learner
100k most frequent words in the British National Corpus
in-domain: training set (8551 examples), a development set (527), and a test set (530), 17 sources
out-domain development set (516) and a test set (533), 6 sources
human 86.1%
aggregate rating, yielding an average agreement of 93%
The encoders are trained on 100- 200 million tokens, which is within a factor of ten of the number of tokens human learners are ex- posed to during language acquisition
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
subject-verb number agreement
highly sensitive to recent but irrelevant nouns
agreement attractors
one-hot encoding input lstm 50 units: 0.83% errors
nouns-only baseline: 4.5%
experiments:
distance between nouns and verbs
attractor: last intervening nouns
count of attractors
relative clauses
word representation learned the concept of singular and plural from scratch
word-by-word activation visualization
training obj
verb inflection: have verb’s singular form before verb(give cue to syntactic clause boundary
grammaticality judge: (human rarely receive ungrammatical signals
LM
training only on hard dataset(with attractors)
Agreement attraction errors in humans are much more common when the attractor is plural than when it is singular
errors:
noun-noun compounds
verbs similar to plural nouns are mistaken as nouns: The ship that the player drives has a very high speed.
limitation of the number prediction task, which jointly evaluates the model’s ability to identify the subject and its ability to assign the correct number to noun phrases
Negative licensor: not none
Negative Polarity Item: any ever
filler-gap dependency
RNNG
word-synchronus beam search (Stern et al., 2017)
Wh-Licensing Interaction
Poverty of theStimulus Argument.
purely data-driven learning is not powerful enough to explain the richness and uniformity of human grammars
flashtext
https://github.com/marcocor/tagme-python
knowledge embedding TransE
masked language model, next sent prediction as pre-training obj
wit.ai