[D] Is attention just a type of belief propagation?
Attention is mostly discussed in the context of sequences, but I was thinking about what it meant in the context of message passing. Clearly when a certain word has "attention" it is relaying long-distance information, but is that synonymous with something like belief propagation?
Posted Aug 8, 2019, 5:09 PM UTC
33
Attention is closer related to dynamic pooling where we enable a network to learn how much of a given member of a sequence should be represented and then collects a weighted sum of the all the time steps so you got a layer thats dynamically pooled based on the overarching context