Pre-Translation for Neural Machine Translation

Size: px

Start display at page:

Download "Pre-Translation for Neural Machine Translation"

Kenneth Banks
5 years ago
Views:

1 Pre-Translation for Neural Machine Translation Jan Niehues, Eunah Cho, Thanh-Le Ha and Alex Waibel KIT - Institute for Anthropomatics and Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association

2 Mixed Input Implementation: Join source sentence and PBMT translation the goalie der Torwart RNN state encode source and PBMT translation Language specific word embeddings E_the E_goalie D_der D_Torwart BPE for word encoding E_the E_go E_al E_ie D_der D_Tor D_wart Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

3 Result by Word Frequency Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

4 Alignment Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

5 Pre-Translation for Neural Machine Translation Jan Niehues, Eunah Cho, Thanh-Le Ha and Alex Waibel KIT - Institute for Anthropomatics and Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association

6 Motivation Neural machine translation sets state-of-the art End-to-End neural network approach to machine translation Comparison to SMT Significant improvements Automatic metrics Manual evaluation More fluent translation Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

7 Motivation NMT has different problems Small vocabulary Problems translating rare words English: NMT: NMT(gloss): the goalie parried der Gott the god Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

8 Motivation NMT has different problems Small vocabulary Problems translating rare words English: NMT: NMT(gloss): the goalie parried der Gott the god Combine SMT and NMT Simplify the task of NMT Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

9 Outline Motivation MT approaches Idea Pipeline Mixed Input Evaluation Conclusion Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

10 Statistical Machine Translation (SMT) Build translations from blocks of source and target words (phrase pairs) Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

11 Statistical Machine Translation (SMT) Build translations from blocks of source and target words (phrase pairs) Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

12 Statistical Machine Translation (SMT) Build translations from blocks of source and target words (phrase pairs) Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

13 Statistical Machine Translation (SMT) Build translations from blocks of source and target words (phrase pairs) Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

14 Neural Machine Translation (NMT) Neural network to predict most probably target sequence Jointly train model Large improvements in translation quality Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

15 Neural Machine Translation (NMT) Fixed vocabulary size Byte pair encoding (Sennrich et al. 2016) Represent all words with n sub-words Start with character representation Join most common bi-gram sequence to new symbol Exampel: the_goalie_parried Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

16 Neural Machine Translation (NMT) Fixed vocabulary size Byte pair encoding (Sennrich et al. 2016) Represent all words with n sub-words Start with character representation Join most common bi-gram sequence to new symbol Exampel: t h e _ g o a l ie _ p a r r ie d Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

17 Neural Machine Translation (NMT) Fixed vocabulary size Byte pair encoding (Sennrich et al. 2016) Represent all words with n sub-words Start with character representation Join most common bi-gram sequence to new symbol Exampel: t h e _ g o a l ie _ p a r r ied Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

18 Neural Machine Translation (NMT) Fixed vocabulary size Byte pair encoding (Sennrich et al. 2016) Represent all words with n sub-words Start with character representation Join most common bi-gram sequence to new symbol Exampel: t h e _ g o a l ie _ pa r r ied Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

19 Neural Machine Translation (NMT) Fixed vocabulary size Byte pair encoding (Sennrich et al. 2016) Represent all words with n sub-words Start with character representation Join most common bi-gram sequence to new symbol Exampel: the _ go al ie _ par ried Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

20 Difference SMT/NMT SMT: Handle large vocabulary Easily extensible Add translation via new phrase pairs NMT: Joint model Long context Better generalization due to word embeddings Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

21 Pre-Translation Combine advantages of both approaches Facilitate advantages of SMT Successful combination of other approaches Idea: Use SMT as input to NMT Encode words using Byte pair encoding Use translation of words not in NMT vocabulary Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

22 Related Work Combination of SMT and Rule-based MT (Dugast et al., 2007, Simard et al, 2007) Automatic Post editing (Junczyd-Dowmunt and Grundkiewicz, 2016) Preprocessing for PBMT Compound splitting Pre-reordering Handling of rare words in NMT (Luong et al 2014, Sennrich et al, 2015) Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

23 Pipeline Input: Source sentence Translate using PBMT Translate from PBMT German to German using NMT Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

24 Pipeline Input: Source sentence Translate using PBMT Translate from PBMT German to German using NMT Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

Pipeline Input: Source sentence Translate using PBMT Translate from PBMT German to German using NMT 10

25 Pipeline Input: Source sentence Translate using PBMT Translate from PBMT German to German using NMT Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

26 Pipeline Input: Source sentence Translate using PBMT Translate from PBMT German to German using NMT Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

27 Mixed Input Input: Source sentence Translate using PBMT Combine source and PBMT Translation Translate joined text using NMT Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

28 Mixed Input Implementation: Join source sentence and PBMT translation the goalie der Torwart RNN state encode source and PBMT translation Language specific word embeddings E_the E_goalie D_der D_Torwart BPE for word encoding E_the E_go E_al E_ie D_der D_Tor D_wart Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

29 Training Training data: Parallel corpus PBMT translation of corpus Problem: PBMT tends to overfit on the training data Filter singletons from phrase table Successful used in other models Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

30 Experiments Training data: WMT EN-DE Data PBMT In-house translation system NMT Nematus BPE with 40K operations Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

31 Results English - German System Dev/Valid Test tst2014 tst2015 tst2016 NMT NMT Ensemble PBMT Advanced PBMT Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

32 Results English - German System Dev/Valid Test tst2014 tst2015 tst2016 NMT NMT Ensemble PBMT Advanced PBMT Pipeline Pipeline Advanced Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

33 Results English - German System Dev/Valid Test tst2014 tst2015 tst2016 NMT NMT Ensemble PBMT Advanced PBMT Pipeline Pipeline Advanced Mix Mix Advanced Mix Advanced Ensemble Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

34 Result by Word Frequency Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

35 Examples English: PBMT: NMT: Pre: Pre(gloss): Then with a shot which the goalie parried with his knee in the 35th minute. Dann mit einem Schuss, die der Torwart pariert mit seinem Knie in der 35. Minute. Dann mit einem Schuss, den der Gott mit seinem Knie in der 35. Minute. Dann mit einem Schuss, das der Torwart mit seinem Knie in der 35. Minute pariert. Then with a shoot, that the goalie with his knee in the 35th minute parried Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

36 Examples English: PBMT: NMT: Pre: Pre (gloss):... a riot in the stadium.... einen Aufruhr im Stadion.... einen Riot im Stadion.... einen Aufruhr im Station.... a riot in_the stadium Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

37 Alignment Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

38 Conclusion Combine advantages of NMT and SMT Improve handling of rare words Easy handling different input streams Increase overall translation performance Further work: Do we need to do a full translation? Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

39 Thanks Jan Niehues - Pre-Translation for Neural Machine Translation KIT - Institute for Anthropomatics and

Machine Translation and Advanced Topics on LSTMs

Machine Translation and Advanced Topics on LSTMs COSC 7336: Advanced Natural Language Processing Fall 2017 Some content on these slides was borrowed from Riloff, Money, and Socher and Manning. Announcements