NIST Machine Translation Workshop
I was in DC last week to attend the follow-up workshop on the shared task that consumed the better part of my first 2 months this year. It is quite humbling to go to such a conference. A lot of people in the room had contributed something very significant to the area of MT in last 10 years and it is a pleasure to see them discussing things with each other and talk about the direction of the field.
The workshop had a rather competitive tinge to it with Google, Microsoft, BBN, IBM and other companies being there apart from many universities and everybody trying to win the bragging rights for next one year at least. What was funny however that most of the people doing MT at these places are relatively young and so most of them have worked together and published joint papers in last 4-5 years. I can count at least one joint paper between most pairs of companies. There were many advisor student pairs that are now in different competing companies. It is a nice little well knit community.
One depressing thing from my point of view was that most MT developers didn't seem too enthusiastic about newer methods of MT Evaluation something on which I have been working for part two years. To give a little background, MT Evaluation tries to build automatic metrics that can tell you how good a MT system is by comparing the output of the system with a gold standard translation done by humans. The field was pioneered by IBM and their paper of 2002 remains a de facto standard till date. There have been tons of papers on the topic and many short comings of IBM metric have been shown but nothing has been able to replace it. Primary reason seems to be that there are known techniques to tune the MT systems towards that metric that are easy, stable and fast and no MT developer seems willing to let go of that luxury. NIST is starting a evaluation campaign for MT Evaluation Metrics this year (I know it is getting ridiculous) and hope is to systematically compare existing metrics and develop better ones. The workshop for this year is in Hawaii but by that time I would be back in India. Too bad :(
Let me end by saying that for languages like Arabic, MT systems are actually getting very good and usable. You should definitely give them a try if you have not yet ! Google's online systems should give you a taste of state of the art pretty much.
The workshop had a rather competitive tinge to it with Google, Microsoft, BBN, IBM and other companies being there apart from many universities and everybody trying to win the bragging rights for next one year at least. What was funny however that most of the people doing MT at these places are relatively young and so most of them have worked together and published joint papers in last 4-5 years. I can count at least one joint paper between most pairs of companies. There were many advisor student pairs that are now in different competing companies. It is a nice little well knit community.
One depressing thing from my point of view was that most MT developers didn't seem too enthusiastic about newer methods of MT Evaluation something on which I have been working for part two years. To give a little background, MT Evaluation tries to build automatic metrics that can tell you how good a MT system is by comparing the output of the system with a gold standard translation done by humans. The field was pioneered by IBM and their paper of 2002 remains a de facto standard till date. There have been tons of papers on the topic and many short comings of IBM metric have been shown but nothing has been able to replace it. Primary reason seems to be that there are known techniques to tune the MT systems towards that metric that are easy, stable and fast and no MT developer seems willing to let go of that luxury. NIST is starting a evaluation campaign for MT Evaluation Metrics this year (I know it is getting ridiculous) and hope is to systematically compare existing metrics and develop better ones. The workshop for this year is in Hawaii but by that time I would be back in India. Too bad :(
Let me end by saying that for languages like Arabic, MT systems are actually getting very good and usable. You should definitely give them a try if you have not yet ! Google's online systems should give you a taste of state of the art pretty much.
Comments