Name of Project: Client-side (offline-first) conversational (multilingual) Machine Translation
Proposal in one sentence: Be able to use conversation mode with (multilingual) machine translation models in offline mode.
Description of the project and what problem it is solving:
Client side offline-first ML has really picked up pace off-late, due to various reasons such as more and more users realising importance of privacy, better security and also better reliability/availability of models. While many solutions exist for deploying smaller models client-side such as tensorflow.js, onnyx for web etc - unfortunately they are still limited to using truly deeper and bigger ML models client-side. I would like to target one such popular usecase which we use widely: offline-first Machine Translation (MT). MT has a wide applicability in our daily workflows, be it while browsing or whenever we travel to foreign countries which use a different language of communication than our native one’s to name a few.
There have been quite a lot of recent SoTA advancements in the sequence-to-sequence modelling space for MT such as mBART etc, but it remains to be seen if they can be utilised offline in a user friendly way to be a genuine alternative to other privacy-intrusive services such as Google translate etc. By virtue of this grant, I plan to explore the same. A super cool feature that I’m planning for this grant is by stitching a robust deep ML speech recognition system (such as OpenAI Whisper) with a translation model for a hands-free translation experience in a good UX. Just say the sentence, and poof you have the translation - super useful when you want to converse with people in another country! 
**Grant Deliverables:**choose deliverables you can complete in a month’s worth of part-time work
- Exploration of breadth of models that can be utilised specifically for client-side MT (with constraints of latency & memory) and speech recognition.
- A tool to achieve support for offline-first conversational (multilingual) MT.
Brief description of how I will approach the project:
First, I will do a survey of the best possible models that can be leveraged given the constraints of offline MT. I will evaluate during this phase, if any existing models can be used out of the box, or if model compression/distillation of any sort is needed. This will tentatively take about couple of weeks. Post that until the end of the grant, I will move on-to building the tool with the best possible UX and constraints. I will evaluate what best libraries to use here again with constraints of offline-first MT, and the framework to build the tool that I have in mind (as proposed above) as well.
Squad
Squad Lead:
- Discord: restandvest#9326