Proposal: In-browser AI

Proposals Round 8
#1

Name of Project: In-browser AI

Proposal in one sentence: Run modern deep learning models in a browser in two lines of code with no domain knowledge.

Description of the project and what problem it is solving: Today, using state-of-the-art neural networks in the browser requires either deploying them to your infrastructure and exposing endpoints or using the models via API (e.g., OpenAI, AssemblyAI). The first option is complex and expensive. The second option involves sending your and your users’ data to a third party, which is sometimes not desirable.

An alternative for that is running the models on the client. There is TensorFlowJS, but it requires a lot of domain knowledge to run the models, and the users are limited by the models supported by the project’s developers. Of course, you can implement a new model for TF.js, but it is way too much unnecessary work for a frontend developer.

I propose a new approach for running neural networks in a browser by combining the powers of ONNX runtime for Web and Hugging Face model hub. I aim to make any (reasonably-sized) model from the hub available to the JavaScript developers. I am developing an NPM library that uses ONNX Web to run models ported from the Hugging Face hub using original config files.

Why using original config files is essential? It makes minimizes the effort of adding new models. You can just convert the original model from PyTorch or Tensorflow to ONNX using built-in functionality, and you are ready to plug it into your JS application!

Another crucial aspect is the ease of use. I design the library in a way that the developer will be able to process text or images with only two lines of code:

const model = await ImagetModel.create("cool-segmentation-model");
const result = await model.process(inputImage);

The library has a lot of potential applications - from standard web applications to hybrid desktop and mobile apps. In the future, I will also train more models to make the library more useful.

At the moment, there are multiple demos that are fully operational. However, the library itself still needs a lot of work to be prepared for use.

Grant Deliverables:

  • Open-source NPM library with the source code published on GitHub.
  • An initial set of models for computer vision and natural language processing.

Squad

Squad Lead:

2 Likes
#2

That would be a competitor to TensorFlow.js?

#3

No. I would call ONNX for Web is a competitor to TF.js. In-browser AI will be built on top of the ONNX for Web. The reason to use ONNX instead of TF.js is the more streamlined export from Hugging Face and the ability to support models created in PyTorch (~70% of the models on the hub).

1 Like
#5
#6

Thank you for the link. I might check this library in the future. But it seems like it is not very actively developed.

#7

Here is the video with the presentation of the results.

Project repo - GitHub - visheratin/web-ai: Run modern deep learning models in the browser.

1 Like
#8

Demo website - https://web-ai-demo.vercel.app/

1 Like