For inference, models should be compiled down directly to WASM or WebGPU or whatever, right? The driving language really shouldn't matter at the end of the day.
Unless you've got massive compute bound transformers or old-school full convolutions, if you're interpreting a list of operations you're going to lose perf.
> For inference, models should be compiled down directly to WASM or WebGPU
I mean, yes? Whether for inference or for training, ML operators are usually implemented in compiled low level languages. That implementation might also take advantage of GPUs.
> ... if you're going interpreting a list of operations ...
I'm not sure what "interpreting a list of operators" means here.
To clarify why I mentioned the implementation language, I recently had to modify the source code of Tract to get my model running (I will push that change upstream at some point). I would not have (been able to do that), had tract been written in C++, for example.
Unless you've got massive compute bound transformers or old-school full convolutions, if you're interpreting a list of operations you're going to lose perf.