• 0 Posts
  • 26 Comments
Joined 3 years ago
cake
Cake day: June 20th, 2023

help-circle
  • MIT: I build my thing on top of your thing, it’s my thing now. I sell it, you have no rights.

    GPL: I build my thing on top of your thing and your license forces me to distribute my thing under the GPL, including the source code.

    There are legitimate arguments for both. Sometimes it sucks when proprietary things get built on top of MIT code but sometimes it takes money for innovation, and in the realm of code the world often benefits anyway by virtue of copycats or distillation.















  • You can’t just write off capital expenditure though. The hardware, even for “effecient” MOE inference is still very expensive to buy, house, run, and cool. Even assuming open-weight model serving at $0 r&d for the models themselves, mixing high-prefill workloads doesn’t batch well with decode heavy concurrency (or other prefill-heavy jobs). The moment you do anything nontrivial you start running into very complicated architectural problems to efficiently solve at scale.

    Hardware that is useful for 5-10 years at most, plus development and support for the inference workflows, doesn’t leave a lot of margin on the table.

    My gut, along with basically everything I read, suggests that not most (even pure inference) shops are not profitable and are still floating on loans or vc money.