Yes, I believe it was for a Mixture of experts model, which just goes to show how naive IA implementations are at this point, The cache was not solving any hard problem and yet for some reason it was not only not already standard practice but also somehow a notable achievement.
The other day I saw a video talking about this new innovation on LLM inference side of things where they keep some more used weights in RAM and others less used on disk. I always suspected from the sample code I stumbled upon on the IA world that should be extreme opportunities for optimizations. But I cannot stress it enough how dumb the LLM world is where the basics of implementing an LRU cache is pass of as some big innovation. Like any half competent comp-sci or comp-eng professional know about the basics of mitigating this basics bottlenecks like “the data does not fit on available RAM”, “The disk is slow”, etc.
So is not surprising that now that it seems that the “powerfulness” of this LLMs is starting to plateau that we would start to see some improvement in performance/resource utilization and hence running costs.
Thank you for your detailed comment.
Unironically the part talking about song lengths and types of showers was really funny, now I keep thinking in characterizing things in terms of song lengths. It opens so many possibilities of how to express and do fun jokes. My wife and I already say stuff like “Are you going for number 1 or 2? … Let me use the bathroom first then I need to quick do X” But now the possibilities are endless, like “Are you going for a Blitzgreig Bop or a Stairway to Heaven?”
Anyway, back to the main topic, I remember doing something like that years ago, like timing the shower or using a playlist to gauge how long I am taking, but I guess nothing that detailed.
I feel like your breakdown maybe leaving some parts behind or take different time for me, like I scrub my ears and because of my piercings it takes a little more time. I also like to scrub between my toes and that is kinda fast but adds up. But I guess for me the true challenge is the same of comics someone else shared in this thread, really hard not to stop for a minute and take a deep breath, feel the hot water and just enjoy it for a while and gather my thoughts and energy, not to start the day for me, but to go to bed.
Anyway, much rumbling about nothing on my part, but I wanted to share, mostly to say I really enjoyed your comment and the time and effort you put into It. At least for me I usually dress leaving long comments because I am usually on mobile and hate typing on the phone.
I know, that is me. Very short and each day less and less hair. Maybe I’m just slow or clumsy ¯\_(ツ)_/¯
I am sorry, one song? 1, uno? That is crazy for me. I am all for quick showers but unless you are like a Pink Floyd fan I would be running like crazy to finish a shower in less than 5min, unless you don’t wash your hair every day which I do but I know for some types of hair people don’t.
I am not sure it would not help commercial solutions, if all experts are used all the time, sure, but if for example the usage is biased for some experts it would enable one machine to serve more users in parallel or save on VRAM or DRAM without compromising response time, hence cutting costs.