Rendered at 12:50:59 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
Giho 3 hours ago [-]
I wonder if Kimis KDA architecture is a way closer to online models(models that learn during interference).As KDA updates its state matrix:
St =St−1 +βt (vt −St−1 kt )ktT
The expression (vt −St−1 kt )ktT is literally the gradient step of a linear regression model updating its weights (S) on a single sample using Mean Squared Error loss.
Interesting what other labs comes up with from this.
1 hours ago [-]
mickael-kerjean 12 hours ago [-]
Genuine question: how reproducible / usable / verifiable are these architectures from the published documentation? Are they similar to PDF/DWG/PSD specifications, where the format look like an open spec at first sight until you attempt to implement it and realize the crucial implementation details are undocumented?
nl 7 hours ago [-]
It's entirely reproducible from the available documentation (which is why you see vLLM, SGLang, MLX etc all racing to produce optimized implementations).
(As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the data anyway (because of randomness!))
FinchNova12 6 hours ago [-]
I agree that if you have the weights you can use/train a model with the same architecture, and that you won't get the exact weights on your own due to randomness. But isn't data an extremely important part of your ability to effectively train/finetune? It might be much harder to get close to the level of the open weight model if you don't have the data that made it, which is why I think the open weights vs open source distinction is useful.
spider-mario 5 hours ago [-]
Isn’t this a bit like saying “ ‘open object files are not open source’ is a complete misunderstanding” because “You can’t rebuild the executable from scratch, but you can’t do this even with the source code anyway (because of build nondeterminism / compiler versions / etc.)”?
desterothx 4 hours ago [-]
not really, compiling isnt a heuristic problem, it has a lot less randomness involved
eru 4 hours ago [-]
Depends on your compiler. You could have a compiler that deliberately uses randomised algorithms. They are often faster and easier to understand and write.
Though in practice you can get all the benefits of both determinism and (that kind of) randomisation by using a PRNG and saving the seed you are using.
It's an open question roughly on par with P vs NP whether true randomisation is ever necessary, or whether PRNGs are enough. So far we haven't found any problem or algorithm where true RNG is necessary and good PRNG ain't enough.
pcmasterr 1 hours ago [-]
Also, most (optimizing) compilers ”optimize” the code for a fixed amount of time, leading to better optimized binaries on faster computers. That’s why developers should have as fast computers money can buy!
eru 4 hours ago [-]
> (As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the data anyway (because of randomness!))
They could give you the random seeds? (Assuming you carefully train in such a way to remove other sources of randomness, like concurrent execution.)
alightsoul 4 hours ago [-]
Do AI labs even keep track of seeds? They don't even keep the checkpoints most of the time.
esperent 7 hours ago [-]
> open weights are not open source" thing is a complete misunderstanding
> You can't rebuild it from scratch
There's is extremely clear and misunderstanding-free.
Open weights is not open source.
embedding-shape 3 hours ago [-]
> You can't rebuild it from scratch
> It's entirely reproducible from the available documentation
You have a very interesting understanding of "reproducibility", I'll give you that :)
But even with that, there are plenty of technical details (especially in regards to the training process) missing from the tech report that leads to these weights not being reproducible in any sense of that word.
nvme0n1p1 3 hours ago [-]
> You have a very interesting understanding of "reproducibility", I'll give you that :)
He was talking about two different things, hence the parentheses. The architecture is reproducible, not the model weights.
embedding-shape 2 hours ago [-]
> The architecture is reproducible
But what's the point of even saying that? Of course it is, otherwise how is it supposed to run in the runtimes? You cannot release model weights that others can run, without also releasing the model architecture, it's in the code at the very least...
nvme0n1p1 1 hours ago [-]
Because for other specs it's not the case.
The K3 arch can be implemented from the spec.
A PDF reader can not be implemented from the spec.
davidguetta 2 hours ago [-]
the beginning of the arguments was good but the "because of randomness" is wild
calebkaiser 12 hours ago [-]
Implementing models directly from papers is typically pretty doable (and is of course more straightforward when the full implementation is open sourced). Often there is some amount of specific knowledge, like particular hyperparameters, that is missing and has to be trial and errored by the community, but generally speaking, getting the core model architecture implemented is a reasonable task for most well documented models.
Reproducing the exact training run, however, is basically impossible without the original dataset and training pipeline (here meaning all of the code + infra involved in actually executing the pre and post training loops). Also, it would be exorbitantly expensive to do if you weren't also a lab trying to train a similar model.
But you can still scale the architecture down and experiment as a solo researcher using the published research. There are probably some open source implementations already on GitHub for any given big open model release.
marcyb5st 4 hours ago [-]
The exact training run is basically impossible anyway. Randomness plays a role. Even if you fix your RNG seed, in a distributed training scenario like this one some weight updates might come at different times and be included in different update steps. Should have minimal impact on the final outcome, but would still be a different model as some of the weights will differ in the end.
eru 4 hours ago [-]
> Should have minimal impact on the final outcome, [...]
I share this expectation. But this is an interesting empiric question that deserves study; even if just to confirm what 'everyone knows'.
pcmasterr 57 minutes ago [-]
Reproducibility isn’t something that has been considered desirable in many pipelines until recent years.
Heck even ffmpeg introduce randomness when stitching together downloaded chunks from youtube. By design.
dannyw 11 hours ago [-]
The architectures are high level concepts and the mechanics usually have enough detail for you to try and implement.
Transformers are very "mendable" in that you can permute the architecture in crazy or random ways, and still basically always end up with get a coherent LLM. The difference comes down to training efficiency, inference efficiency, and usually minor differences in performance.
Hyperparams and stuff, I mean it's standard to do a sweep anyway.
GaggiX 12 hours ago [-]
The code is open source, and there probably several different implementations.
pepinal 5 hours ago [-]
[flagged]
thatsgcasey 17 hours ago [-]
Sabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substack
rsolva 16 hours ago [-]
This seems like a rare gem in a space swamped with lazily LLM generated summaries. Added to my RSS feed!
Upvoter33 13 hours ago [-]
I would describe him as a leading educator / explainer / teacher in this space.
pcmasterr 52 minutes ago [-]
Thoguhtleader
hugopuybareau 15 hours ago [-]
Never heard of him before but I was amazed how straight to the point yet very easy to grasp it was
3abiton 16 hours ago [-]
I second that. I've always enjoyed his work!
augment_me 10 hours ago [-]
I feel like the Kimi team is amongst the best in the industry to pick and choose what is meaningful from the other models. For example, avoiding the expensive and empirically uncertain mHC in favor of simpler residuals. Latent MoE.
My only doubts are around Linear Attention instead of DSA as this is inherently lossy. You are kind of banking on that your query is inherently in the embedding space of the model already and can be lossy.
mensetmanusman 10 hours ago [-]
They need to work on token cost for unrestrained problem solving.
madhu_ghalame 4 hours ago [-]
A few diagrams showing how the data flows through the architecture would make these concepts much easier to grasp.
anentropic 3 hours ago [-]
Ask Kimi
gboss 9 hours ago [-]
Anybody getting the result that Kimi 3 is more expensive than Opus 5 or Sol on Cursor? Pretty sure Kimi 3 sucked up a good chunk of my ultimate plan in a few prompts. Anyone have any tools or ways to understand per model usage towards cursor subscriptions? I know there are alternatives to cursor just haven’t made the move yet. (Edit spelling)
cadamsdotcom 8 hours ago [-]
Oh no! First time dealing with an American corporation?
igravious 6 hours ago [-]
The assertions doing the rounds that Kimi K3 and GLM 5.2 are way cheaper than Claude/GPT are not true -- DeepSeek V4 Pro is a lot cheaper but K3 and 5.2 ain't. Turns out that you actually have to fork out some cash for frontier-esque models, be they Chinese or American. Hope that helps.
Source: my bank balance
negativezone 35 minutes ago [-]
is it cheaper for medium scale folks? not everyone needs to go big on ai subscriptions and a solid, reliable midrange option wouldn't go amiss
Ilaurens 16 hours ago [-]
"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead."
It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?
yhocd 11 hours ago [-]
The primary author of RoPE, Jianlin Su, is currently working at Moonshot (KIMI)
itkovian_ 16 hours ago [-]
Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.
ModelForge 14 hours ago [-]
And adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"
cma 2 hours ago [-]
The paper says:
"Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."
dwohnitmok 16 hours ago [-]
As a sibling comment points out you don't strictly need positional embeddings for decoder-only causal transformers. You definitely need it for non-causal ones (e.g. the encoder of the original transformer paper!).
And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.
fspeech 15 hours ago [-]
Linear layers use decays (like IIR filters) that naturally provide relative positions. Full attention layers can then be free to develop concepts that attend to each other regardless of distance.
augment_me 11 hours ago [-]
When you have recurrent blocks in your model, you implicitly have a timestep T(amount of recurrent steps). Similar to Diffusion Transformers, it then becomes valuable to encode the knowledge of where you are in this chain somehow. NoPE is more flexible than RoPE for this.
natrys 15 hours ago [-]
At least the SSM layers should have relative PE built in via recurrence, in a hybrid attention model like this.
dools 3 hours ago [-]
They may have all this amazing architecture but Kimi has been super dumb recently. I reckon they’re under compute pressure and quantising to stay afloat. I was a heavy k2.5/2.6 user earlier in the year and built serious features with it, but even k3 now does stupid shit like fail tool calls and get stuck in endless thought trains. K2.6 was spinning its wheels on a problem for over 10 minutes today and then deepseek v4 fixed it in under a minute.
Something is wrong at moonshot.
irthomasthomas 3 hours ago [-]
Yeah something is up. I have the same problem with K3 as I had with earlier kimis. I ask it to write <bash>code</bash> every turn, that does not seem very difficult, but kimi gets this wrong a large percentage of the time. Yet, the code it writes is pretty good.
rurban 3 hours ago [-]
Deepseek is also struggling with simple tasks for the last two days. Claude is perfect, with the additional credits.
Aldipower 2 hours ago [-]
I have a lot of problems with Opus 5 in the last days. So, maybe the problem is to expect reliability from something probabilistic?
dools 3 hours ago [-]
I’ve had no issues with deepseek at all. Maybe it’s just luck of the draw
rurban 2 hours ago [-]
Good to hear. Most likely one of those monthly hickups, destroying main. Needed lots of reverts, and manual fixing. If in a branch I wouldn't care that much.
rglover 13 hours ago [-]
Just tried K3 out for the first time today and it's a legitimate threat.
Temporarily (maybe permanently) using it as my daily driver but it's wild how comparable it is to Opus 4.7/4.8 (what's been my go to for a bit now—wrote a quick post on what I found today [1]).
Better than Opus 4.8 on complex tasks but tends to overthink.
It found a bunch of bugs and architecture issues that only 5.6 Sol Max and Fable on my C++ projects.
ATMLOTTOBEER 11 hours ago [-]
Agreed. Dropped my $200/mo Claude I was saturating for the $100/mo moonshot sub and have no complaints
Plus I can actually see the thinking traces and I’m not getting downgraded/denied by anthropic constsntly…
nmfisher 7 hours ago [-]
How much usage do you get out of the $100 Moonshot plan? I haven't heard great things.
igravious 6 hours ago [-]
threat?
that's a weird way of describing a near frontier open weights un-crippled useful coding buddy
do you work for OpenAI or Anthropic per chance?
rglover 23 minutes ago [-]
I work for myself. A quick check of my profile here would make that clear.
And yes, it's a legitimate threat to frontier models. Both K3 and the distillation approach.
rekpero 4 hours ago [-]
kimi K3 felt on par with my claude opus 4.8 in my first tests and is exceptionally good at frontend and website motion design, but I still find it hard to switch back and forth between claude and kimi, and i’m not sure if others are experiencing the same thing.
constantlm 15 hours ago [-]
So, unlike what leaders of western labs labs would like you to believe (that Kimi is just the result of distillation attacks), they are introducing new and novel approaches.
nostromo 13 hours ago [-]
Even if they are distilling, I don't particularly care. Anthropic and others have been distilling copyrighted material by to build these models, largely without permission.
ransom1538 11 hours ago [-]
Exactly,let Anthropic get back to training on the New York times.
wnsdy95 11 hours ago [-]
this is so true honestly
xdennis 9 hours ago [-]
People don't remember now, but in the old HN comments like "this is so true honestly" would have been downvoted. If a comment just agrees with the parent, people would have said that's what the upvote button is for. Adding a "I agree" comment is just noise.
protocolture 9 hours ago [-]
This is so true honestly.
Bluestein 3 hours ago [-]
Load-bearing, actually.-
xdennis 9 hours ago [-]
Also Reddit-style humorous replies were frowned upon because they ruin serious discussion and encourage karma chasing as opposed to providing valuable insight.
mejutoco 5 hours ago [-]
This is why on Slashdot you could vote on insightful, humorous, etc. Something neither reddit, not hn have caugh on yet. Digg could have done it.
ThePinion 7 hours ago [-]
With all the botting and slop going on right now on HN I'd say these once annoyances are now quite arbitrary. I'm sure it bothered you and maybe one other person at this point in time, but it's pretty damn easy to scroll past it compared to actively reading comments that you half way through realize are just LLMs.
newswasboring 7 hours ago [-]
To be honest, comment like yours were also frowned upon haha. If you think something is bad for discussion, downvote. If you think something is objectionable, flag it. And now I have made a faux pas by explaining the rule to you rather than just downvoting and moving on. Its turtles all the way down today.
mensetmanusman 10 hours ago [-]
If you redefine the technical world distilling to mean training on text, you are correct.
9 hours ago [-]
brookst 8 hours ago [-]
We’re all distillers now.
ozgung 6 hours ago [-]
Moreover, Distillation is a misused term here. Distillation means training a smaller student model using a larger teacher model to completely mimic its behavior. As in you take a base model and create its smaller “turbo” version. Like distillation in Chemistry it means it’s %100 purified version of its teacher.
boomskats 14 hours ago [-]
No! It's just not fair!! What about our financial bubble!
FinchNova12 11 hours ago [-]
"Kimi is largely a byproduct of distillation" and "Kimi is introducing new and novel approaches" are not mutually exclusive, and I'm not sure it's clear from the paper how much of the improvement comes from the new approaches. So I wouldn't take the new approaches to be much evidence about whether the distillation attacks occurred.
anon373839 10 hours ago [-]
> distillation attacks
Like the distillation “attacks” Anthropic conducted against literally millions and millions of different creators?
boomskats 4 hours ago [-]
Or against all those old books that they ingested and then literally destroyed
mensetmanusman 10 hours ago [-]
Scanning google is an ‘attack’
waterTanuki 7 hours ago [-]
Reading Claude's outputs is an 'attack'
nl 7 hours ago [-]
Even Anthropic hasn't claimed "Kimi is largely a byproduct of distillation"
There's been some distillation.
Just like how Elon said in court they distill to make Grok.
FinchNova12 6 hours ago [-]
I thought there is discussion circulating around whether the main reason Kimi is impressive is due to distillation. Though possibly if this occurred, this was just one component of their training pipeline and not a majority
hlynurd 14 hours ago [-]
"attack"
alealvarezarg 17 hours ago [-]
Great breakdown. After using Kimi extensively, it's fascinating to see how architectural choices like KDA and NoPE translate into such strong real-world performance. Really impressive engineering.
3 hours ago [-]
souravsspace 17 hours ago [-]
i like your detailed breakdown. Thanks. <3
RobLach 15 hours ago [-]
Concise. Nice.
TokenLat 5 hours ago [-]
[flagged]
gokohl 20 hours ago [-]
Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
thunderbird120 19 hours ago [-]
Kimi Delta Attention (KDA), despite having "Attention" in the name, isn't really attention at all in any conventional sense. It's more like an RNN which can be efficiently parallelized during training. It's a very small modification to Gated DeltaNet, which can be described as an RNN whose hidden state acts like a small, editable attention memory.
Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.
Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.
RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.
>Curious to see if it holds up at frontier scale.
I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.
wongarsu 20 hours ago [-]
Is frontier scale larger than this? Kimi K3 seems to benchmark in the same range as Opus and Fable. I would have expected they are all in the 2-4T range, with quality of the training and architecture differences as the major differentiators
porridgeraisin 20 hours ago [-]
The number of active parameters is vastly different. Deepseek CEO hinted that he estimates it as an order of magnitude difference in one of his recent interviews.
> Seems to benchmark
yes, but in human usage the differences show up
kamranjon 20 hours ago [-]
Would you happen to have a link to that interview? Sounds like an interesting read.
It has some weird side effects though. for example KV-caches are implemented in fixed incremental token blocks (1024 from the providers I used) instead of simply caching up to the most recent input prompt input. It results in up to 1023 additional input (cache miss) tokens per inference.
wren6991 18 hours ago [-]
Sounds a lot like running the Qwen3.5/3.6-series models at home: you need checkpoints for the recurrent state (GDN in the case of Qwen). You avoid the miss for the common case of 100% prefix match (e.g. during interleaved tool calls and thinking) by keeping an additional checkpoint for the actual last generated token. If you're a cloud provider serving many concurrent clients then you might prefer to skip that complexity and always take the 1k worst-case prefill hit.
SWA has a similar issue. Unless you keep the entire KV prefix lying around (which is not unreasonable: you retain flop + bandwidth benefits but lose capacity benefits), you need to start 1 window back from the rollback point, in order to refill the sliding window before going into normal prefill.
cubefox 18 hours ago [-]
This comment smells LLM generated.
losvedir 18 hours ago [-]
Huh, which model? I didn't pick up any of the tells. It seemed normal enough to me. (I vouched it, oops.)
cubefox 15 hours ago [-]
Telling which model wrote it is much harder than telling that some model wrote it.
The expression (vt −St−1 kt )ktT is literally the gradient step of a linear regression model updating its weights (S) on a single sample using Mean Squared Error loss.
Interesting what other labs comes up with from this.
(As an aside, this is why the "open weights are not open source" thing is a complete misunderstanding. The weights themselves along with the documentation give you enough to fine tune the LLM. You can't rebuild it from scratch, but you can't do this even with the data anyway (because of randomness!))
Though in practice you can get all the benefits of both determinism and (that kind of) randomisation by using a PRNG and saving the seed you are using.
It's an open question roughly on par with P vs NP whether true randomisation is ever necessary, or whether PRNGs are enough. So far we haven't found any problem or algorithm where true RNG is necessary and good PRNG ain't enough.
They could give you the random seeds? (Assuming you carefully train in such a way to remove other sources of randomness, like concurrent execution.)
> You can't rebuild it from scratch
There's is extremely clear and misunderstanding-free.
Open weights is not open source.
> It's entirely reproducible from the available documentation
You have a very interesting understanding of "reproducibility", I'll give you that :)
But even with that, there are plenty of technical details (especially in regards to the training process) missing from the tech report that leads to these weights not being reproducible in any sense of that word.
He was talking about two different things, hence the parentheses. The architecture is reproducible, not the model weights.
But what's the point of even saying that? Of course it is, otherwise how is it supposed to run in the runtimes? You cannot release model weights that others can run, without also releasing the model architecture, it's in the code at the very least...
The K3 arch can be implemented from the spec.
A PDF reader can not be implemented from the spec.
Reproducing the exact training run, however, is basically impossible without the original dataset and training pipeline (here meaning all of the code + infra involved in actually executing the pre and post training loops). Also, it would be exorbitantly expensive to do if you weren't also a lab trying to train a similar model.
But you can still scale the architecture down and experiment as a solo researcher using the published research. There are probably some open source implementations already on GitHub for any given big open model release.
I share this expectation. But this is an interesting empiric question that deserves study; even if just to confirm what 'everyone knows'.
Heck even ffmpeg introduce randomness when stitching together downloaded chunks from youtube. By design.
Transformers are very "mendable" in that you can permute the architecture in crazy or random ways, and still basically always end up with get a coherent LLM. The difference comes down to training efficiency, inference efficiency, and usually minor differences in performance.
Hyperparams and stuff, I mean it's standard to do a sweep anyway.
My only doubts are around Linear Attention instead of DSA as this is inherently lossy. You are kind of banking on that your query is inherently in the embedding space of the model already and can be lossy.
Source: my bank balance
It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?
"Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."
And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.
Something is wrong at moonshot.
Temporarily (maybe permanently) using it as my daily driver but it's wild how comparable it is to Opus 4.7/4.8 (what's been my go to for a bit now—wrote a quick post on what I found today [1]).
[1] https://graybearding.bearblog.dev/kimi-k3-is-insane/
Plus I can actually see the thinking traces and I’m not getting downgraded/denied by anthropic constsntly…
that's a weird way of describing a near frontier open weights un-crippled useful coding buddy
do you work for OpenAI or Anthropic per chance?
And yes, it's a legitimate threat to frontier models. Both K3 and the distillation approach.
Like the distillation “attacks” Anthropic conducted against literally millions and millions of different creators?
There's been some distillation.
Just like how Elon said in court they distill to make Grok.
Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.
Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.
RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.
>Curious to see if it holds up at frontier scale.
I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.
> Seems to benchmark
yes, but in human usage the differences show up
1. https://news.ycombinator.com/item?id=49019012 (original chinese)
2. https://news.ycombinator.com/item?id=49052912 (translated english [pdf])
SWA has a similar issue. Unless you keep the entire KV prefix lying around (which is not unreasonable: you retain flop + bandwidth benefits but lose capacity benefits), you need to start 1 window back from the rollback point, in order to refill the sliding window before going into normal prefill.