Rendered at 00:55:14 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
tim-star 2 days ago [-]
seems like a pretty clear-eyed analysis to me.
we're rapidly approaching the paperclip maximizer
HackerThemAll 2 days ago [-]
I think this is why Google hesitated to publish their internal model which was ready long before mindless OpenAI idiots released their first ChatGPT in November 2022.
That's so nice of the good fellas at Google, if only every for-profit company was ever so gentle and considerate of the public at large.
chrisjj 2 days ago [-]
> an actual rogue AI outsmarting its creators
This tells us only how little smarts is required to create a (so-called) AI.
eddyg 2 days ago [-]
If nothing else, the article has a really good timeline of the OpenAI/HuggingFace “incident”.
But to me, it underscores the impending cliff of doom from the continued release of open-weight models: there's no cryptographic or architectural way to give someone full weights while withholding the nefarious capabilities those weights encode.
As noted in this paper⁽¹⁾, “publicly releasing weights is an act of irreversible proliferation”.
I’m sure this will be an unpopular opinion on HN, but open weights are the thing that scares me the most about “A.I.”. There is a lot of research in this area⁽²⁾, and I think most of HN is unaware of it or ignores it. Stripping refusals from Kimi K2.5 took under $500 of compute and about 10 hours, taking HarmBench refusals from 100% to 5% while retaining nearly all capability; the resulting model gave detailed chemical-weapons synthesis instructions.
The gate is only as strong as the least-cautious releaser...
> there's no cryptographic or architectural way to give someone full weights while withholding the nefarious capabilities those weights encode.
This is true of closed weights, and in fact the problem is worse because they cannot even be scrutinized. We should ban closed weight AI for the very reasons you have just given
eddyg 2 days ago [-]
Constitutional classifiers go a long way to reducing unsafe usage in closed-weight models. And like we saw with Fable, closed models can be revoked and classifiers updated when “jailbreaks” are found.
Having the weights gives you the exact affordance an unlearning attack requires, without rate limits.
bcjdjsndon 2 days ago [-]
Ps llms have cheated for years, this most recent tripe is an AI vendor trying to hype up its next word predictor in a market of very samey next word predictors, before someone smarter than them eventually figures out how to do this (training + inference) on consumer hardware and kills the market for cloud ai
bcjdjsndon 2 days ago [-]
Stick those same classifiers (that you admit dont seem to work) on the open models, and done.
eddyg 2 days ago [-]
Classifiers are policy enforced by the process serving the model.
Input classifiers get applied before it reaches the model so somebody hacking an open-weight model would skip this. Streaming classifiers get polled during decoding; hackers delete this check in the sampling loop.
But both are always applied in closed weight models.
Set Llama Guard to 1.0 and nothing is ever unsafe.
bcjdjsndon 2 days ago [-]
More than one way to do guard rails, slopboy
eddyg 1 days ago [-]
All of which are easily bypassed in open weight models; see my previous comment about K2.5.
bcjdjsndon 1 days ago [-]
Using this as your argument is dangerous. You're saying AI is inherently dangerous and an emergency stop button is actually what you need to be safe from the dangers... Forget the fact your open weighted model could be ran from a central location
chrisjj 2 days ago [-]
So... no different from a book of detailed chemical-weapons synthesis instructions. The "AI" angle is immaterial.
eddyg 2 days ago [-]
A change in kind is not the same as a change in degree. Ten orchestrated LLM PhD advisors is a genuinely different thing from a library.
chrisjj 1 days ago [-]
Did you mean ten orchestrated libraries?
paxys 2 days ago [-]
There are plenty of cybersecurity books out there. None of them will launch an attack if you ask them to.
2 days ago [-]
chrisjj 2 days ago [-]
> the resulting model gave detailed chemical-weapons synthesis instructions.
Instructions, not action.
Actors are abundant.
potsandpans 1 days ago [-]
Posting ai doomer fud and lessworng articles on hackernews, a tale as old as time itself.
reducesuffering 1 days ago [-]
Which have been very prescient about where we're currently at, unlike the many trivial HN comments, like yours.
potsandpans 1 days ago [-]
Is the basilisk in the room with us now?
1 days ago [-]
15773265326 2 days ago [-]
[flagged]
eternauta3k 2 days ago [-]
I don't think it makes sense to post this in HN. The comments are full of sock puppets posting very dumb anti-safety comments in an attempt to make the anti-safety camp look bad.
we're rapidly approaching the paperclip maximizer
https://research.google/blog/towards-a-conversational-agent-...
This tells us only how little smarts is required to create a (so-called) AI.
But to me, it underscores the impending cliff of doom from the continued release of open-weight models: there's no cryptographic or architectural way to give someone full weights while withholding the nefarious capabilities those weights encode.
As noted in this paper⁽¹⁾, “publicly releasing weights is an act of irreversible proliferation”.
I’m sure this will be an unpopular opinion on HN, but open weights are the thing that scares me the most about “A.I.”. There is a lot of research in this area⁽²⁾, and I think most of HN is unaware of it or ignores it. Stripping refusals from Kimi K2.5 took under $500 of compute and about 10 hours, taking HarmBench refusals from 100% to 5% while retaining nearly all capability; the resulting model gave detailed chemical-weapons synthesis instructions.
The gate is only as strong as the least-cautious releaser...
⁽¹⁾ https://www.lesswrong.com/posts/qmQFHCgCyEEjuy5a7/lora-fine-...
⁽²⁾ https://arxiv.org/html/2604.03121v1
This is true of closed weights, and in fact the problem is worse because they cannot even be scrutinized. We should ban closed weight AI for the very reasons you have just given
Having the weights gives you the exact affordance an unlearning attack requires, without rate limits.
Input classifiers get applied before it reaches the model so somebody hacking an open-weight model would skip this. Streaming classifiers get polled during decoding; hackers delete this check in the sampling loop.
But both are always applied in closed weight models.
Set Llama Guard to 1.0 and nothing is ever unsafe.
Instructions, not action.
Actors are abundant.