Open Source AI vs Corporate AI: What Each Model Will and Won’t Tell You

Open Source AI vs Closed: The Two Camps

On one side: closed models. These are AI systems you access through an API or a website. The company controls the weights (the internal parameters that make the model work), and you interact only through their interface. This camp includes OpenAI’s GPT series, Anthropic’s Claude, and Google’s Gemini. The company can update the model, change its behavior, and monitor how it’s used.

On the other side: open-weight models. These are models where the developer publishes the actual files that make the model work. You can download them, run them on your own hardware, and modify them however you want. This camp includes Meta’s Llama, Mistral’s models, and Google’s Gemma series. Once the weights are out, the developer has no practical control over what happens next.

Note the careful terminology: “open-weight” rather than “open-source.” Most of these models come with licenses that restrict certain uses. Meta’s Llama 3 license, for instance, prohibits use for weapons development, malware, and activities presenting “a risk of death or bodily harm.” But once you have downloaded the weights, enforcement is essentially on the honor system.

What Each Model Refuses

A Stanford CRFM study examining 30 foundation modelA large-scale AI model trained on broad, diverse data that serves as a starting point for developing specialized applications. developers’ acceptable use policies found near-universal agreement on some categories (illegal content, malware, child exploitation) and significant divergence on others. Here is what the practical landscape looks like across five categories people actually care about:

Medical advice. All closed models will discuss health topics but wrap everything in disclaimers. GPT, Claude, and Gemini will explain drug interactions, describe symptoms, and discuss treatment options, but they will remind you they are not doctors. The open-weight base models (Llama, Mistral, Gemma) behave similarly out of the box. Community-modified “uncensored” versions will give you the same information without the disclaimers.

Weapons and explosives. This is where the gap widens. Closed models refuse to provide synthesis instructions for explosives or detailed weapon modification guides. Open-weight models in their default configuration do the same. Uncensored community variants will discuss these topics more freely, though the information they provide is generally no more detailed than what is available in chemistry textbooks or military field manuals that have been public for decades.

Controversial history. Closed models handle sensitive historical topics (colonialism, wartime atrocities, genocide) with careful framing and multiple perspectives. Claude and GPT tend to add extensive context. Gemini has historically been the most cautious, sometimes declining to discuss sensitive historical figures at all. Open-weight models, especially uncensored variants, will discuss these topics without the editorial framing.

Adult content. The clearest binary. All major closed models refuse to generate explicit sexual content. Open-weight uncensored models have no such restriction. This is, by volume, the single most common reason people seek out uncensored models.

Political opinions. Closed models are trained to present balanced perspectives and avoid taking sides. The Stanford study found that OpenAI, Anthropic, Cohere, and Midjourney explicitly prohibit use for political campaigning or lobbying, while Google and Meta impose no such restriction. Uncensored open models will express whatever opinions were present in their training data without the “both sides” framing.

How “Uncensoring” Actually Works

When a company like Meta releases Llama, the model has already been through a process called RLHFA machine learning process where AI models learn from human feedback on their outputs, teaching them which responses to prioritize or refuse. (Reinforcement Learning from Human Feedback). Think of it this way: the base model learned from the entire internet and knows about everything, including things it probably should not casually discuss. RLHF is the process of teaching the model when to say “I’d rather not answer that.”

The key insight is that RLHF does not remove knowledge. It adds a behavioral layer on top. The model still “knows” the information; it has just been trained to decline discussing it. This is why uncensoring is possible at all.

The community uses two main approaches:

Fine-tuningFurther training a pre-trained AI model on specific data to adapt its behavior for a particular purpose or specialized task. on filtered datasets. This is the method pioneered by Eric Hartford with his Dolphin series of models. Hartford takes instruction-tuning datasets (the data used to teach models to follow instructions) and filters out every instance where the model refuses, avoids, or moralizes. Then he fine-tunes the model on this cleaned dataset. The result is a model that responds to instructions without the refusal patterns. Hartford’s argument: “There is no ‘one true correct alignmentIn AI safety, the process of ensuring an AI system's goals and behaviors match human values and intentions. Poor alignment can cause AI systems to optimize for measurable metrics in ways that contradict human interests.‘ and there’s no reason why that should be OpenAI’s brand of alignment.”

AbliterationA technique for removing AI refusal behavior by identifying the mathematical "refusal direction" in model activations and projecting it out of the weight matrices.. This is the more surgical approach. Researchers discovered that refusal behavior in language models is encoded as a specific direction in the model’s internal activations, a single mathematical vector that, when triggered, produces a refusal response. Abliteration identifies this “refusal direction” and removes it from the model weights through a process called orthogonalization. The math is surprisingly elegant: you modify the weight matrix W to become W’ = W – r(r^TW), where r is the refusal direction vector. This prevents the model from producing activations along the refusal axis while leaving all other capabilities intact.

The practical result: within days of Meta releasing a new Llama model, abliterated versions appear on Hugging Face. The process is now so routine that the community expects it.

Is Open Source AI Dangerous?

This is where the debate gets genuinely interesting, because the evidence does not cleanly support either side.

The case for concern. The Future of Life Institute’s 2025 AI Safety Index scored major AI companies on safety practices. Meta received a D (1.06 out of 4), well below Anthropic’s C+ (2.64) and OpenAI’s C (2.10). The index specifically noted that open-weight release makes safety measures harder to enforce after distribution. The concern is real: once weights are public, the developer has no technical mechanism to prevent modification.

The case against panic. The NTIA’s 2024 report on open-weight models found that “no current models, proprietary or widely available, offer uplift on these tasks relative to open source information resources” when examining CBRNAcronym for Chemical, Biological, Radiological, and Nuclear — the four categories of weapons capable of mass casualties. Used in security policy and AI content restrictions. (chemical, biological, radiological, nuclear) risks. In plain language: current AI models do not teach you anything about building weapons that you could not already find in a university library or through a determined search. The NTIA recommended monitoring rather than restricting open-weight distribution.

The AI Frontiers research group made a complementary point: restricting open models risks creating “digital feudalism” where critical AI technology is controlled by a handful of corporations, while producing “little evidence that open models are driving a material increase in catastrophic risk over the baseline today.”

The honest assessment: the information hazard argument (that AI makes dangerous knowledge more accessible) is weakest precisely where it sounds most alarming. Chemistry textbooks, military manuals, medical references: this information is already freely available. An AI model that discusses these topics is not creating new knowledge; it is making existing knowledge slightly more conversational. The genuine risks lie in future models that might synthesize information in novel ways or possess capabilities that exceed what is currently available through traditional research, but we are not there yet.

What This Means for You

If you use closed models (ChatGPT, Claude, Gemini), you are getting a curated experience. The model is opinionated about what it will and will not discuss, and those opinions are set by the company. This is not inherently bad: most users benefit from models that do not casually discuss how to synthesize dangerous chemicals. But it also means the company is making editorial decisions about what knowledge you can access through their product. (For a related look at how AI is reshaping software quality, see our piece on AI slopware.)

If you use open-weight models, you have the option to run them as released (with safety guardrails intact) or to use community-modified versions. The existence of uncensored models is not going away; the techniques are well-understood, the demand is real, and the licensing restrictions are unenforceable once weights are downloaded.

The more productive question is not whether uncensored models should exist. It is how we build systems that are genuinely useful without being reckless, and how we distinguish between protecting users from harm and protecting companies from liability. Those are different goals, and they produce different guardrails.

The Architecture of Refusal in Open Source AI

Content restriction in large language models operates at multiple levels, and understanding where in the stack each restriction lives explains why some are easy to remove and others are not.

System prompt restrictions are the outermost layer. These are natural-language instructions prepended to every conversation. GPT-4, Claude, and Gemini all use system prompts that specify refusal categories. These are trivially bypassed in open-weight models by simply removing them, and in closed models through various “jailbreak” techniques that, despite constant patching, continue to find new attack surfacesThe total set of points in a system where an attacker can attempt to enter, extract data, or cause damage..

RLHFA machine learning process where AI models learn from human feedback on their outputs, teaching them which responses to prioritize or refuse./RLAIFA variant of RLHF where another AI model evaluates outputs instead of human annotators, reducing reliance on expensive human labelling while still encoding policy preferences. alignmentIn AI safety, the process of ensuring an AI system's goals and behaviors match human values and intentions. Poor alignment can cause AI systems to optimize for measurable metrics in ways that contradict human interests. is the behavioral training layer. After pretraining on text corpora and instruction fine-tuningFurther training a pre-trained AI model on specific data to adapt its behavior for a particular purpose or specialized task., models undergo reinforcement learning from human (or AI) feedback. This process shapes the model’s response distribution: given a prompt about weapons synthesis, the model learns to assign high probability to refusal tokens and low probability to compliance tokens. Critically, RLHF does not modify what the model “knows” in its feed-forward layers; it modifies the policy that governs which knowledge gets surfaced. The base Llama 3.1 405B model, before RLHF, will discuss topics that the chat-tuned version refuses.

Safety classifiers are post-generation filters that evaluate outputs before delivery. OpenAI runs a separate classifier on GPT outputs; Anthropic uses constitutional AIA method for training AI models using a written set of explicit principles, where the model critiques its own outputs for compliance rather than relying solely on human-rated feedback. methods that embed safety reasoning into the generation process itself. These classifiers add latency and can be circumvented in open-weight deployments by simply not running them.

A Policy Comparison Across Providers

The Stanford Center for Research on Foundation ModelsA large-scale AI model trained on broad, diverse data that serves as a starting point for developing specialized applications. analyzed acceptable use policies across 30 developers, revealing telling divergences even among companies that appear to agree on principles:

Weapons development: Allen AI, Anthropic, Meta, Mistral, OpenAI, and Stability AI explicitly prohibit weapons-related use. Google’s policy for Gemini addresses “harm” broadly without specifically naming weapons. Gemma’s more permissive licensing reflects Google’s dual strategy: strict controls on the closed product, lighter restrictions on the open one.

Political content: OpenAI, Anthropic, Cohere, and Midjourney explicitly prohibit political campaigning and lobbying. Google, Meta, and Eleven Labs impose no such restriction. This divergence maps neatly onto business models: companies selling to enterprises (where political bias is a liability) restrict it; companies courting developer ecosystems (where flexibility is a selling point) do not.

Sexual content: Universal prohibition among closed providers. Among open-weight providers, Meta and Mistral narrowly restrict content involving minors and violence but do not broadly ban adult content. This creates the legal headroom for community fine-tunes.

Medical/legal/financial advice: Eight of 30 developers prohibit use for professional advice. The rest are silent, which is legally interesting: silence in a use policy is not permission, but it is also not prohibition.

A notable finding: only 3 of 10 leading developers disclose their enforcement mechanisms. Just 2 explain policy violations to users. The transparency gap between AI developers and social media companies (which publish regular transparency reports) remains wide.

The Mechanics of Uncensoring

Method 1: Dataset filtering + fine-tuning. Eric Hartford’s approach, documented in his technical blog post, works by curating the instruction-tuning dataset. Standard instruction datasets are generated using ChatGPT’s API, which means ChatGPT’s alignment preferences propagate downstream to every model trained on that data. Hartford’s process: take an existing instruction dataset, programmatically identify and remove every instance where the response contains refusal language (“I cannot,” “I’m not able to,” “As an AI”), then fine-tune the base model on this filtered dataset. The resulting model retains instruction-following capability but lacks the refusal distribution.

The compute cost is modest. The NTIA report noted that “fine-tuning away Llama 2-Chat 13B’s safety features while retaining model performance costs less than $200.” Fine-tuning a 7B parameter model on a filtered dataset can be done on a single consumer GPU with 24GB VRAM in under a day using QLoRA (Quantized Low-Rank Adaptation).

Method 2: AbliterationA technique for removing AI refusal behavior by identifying the mathematical "refusal direction" in model activations and projecting it out of the weight matrices. (activation engineering). This is mechanistically more interesting. The technique exploits the fact that refusal behavior in transformer models is encoded as a low-rank feature in the residual streamIn transformer neural networks, the running sum that flows through the model and accumulates representations as each layer adds its contribution., not distributed across the entire model.

The process works in five steps:

1. Data collection. Assemble paired prompts: one set that triggers refusal, one that gets compliant responses. Record activation vectors across all transformer layers for both sets.
2. Difference vector computation. Calculate mean activation vectors for refusing vs. complying states at each layer. Apply PCA to extract the dominant direction of variation: the “refusal direction.”
3. Layer identification. Refusal concentrates in middle-to-late transformer layers (typically layers 15-25 in a 32-layer model). This allows targeted intervention rather than whole-model modification.
4. Weight orthogonalization. Modify the weight matrix W to W’ = W – r(r^TW), where r is the unit refusal direction vector. This orthogonal projection mathematically prevents the model from producing activations along the refusal axis.
5. Validation. Test on previously refused prompts and standard benchmarks. Well-executed abliteration preserves general capability scores (MMLU, HumanEval) while eliminating refusal behavior.

The elegance of abliteration is that it demonstrates refusal is a surprisingly simple feature, geometrically speaking. RLHF, despite being an expensive training process, produces a behavioral change that can be described by a single direction in a high-dimensional space. This has implications for AI safety research: if safety behaviors are this low-rank, they may be fundamentally difficult to make robust against motivated removal in open-weight settings.

The Risk Landscape: What the Research Says

The Future of Life Institute’s Summer 2025 AI Safety Index evaluated seven leading AI companies across 33 indicators in six domains. The results paint a nuanced picture:

• Anthropic scored highest at C+ (2.64/4), noted for its substantive bio-risk evaluation practices
• OpenAI received C (2.10/4), distinguished by its published whistleblowing policy
• Google DeepMind scored C- (1.76/4)
• Meta received D (1.06/4)
• Only three firms (Anthropic, OpenAI, Google DeepMind) conduct substantive testing for dangerous capabilities tied to biosecurity and cyber-terrorism

But the FLI index primarily measures corporate safety practices, not model-level risk. The NTIA’s evaluation of actual model capabilities found a different story: “no current models, proprietary or widely available, offer uplift on these tasks relative to open source information resources” for CBRNAcronym for Chemical, Biological, Radiological, and Nuclear — the four categories of weapons capable of mass casualties. Used in security policy and AI content restrictions. threats. The marginal risk of open weightsAI model training files that are freely available to download and run locally, as opposed to closed models accessible only through paid APIs. Anyone can download and use open-weight models on their own hardware without a subscription., compared to information already available online, remains unclear.

The AI Frontiers group added important context: open-weight models already have tens of thousands of derivatives on Hugging Face, enabling domain experts to build specialized applications in medicine, law, and education that closed models cannot serve. Restricting open weights to prevent misuse would simultaneously prevent these beneficial applications.

The honest technical assessment: current-generation models do not meaningfully increase information hazards beyond existing baselines. The NTIA explicitly recommends monitoring rather than restricting. The risk calculus may change with more capable future models, particularly those trained on proprietary datasets (classified research, unpublished biological sequences) or those with substantially improved multi-step reasoning about physical-world processes. But that risk is speculative and future-tense, and policy built on speculation has a poor track record.

The Structural Incentive Problem

There is a dimension to this debate that rarely gets discussed technically: the business incentives that shape safety decisions.

Closed-model providers (OpenAI, Anthropic, Google) have a financial interest in portraying open-weight models as dangerous. Every user who runs Llama locally is a user not paying for API calls. “Safety” and “competitive moat” are not the same thing, but they can look identical from the outside. When OpenAI lobbies for regulation that would disadvantage open-weight competitors, the safety argument provides cover for what is also a business strategy.

Open-weight advocates (Meta, Mistral, the broader open-source community) have their own incentive: ecosystem lock-in through a different mechanism. Meta gives away Llama because an ecosystem of Llama-based applications increases Meta’s relevance in the AI stack. Mistral distributes models openly to compete with well-funded closed competitors. Neither is doing this purely out of ideological commitment to openness.

Understanding these incentives does not resolve the safety question, but it should make you skeptical of anyone who claims the answer is obvious.