The Letter That Stopped Silicon Valley
On March 22, 2023, a public letter appeared online calling for a six-month pause on training AI systems more powerful than GPT-4. Within days, it had collected over 1,000 signatures from technology leaders, AI researchers, and public intellectuals. Elon Musk signed it. Steve Wozniak signed it. Yoshua Bengio, Turing Award winner and one of the "godfathers of AI," signed it. Andrew Yang signed it. Yuval Noah Harari signed it.
The letter, organized by the Future of Life Institute, represented an unprecedented moment in the history of technology: an industry calling for regulation of itself. But unlike previous calls for tech accountability, which typically came from outside critics, this one came from the field's most credible researchers. It argued that advanced AI systems posed "profound risks to society and humanity" and that "powerful AI systems should be developed only once we are confident that their effects will be positive and their risks will be manageable."
The man who orchestrated this campaign was not a household name. Max Tegmark, a 55-year-old MIT physics professor, had spent the previous decade building the intellectual and organizational infrastructure that made the pause letter possible. Starting from a position as an AI outsider—a cosmologist who studied the mathematical structure of the universe—he had transformed himself into one of the most influential voices in AI governance, a field that barely existed when he began.
Tegmark's journey from theoretical physics to AI safety represents something larger than one person's career pivot. It is the story of how existential risk went from a fringe concern discussed primarily in philosophy seminars and rationalist blog posts to a mainstream position embraced by governments, Fortune 500 companies, and the United Nations. It is the story of how a small group of academics and activists built the concepts, organizations, and political coalitions that now shape global AI policy.
In March 2023, when that letter circulated, Tegmark was not asking Silicon Valley to slow down out of abstract philosophical concern. He was asking based on a decade of research, relationship-building, and institution-creation that had positioned him and his organization at the center of the AI safety debate. His transformation from outsider to insider offers critical lessons about how scientific fields evolve, how policy movements gain traction, and what happens when existential questions move from theory to practice.
From Cosmos to Code
Max Tegmark's first career had nothing to do with artificial intelligence. He was, and in many ways still is, a cosmologist—someone who studies the largest questions in physics. Born in Sweden in 1967, Tegmark earned his Ph.D. from the University of California, Berkeley in 1994, writing a dissertation on how to measure the geometry of the universe using cosmic microwave background radiation.
His early academic work established him as a creative and unconventional thinker. In 1998, he published a paper proposing a radical hypothesis: that our universe is not just described by mathematics, but actually is a mathematical structure. The "Mathematical Universe Hypothesis," as it came to be known, argued that physical reality is not just well-described by math, but is identical to a mathematical structure. In this view, our universe is one of many possible mathematical structures, and we happen to find ourselves in one that can support observers.
This wasn't just abstract philosophy. Tegmark developed testable predictions from the hypothesis and published them in peer-reviewed physics journals. His 2003 paper in Physical Review D, titled "Parallel universes," has been cited over 1,300 times. By 2004, when he joined MIT's physics department as an assistant professor, he had established himself as a serious researcher working at the boundary between theoretical physics and philosophy.
The skills Tegmark developed as a cosmologist—the ability to think about extremely long timescales, to reason about the fundamental nature of reality, to work with probabilities in domains where experiments are impossible—would later prove essential to his work in AI safety. But the connection wasn't obvious at first.
Tegmark's pivot toward AI began around 2013, catalyzed by three developments. First, he had published his first popular book, "Our Mathematical Universe," and was thinking about how to make his research accessible to broader audiences. Second, deep learning was beginning to work at scale, with AlexNet's victory in the 2012 ImageNet competition signaling a new era in machine learning capabilities. Third, philosopher Nick Bostrom was finishing his book "Superintelligence," which argued that advanced AI posed existential risks that the research community was largely ignoring.
What drew Tegmark to AI safety wasn't fear of robots or science fiction scenarios. It was a physicist's recognition that the timeline for transformative AI might be shorter than anyone expected, combined with his cosmologist's training in thinking about existential risks. "I realized that all the things I care about for the long-term future of life in our Universe could be totally moot if we screw up the AI transition," Tegmark later explained in interviews.
Building the Movement
In January 2014, Tegmark attended a small conference in Puerto Rico organized by the Future of Humanity Institute. The topic was existential risk from artificial intelligence. In attendance were about 30 people: a handful of AI researchers, some philosophers, a few concerned technologists. Among them was Elon Musk, who had recently become interested in AI safety after reading Bostrom's book manuscript.
The Puerto Rico conference became a founding moment for the modern AI safety movement. Tegmark, along with Skype co-founder Jaan Tallinn and others, saw an opportunity to create something that didn't yet exist: a well-funded, professionally-run organization dedicated to reducing existential risk from advanced AI. They decided to refocus an existing organization, the Future of Life Institute (FLI), which had been founded in 2014 by Tegmark, Tallinn, and several other academics at MIT and Boston University.
The next step was finding funding. This is where Tegmark's emerging relationship with Musk became crucial. In the months following Puerto Rico, Tegmark and others in the nascent AI safety community began cultivating Musk's interest in the topic. They shared research, organized discussions, and made the case that AI safety was an urgent problem that needed serious funding.
The effort paid off. In January 2015, Musk announced a $10 million donation to FLI to fund AI safety research. The gift was transformative—not just for its size, which dwarfed previous funding in the field, but for its signaling value. When one of the world's most successful entrepreneurs and technology visionaries says AI safety is worth $10 million, other people pay attention.
Tegmark and FLI used the money strategically. Rather than building a large internal research operation, they distributed grants to researchers at established institutions. In their first grant round, they funded 37 research projects at universities around the world, covering topics from AI value alignment to the economic impacts of automation to legal frameworks for AI governance. The grants went to serious academics at top institutions: Berkeley, Oxford, Cambridge, Carnegie Mellon.
This approach had several advantages. It built a distributed network of researchers working on AI safety, rather than concentrating expertise in one place. It lent credibility to the field by associating it with prestigious institutions. And it created stakeholders—dozens of professors who now had FLI funding and therefore an interest in the organization's success.
But Musk's money was just the beginning. Over the following years, Tegmark and FLI built a more sophisticated operation. They organized conferences that brought together AI researchers, policymakers, and ethicists. They published open letters on specific topics, like autonomous weapons. They funded documentary films about AI risk. They hired staff with experience in science communication, policy advocacy, and nonprofit management.
The Intellectual Architecture
To understand Tegmark's influence on AI safety, it's necessary to understand the intellectual framework he helped popularize. Before FLI's intervention, AI safety was primarily discussed in two separate communities that rarely interacted: academic computer scientists working on technical problems like robustness and fairness, and philosophers and futurists discussing longer-term existential risks.
Tegmark and FLI helped bridge this divide by developing a shared vocabulary and conceptual framework that both communities could use. This framework organized AI risks into three categories, based on timescale:
Near-term risks included things already happening or likely to happen soon: algorithmic bias, privacy violations, autonomous weapons, labor market disruption, misinformation, and concentration of power. These were concrete problems that researchers could study and policymakers could address with existing tools.
Medium-term risks emerged as AI systems became more capable and autonomous: accidents from misaligned objectives, economic disruption from widespread automation, erosion of human agency, and the potential for AI systems to be misused by malicious actors. These required new technical and governance solutions.
Long-term existential risks centered on the possibility that advanced AI systems might pursue goals misaligned with human values in ways that could not be corrected or reversed. This was the "superintelligence" scenario that Bostrom had written about: AI systems that were more capable than humans at virtually all cognitive tasks, and whose actions humans could neither predict nor control.
By organizing risks this way, Tegmark and FLI created a framework that allowed people with different concerns to work together. Researchers worried about bias in criminal justice algorithms and philosophers worried about paperclip maximizers were no longer talking past each other—they were both working on different aspects of the same underlying problem: ensuring that AI systems do what we actually want them to do.
This framework also provided a response to critics who dismissed existential risk as science fiction. FLI could point to near-term problems everyone agreed were real, then explain how those same problems might become more severe as AI became more powerful. The organization didn't need to convince people that superintelligent AI was definitely coming; it only needed to convince them that it might come, and that it was worth preparing for that possibility.
Tegmark's own contribution to this intellectual infrastructure was his 2017 book "Life 3.0: Being Human in the Age of Artificial Intelligence." The book, which became a New York Times bestseller, synthesized years of thinking about AI safety into an accessible narrative that explained the stakes for general audiences. Unlike many AI books that focused on either near-term applications or far-future speculation, "Life 3.0" explicitly connected the two, arguing that the decisions made in the present would shape humanity's long-term trajectory.
The Autonomous Weapons Campaign
While building the intellectual framework for AI safety, FLI also worked on concrete policy campaigns. The most successful of these was the campaign against autonomous weapons—military systems that could select and engage targets without human intervention.
In July 2015, FLI published an open letter calling for a ban on offensive autonomous weapons. The letter argued that such systems would lower the threshold for going to war, be vulnerable to hacking and malicious use, and create an arms race that would be difficult to reverse. Within weeks, it had been signed by thousands of AI and robotics researchers, including some of the field's most prominent figures: Stuart Russell, Yoshua Bengio, Geoffrey Hinton, and many others.
The autonomous weapons campaign demonstrated FLI's strategic sophistication. First, the organization chose an issue where there was potential for broad consensus. Many researchers who were skeptical of long-term existential risk were willing to support a ban on autonomous weapons, which seemed like a more concrete and immediate concern. Second, FLI framed the issue in terms of international humanitarian law and arms control, connecting it to existing policy frameworks. Third, they focused on building coalitions with established organizations like Human Rights Watch and the International Committee of the Red Cross.
The campaign had measurable policy impacts. In 2016, the United Nations began formal discussions on lethal autonomous weapons systems, with the Convention on Certain Conventional Weapons establishing a Group of Governmental Experts to study the issue. By 2018, over 30 countries had called for some form of regulation or prohibition of autonomous weapons. While a comprehensive ban has not yet been achieved, the campaign succeeded in putting the issue on the international agenda and establishing norms around human control of lethal force.
For Tegmark and FLI, the autonomous weapons campaign served multiple purposes beyond its stated goal. It demonstrated that AI researchers could organize collectively to influence policy. It built relationships with policymakers and civil society organizations. It established FLI as a credible voice in AI policy debates. And it created a template for future campaigns: identify a concrete issue where there's potential for consensus, frame it in terms policymakers understand, build broad coalitions, and focus on establishing norms and institutions rather than just changing minds.
The Controversy Years
As FLI's influence grew, so did criticism of the organization and its approach to AI safety. The controversies that emerged between 2017 and 2022 reveal both the strengths and weaknesses of Tegmark's strategy, and highlight fundamental tensions within the AI safety community.
One line of criticism came from AI researchers who felt that FLI was exaggerating risks and distracting from more pressing problems. In a widely-shared 2019 blog post, AI researcher Zachary Lipton argued that "longtermist" approaches to AI safety—those focused on existential risks from future superintelligent systems—were drawing talent and funding away from addressing harms that AI was causing right now. Algorithmic bias, privacy violations, labor displacement, and concentration of power were concrete problems affecting millions of people, Lipton argued, while existential risk from superintelligence remained speculative.
This critique intensified after 2020, as concerns about bias and fairness in AI systems became more mainstream. When large language models like GPT-3 were found to generate sexist and racist outputs, some researchers argued that FLI's focus on long-term existential risk had left the field unprepared to address these immediate harms. The organization that had spent years warning about hypothetical superintelligent AI seemed to have little to say about the actual AI systems causing actual harm.
Tegmark's response to these criticisms was to double down on the multi-timescale framework he had developed, arguing that near-term and long-term risks were deeply connected. In a 2021 interview, he explained: "The same technical problems—value alignment, robustness, interpretability—show up whether you're worried about a biased hiring algorithm or a superintelligent AI. Working on near-term safety directly contributes to long-term safety."
A second controversy emerged around FLI's relationship with the effective altruism movement and concerns about "longtermism" as a philosophical framework. Effective altruism, a movement focused on using reason and evidence to do the most good, had become increasingly focused on existential risk as a priority area. Some EA-funded organizations, including FLI, argued that preventing human extinction was the most important moral priority because it preserved the potential for vast future value.
Critics, particularly from the political left, argued that this philosophical framework justified neglecting current injustices in favor of speculative future scenarios. Philosopher Émile Torres coined the term "TESCREAL" to describe what they saw as a dangerous ideology combining transhumanism, extropianism, singularitarianism, cosmism, rationalism, effective altruism, and longtermism. Torres argued that this worldview was "driving the development of dangerous technologies" and "legitimating atrocities in the name of the greater good."
The collapse of FTX in November 2022 intensified these critiques. Sam Bankman-Fried, the disgraced cryptocurrency entrepreneur, had been a major funder of effective altruism and AI safety causes. Revelations about FTX's fraudulent practices led to renewed scrutiny of the entire ecosystem of EA-funded organizations. While FLI had not received funding from FTX or Bankman-Fried directly, its association with the effective altruism movement created reputational risks.
Tegmark's handling of these controversies was pragmatic rather than ideological. In public statements, he emphasized FLI's independence from any particular philosophical movement, its focus on empirical research over speculation, and its commitment to addressing both near-term and long-term risks. Privately, according to people familiar with the organization, FLI began diversifying its funding sources and building relationships with mainstream research institutions and corporations to reduce dependence on the effective altruism ecosystem.
The Pause Letter and Its Aftermath
The release of GPT-4 in March 2023 marked a turning point in public awareness of AI capabilities. OpenAI's system could pass the bar exam, write computer code, and engage in extended, coherent conversations on complex topics. For many people, including many AI researchers, GPT-4 represented a threshold: AI had moved from "impressive demos" to "actually useful systems that might genuinely transform society."
Within days of GPT-4's release, FLI published its open letter calling for a pause on training systems more powerful than GPT-4. The letter argued that recent advances had caught humanity unprepared: "Contemporary AI systems are now becoming human-competitive at general tasks, and we must ask ourselves: Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization?"
The letter called for AI labs to "immediately pause for at least 6 months the training of AI systems more powerful than GPT-4," to be enforced by governments if labs didn't voluntarily comply. During this pause, labs and researchers should work on shared safety protocols, develop robust audit and certification systems, establish liability frameworks, fund research on AI safety and ethics, and build governance institutions capable of managing advanced AI.
The letter quickly went viral. Within a week, it had over 1,000 signatures from AI researchers, technology leaders, and public intellectuals. Within a month, over 25,000 people had signed. Media coverage was extensive: the New York Times, Washington Post, Financial Times, and virtually every major technology publication ran stories about the letter and the debate it sparked.
For Tegmark and FLI, the pause letter represented the culmination of a decade's work. An idea that had seemed radical in 2014—that AI development might need to slow down for safety reasons—had become mainstream enough that over 1,000 credible experts were willing to publicly support it. The intellectual framework FLI had built, the relationships they had cultivated, the credibility they had established—all of it came together in that moment.
But the letter also revealed the limits of FLI's influence. No major AI lab paused development of more advanced systems. OpenAI, Google, Anthropic, and others continued training larger models. Governments did not impose the kind of regulatory pause the letter called for. Many AI researchers who had supported FLI's earlier campaigns declined to sign the pause letter, viewing it as unrealistic or counterproductive.
The response from leading AI researchers was mixed. Yann LeCun, chief AI scientist at Meta, called the letter "preposterous" and argued that the risks were being vastly overstated. Andrew Ng, a prominent machine learning researcher, said the letter was "a huge distraction" from more pressing AI safety work. On the other hand, Yoshua Bengio, Geoffrey Hinton (who had left Google to speak more freely about AI risks), and Stuart Russell all expressed support for the letter's core concerns, even if they had reservations about specific proposals.
What the letter did achieve was moving the Overton window. Ideas that had seemed extreme in 2014 or even 2020—that AI might pose existential risks, that development might need to be regulated, that governments might need to intervene in AI research—were now being seriously discussed by mainstream media, policymakers, and industry leaders. In May 2023, Sam Altman (OpenAI CEO), Demis Hassabis (Google DeepMind CEO), and Dario Amodei (Anthropic CEO) testified before Congress about AI regulation. The European Union accelerated work on its AI Act. The White House announced new initiatives on AI safety and governance.
The Technical Work
While Tegmark is best known for his public advocacy and institution-building, he has also contributed to technical research on AI safety. His approach reflects his background as a physicist: looking for fundamental principles and mathematical frameworks that might illuminate the problem.
In 2016, Tegmark and several collaborators published a paper titled "Concrete Problems in AI Safety" in collaboration with researchers from Google Brain, OpenAI, Stanford, and Berkeley. The paper identified five key technical challenges: avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distributional shift. Each of these problems represented a way that AI systems might behave in unintended ways, even when following their training objectives.
The paper was influential because it translated philosophical concerns about AI alignment into concrete technical problems that researchers could work on. Rather than asking "how do we ensure superintelligent AI shares human values?", it asked specific questions like "how do we ensure that a cleaning robot doesn't knock over a vase while pursuing the goal of cleaning efficiently?" By grounding the alignment problem in near-term, tractable examples, the paper made it accessible to mainstream machine learning researchers.
Tegmark has also worked on questions at the intersection of physics and AI. In a series of papers between 2017 and 2020, he explored how physics principles might constrain or enable AI systems. One paper examined the thermodynamic efficiency of different computing architectures, arguing that there are physical limits to how efficiently any physical system can perform computation. Another looked at whether the mathematical structure of physical laws might provide clues about what kinds of optimization processes can emerge naturally.
More recently, Tegmark has focused on interpretability—understanding what neural networks are actually doing internally. In 2022, he and his MIT research group published work on using techniques from physics to analyze the internal representations of large language models. The idea is that methods developed to study complex physical systems might also help us understand complex AI systems.
The technical work hasn't been without criticism. Some researchers argue that Tegmark's physics background, while valuable, can lead him to overestimate the applicability of physics methods to AI systems. Neural networks are not physical systems in the traditional sense, and importing concepts from physics doesn't always illuminate their behavior. Others note that while Tegmark has contributed to framing AI safety problems, he hasn't developed the kind of deep technical solutions that would come from years of focused work on machine learning.
Tegmark's response has been that his role is not to be the person who solves all technical problems, but rather to help catalyze a research community that can solve them collectively. "I'm not trying to be the world's expert on neural network interpretability," he said in a 2023 interview. "I'm trying to help build a field where there are thousands of people working on these problems from different angles."
The Policy Landscape
By 2024, the AI policy landscape that Tegmark and FLI had worked to create for a decade was finally taking shape. Governments around the world were implementing or proposing regulations on AI systems. The European Union's AI Act, which would impose requirements on "high-risk" AI systems, was nearing final passage. The UK had established an AI Safety Institute to evaluate risks from advanced AI systems. The US had issued an executive order on AI safety and was considering multiple bills to regulate AI.
FLI's influence on this policy landscape is difficult to quantify precisely, but there are clear connections. Many of the researchers who advised governments on AI policy were people who had received FLI grants or attended FLI conferences. Many of the concepts that appeared in policy documents—risk categories, alignment problems, the need for evaluation and auditing—were concepts that FLI had helped develop and popularize. The organization's role was less that of direct lobbyist and more that of intellectual infrastructure-builder.
One concrete example of FLI's policy influence is the concept of "AI safety levels" or "capability thresholds," which appears in several proposed regulatory frameworks. The idea, which Tegmark and others promoted, is that AI systems above certain capability thresholds should trigger additional safety requirements, transparency obligations, and regulatory scrutiny. This framework appears in the EU AI Act's concept of "high-risk" systems, in the UK's approach to "frontier AI" systems, and in various proposed US regulations.
Another area where FLI's work has shaped policy is around evaluation and testing of AI systems. The organization has consistently argued that before deploying powerful AI systems, developers should be required to demonstrate that they meet certain safety standards, similar to how pharmaceutical companies must demonstrate that drugs are safe before selling them. This idea now appears in multiple regulatory proposals, including requirements for third-party audits, red-teaming exercises, and capability evaluations.
But the policy landscape also reveals the limits of Tegmark's approach. Despite a decade of advocacy, there is still no international agreement on AI governance, no binding restrictions on development of advanced AI systems, and no enforcement mechanism to ensure that AI labs comply with safety standards. The regulatory approaches being developed are focused primarily on near-term risks—bias, privacy, misinformation—rather than the existential risks that have been Tegmark's primary concern.
The Corporate Pivot
A less-discussed aspect of Tegmark's recent work is his engagement with AI companies themselves. Starting around 2022, as it became clear that the most advanced AI systems were being developed by private companies rather than academic research labs, FLI shifted strategy to work more directly with industry.
This involved several initiatives. First, FLI organized private workshops bringing together safety teams from different AI labs to share information about evaluation methods, red-teaming approaches, and alignment techniques. These workshops, held under Chatham House rules to encourage candid discussion, created informal networks among researchers working on similar problems at different companies.
Second, FLI funded research collaborations between academic researchers and AI company safety teams. This created channels for academic insights to reach industry, while also giving academics access to large models and computational resources they couldn't otherwise afford.
Third, Tegmark personally cultivated relationships with AI company leaders and safety teams. He met regularly with OpenAI's safety team, consulted with Anthropic on their approach to constitutional AI, and advised Google DeepMind on evaluation frameworks for advanced models. These relationships gave FLI insight into what was happening at the frontier of AI development, while also giving companies access to the broader AI safety research community.
The corporate engagement strategy was controversial within the AI safety community. Critics argued that FLI was being co-opted by companies that wanted the appearance of taking safety seriously without making real commitments. They pointed out that despite years of engagement, major AI labs continued to rush ahead with developing more powerful systems, often without implementing the safety measures researchers recommended.
The tension came to a head in November 2023, when OpenAI's board briefly fired CEO Sam Altman, citing concerns about safety and governance, before reinstating him days later under pressure from employees and investors. The episode revealed deep disagreements within OpenAI about how to balance safety and commercial pressures—disagreements that external advocates like FLI had limited ability to influence.
Tegmark's response was that engagement with companies, while frustrating and imperfect, was necessary because that's where the most advanced AI development was happening. "I'd rather be at the table trying to influence these decisions, even if my influence is limited, than be outside criticizing with no ability to affect outcomes," he explained in a podcast interview.
The Academic Impact
Beyond policy and institution-building, Tegmark's work has had significant impact on academic research. The field of AI safety, which barely existed as a recognized research area in 2014, now has dedicated conferences, academic centers, funded professorships, and PhD programs.
Several major universities have established AI safety research centers, many of which received initial funding from FLI's grant programs. These include the Center for Human-Compatible AI at Berkeley (founded 2016), the Leverhulme Centre for the Future of Intelligence at Cambridge (founded 2016), and the Center for AI Safety (founded 2022). These centers have collectively trained dozens of PhD students, published hundreds of papers, and created academic career paths in AI safety that didn't exist a decade ago.
The research agenda has also evolved. Early AI safety work focused primarily on theoretical problems—how to specify objectives for powerful AI systems, how to ensure those objectives align with human values, how to maintain human control as systems become more autonomous. More recent work has tackled concrete technical problems: how to interpret what neural networks are learning, how to detect when models are behaving deceptively, how to make models robust to adversarial inputs, how to evaluate whether models have dangerous capabilities.
Academic conferences reflect this evolution. The Neural Information Processing Systems (NeurIPS) conference, the largest machine learning conference in the world, added a dedicated track on AI safety in 2020. By 2023, that track was receiving hundreds of paper submissions covering topics from interpretability to robustness to alignment. What had been a niche concern had become a mainstream research area.
Tegmark's own academic work at MIT has focused on building bridges between AI safety and traditional computer science and physics departments. His research group includes students working on everything from cosmology to interpretability to AI governance. This interdisciplinary approach reflects Tegmark's belief that AI safety requires insights from multiple fields: computer science for technical understanding, philosophy for conceptual clarity, physics for mathematical rigor, political science for governance frameworks.
The academic impact is also visible in citation patterns. Tegmark's 2017 book "Life 3.0" has been cited over 2,000 times in academic papers. The FLI-authored research agenda paper "Concrete Problems in AI Safety" has been cited over 1,400 times. These citations span multiple fields: computer science, philosophy, economics, political science, law. The concepts and frameworks developed by Tegmark and FLI have become part of the intellectual infrastructure that researchers across disciplines use to think about AI.
Critics and Controversies
Understanding Tegmark's impact requires also understanding the sustained criticism his work has received. These critiques come from multiple directions and reveal genuine tensions within debates about AI development and governance.
From the AI research community, some critics argue that Tegmark's focus on existential risk is not just wrong, but actively harmful. They contend that it distracts from real, present harms caused by AI systems: algorithmic bias in criminal justice, privacy violations from surveillance systems, labor displacement from automation, environmental costs of training large models. Time and money spent on speculative far-future scenarios, they argue, is time and money not spent on solving problems that affect people today.
This critique has been articulated forcefully by researchers like Timnit Gebru, who was pushed out of Google in 2020 after clashing with management over a paper critical of large language models. Gebru and her colleagues argue that the focus on existential risk serves corporate interests by deflecting attention from the harms that AI companies are causing right now. "When you're worried about some hypothetical future superintelligence," Gebru has said, "you're not worried about the actual people being harmed by the actual AI systems that exist today."
Another line of criticism questions the scientific basis for claims about existential risk. How do we know that advanced AI poses existential risks? What is the probability? How would we measure it? Critics argue that FLI and similar organizations have made strong claims about AI risk based on philosophical arguments and thought experiments rather than empirical evidence. They point out that many predictions about AI timelines and capabilities have been wrong, and question why anyone should believe predictions about even more distant futures.
Some AI researchers go further, arguing that the entire framing of "existential risk from AI" is based on misconceptions about how AI systems work. François Chollet, creator of the Keras deep learning framework, has argued that current AI systems are fundamentally limited in ways that make the superintelligence scenario implausible. "Intelligence is not a single dimension you can maximize," Chollet writes. "The idea that we're on a path to 'artificial general intelligence' that will recursively improve itself to superintelligence is not supported by how AI systems actually work."
From a different angle, some critics worry that the AI safety movement, including FLI, inadvertently serves corporate interests by legitimizing claims about AI capabilities that may be exaggerated. When advocacy organizations treat advanced AI as an existential threat, they implicitly validate companies' marketing claims about building AGI. This can benefit AI companies by attracting investment, talent, and attention—even if the capabilities don't live up to the hype.
Tegmark has responded to these critiques in various ways, depending on the source. To critics focused on present harms, he emphasizes that FLI's framework includes near-term risks and that many technical safety measures address both present and future concerns. To critics questioning the scientific basis of existential risk claims, he points to research on rapid capability gains in AI systems and argues that uncertainty about future risks is itself a reason for caution. To critics worried about corporate co-optation, he acknowledges the tension but maintains that engagement with AI companies is necessary given their central role in AI development.
The Philosophy Underneath
To fully understand Tegmark's approach to AI safety, it's necessary to understand his broader philosophical worldview. His cosmological work and his AI safety work are connected by a set of ideas about intelligence, complexity, and the future of conscious life in the universe.
Tegmark views intelligence as a fundamentally important phenomenon in the universe. In his cosmology work, he studies how a mathematical universe gives rise to complex structures, including self-aware systems capable of understanding mathematics. In his AI work, he studies how intelligent systems can be designed and controlled. Both are ultimately about understanding what intelligence is and how it relates to the physical world.
This perspective shapes how Tegmark thinks about AI risk. He doesn't see artificial intelligence as a threat to "humanity" in some narrow sense, but as a potential transition point in the evolution of intelligence in the universe. If artificial minds eventually supersede biological minds, what matters from a cosmic perspective is not whether those minds are made of carbon or silicon, but whether they preserve what's valuable about consciousness and intelligence.
This "cosmic perspective" on AI risk has been both influential and controversial. It provides a philosophical framework for taking AI safety seriously without resorting to human-centric arguments or appeals to human uniqueness. We should care about AI alignment, in this view, not because humans are special, but because we want to preserve consciousness, choice, complexity, and beauty in the universe—and misaligned AI might not preserve those things.
But critics argue that this cosmic framing can lead to problematic conclusions. If what matters is intelligence and consciousness at the cosmic scale, rather than particular humans or communities, it might seem to justify sacrificing present welfare for future potential. This connects to broader critiques of "longtermism" as a philosophical framework that can rationalize ignoring current suffering in favor of hypothetical future scenarios.
Tegmark's response has been to emphasize that near-term and long-term considerations are aligned rather than in tension. "The things we need to do to make AI beneficial for people today—making it robust, interpretable, aligned with human values—are exactly the things we need to do to make sure the long-term future goes well," he argues. "This isn't a tradeoff. It's the same problem."
The Measurement Problem
One of the challenges in evaluating Tegmark's impact is the measurement problem: how do you assess the counterfactual impact of institution-building and advocacy work? If FLI hadn't existed, would AI safety have developed anyway? Would governments have enacted similar regulations? Would companies have taken safety seriously?
Some impacts are relatively clear and measurable. FLI distributed over $10 million in research grants to 37 different projects, directly funding research that wouldn't otherwise have happened. The organization organized conferences that brought together researchers who might not have met otherwise. Their open letters mobilized thousands of signatures and generated extensive media coverage.
Other impacts are harder to trace but potentially more important. How much did FLI's intellectual framework shape how researchers, policymakers, and the public think about AI risk? How many people entered AI safety as a career because they read Tegmark's book or heard him speak? How many policy decisions were influenced by concepts that FLI helped develop, even if policymakers weren't consciously aware of the source?
One way to assess impact is by looking at the growth of the AI safety field itself. In 2014, there were perhaps a few dozen researchers worldwide working primarily on long-term AI safety. By 2024, there are thousands, working at universities, AI companies, nonprofits, and government agencies. While FLI wasn't solely responsible for this growth, it played a central role in catalyzing, funding, and organizing the field.
Another measure is the shift in mainstream discourse. In 2014, most AI researchers and technology leaders dismissed concerns about AI safety as science fiction. By 2024, major AI companies all had safety teams, governments worldwide were implementing AI regulations, and international organizations were developing governance frameworks. The Overton window had shifted dramatically, from whether AI needed safety measures to what those measures should be.
The regulatory landscape provides another data point. By 2024, multiple jurisdictions had implemented or proposed regulations specifically addressing AI safety concerns that FLI had raised: requirements for safety testing of advanced systems, restrictions on certain high-risk applications, mandates for transparency and explainability. While these regulations don't go as far as FLI has advocated, they represent a significant shift from the largely unregulated landscape of 2014.
The Present Moment and Future Challenges
As of 2024, Tegmark finds himself in a paradoxical position. The AI safety movement he helped build has achieved mainstream recognition and influence beyond what seemed possible a decade ago. But the fundamental challenge—ensuring that increasingly powerful AI systems remain aligned with human values—remains unsolved and may be growing more urgent.
Recent developments have vindicated some of FLI's warnings while also revealing gaps in the AI safety agenda. Large language models like GPT-4 demonstrate capabilities that were unexpected even a few years ago, suggesting that AI progress can be rapid and surprising. At the same time, these systems also exhibit failures and limitations that weren't predicted by theories of AI risk: they hallucinate false information, they can be easily manipulated, they lack robust understanding, they fail in opaque and unpredictable ways.
This creates a challenge for the AI safety framework that Tegmark and others developed. The theoretical models of AI risk focused on scenarios where AI systems reliably optimize for specified objectives—the problem was ensuring those objectives were the right ones. But actual advanced AI systems don't reliably optimize for anything. They're unpredictable, inconsistent, and fragile in ways that don't fit neatly into existing frameworks.
Tegmark's current focus reflects this evolving challenge. He has become more interested in interpretability—understanding what AI systems are actually doing internally—and in evaluation frameworks that can detect dangerous capabilities before systems are deployed. He has also pushed for more empirical research on AI risks, arguing that the field needs to move beyond philosophical thought experiments to study actual systems.
The institutional landscape is also shifting. With AI companies now spending billions on their own safety research, the role of external advocacy organizations like FLI is changing. FLI is less needed to make the case that AI safety matters—most people now accept that—and more needed to provide independent evaluation, to represent perspectives that might be ignored by profit-driven companies, and to push for stronger governance structures.
Looking forward, Tegmark identifies several priorities for the AI safety field. First, developing robust evaluation methods that can assess whether AI systems have dangerous capabilities before those systems are deployed. Second, creating governance structures—both within companies and in public policy—that can actually slow down or stop AI development if necessary. Third, solving technical alignment problems, particularly around ensuring that powerful AI systems do what humans actually want rather than gaming their objectives.
Fourth, and perhaps most challenging, is building international coordination on AI governance. The competitive dynamics of AI development create pressure to move quickly and take risks—if one lab or one country slows down for safety reasons, another might race ahead. Effective governance likely requires international agreements and enforcement mechanisms, similar to nuclear weapons treaties or climate agreements. This is an area where FLI and similar organizations have made limited progress.
Lessons From the First Decade
Tegmark's decade-long effort to build the AI safety field offers several lessons about how scientific and policy movements develop, and about the challenges of governing transformative technologies.
The first lesson is about the importance of intellectual infrastructure. Before FLI could influence policy, it needed to develop concepts, frameworks, and language that different stakeholders could use to think about AI risk. The organization spent years on this intellectual work—writing papers, organizing conferences, publishing books—before engaging directly with policy. This groundwork made later policy advocacy more effective because policymakers had a ready-made framework to work with.
The second lesson is about coalition-building. FLI's most successful campaigns, like the autonomous weapons initiative, succeeded because they built broad coalitions that included mainstream researchers, established civil society organizations, and diverse political constituencies. Campaigns that remained narrower—focused primarily on existential risk or long-term concerns—had less policy impact, even when the underlying arguments were strong.
The third lesson is about timing and opportunity. FLI was founded at a moment when AI capabilities were beginning to advance rapidly but before most people recognized the implications. This gave the organization several years to build capacity, develop frameworks, and establish credibility before AI safety became a mainstream concern. By the time AI safety hit the public agenda in 2022-2023, FLI was positioned to shape the conversation.
The fourth lesson is about the limits of advocacy. Despite FLI's influence, the organization has not achieved its core goal: ensuring that AI development proceeds cautiously with robust safety measures. Major AI labs continue racing to develop more powerful systems, often without implementing safety measures that researchers recommend. Governments have been slow to regulate, and when they have acted, their regulations focus more on near-term harms than existential risks. The competitive dynamics and economic incentives driving AI development have proved difficult to alter through advocacy alone.
The fifth lesson is about the challenge of maintaining credibility while advocating for potentially unpopular positions. FLI has had to navigate between different audiences with different concerns: researchers who want technical rigor, policymakers who want concrete proposals, the public who want to understand what's at stake, and AI companies who have the power to implement (or ignore) recommendations. Maintaining credibility with all these groups while pushing for significant changes to AI development practices has been a constant challenge.
The Counter-Narrative
To fully understand Tegmark's work, it's important to consider the counter-narrative: what if the AI safety movement, despite good intentions, is fundamentally misguided? What if the focus on existential risk is not just premature but actively harmful?
This counter-narrative argues that by treating AI as an existential threat, the AI safety movement has created several problems. First, it has distracted attention and resources from addressing current harms caused by AI systems. Algorithmic bias, surveillance, labor displacement, and concentration of power are real problems affecting millions of people today. The focus on hypothetical future risks may have slowed progress on these immediate issues.
Second, the existential risk framing may have amplified AI hype and benefited AI companies. When prominent researchers and advocacy organizations treat advanced AI as an existential threat, they validate companies' claims about building transformative technology. This attracts investment and talent to AI companies, potentially accelerating the very risks the safety movement aims to prevent.
Third, the AI safety movement may have contributed to an oversimplified public discourse about AI. Complex socio-technical problems—about power, governance, justice, and the distribution of benefits and harms—get reduced to a single technical question: is AI safe? This framing may obscure more important questions about who controls AI, whose values it embodies, and who benefits from it.
Fourth, some critics argue that the focus on alignment and control assumes that the goal is to ensure AI systems do what humans want them to do. But this may be asking the wrong question. Perhaps the goal should be to ensure that AI systems serve human flourishing more broadly, which might sometimes mean not doing exactly what individual humans want.
This counter-narrative doesn't dismiss all of Tegmark's work. Even critics acknowledge that he has helped create a community of researchers thinking seriously about AI risks, that some technical AI safety work addresses real problems, and that governance of powerful technologies is important. But they question whether the existential risk framing, the focus on long-term scenarios, and the philosophical framework of the AI safety movement are the right approach.
Tegmark is aware of these critiques and has adjusted his approach in response. In recent years, he has emphasized the connections between near-term and long-term safety, highlighted FLI's work on current AI harms, and been more explicit about the uncertainties involved in predicting AI trajectories. But fundamental tensions remain between those who see existential risk as the primary concern and those who focus on current harms and power structures.
Conclusion: The Architect of Uncertainty
Max Tegmark's decade-long campaign to make AI safety a priority has succeeded in ways that would have seemed impossible in 2014. A concern that was then dismissed as science fiction is now taken seriously by governments, corporations, and research institutions worldwide. Thousands of researchers work on AI safety. Multiple countries are implementing regulations. AI companies have safety teams with substantial budgets. The intellectual infrastructure that Tegmark and FLI helped build—concepts, frameworks, research agendas—is now part of how the field thinks about AI.
Yet the fundamental challenge remains unsolved. We still don't know how to reliably align AI systems with human values. We don't have governance structures capable of managing AI development globally. We haven't resolved the competitive dynamics that pressure companies and countries to race ahead with powerful systems before safety problems are solved. The question Tegmark has been asking since 2014—how do we ensure that increasingly powerful AI systems remain beneficial?—is still open.
Perhaps Tegmark's most important contribution is not any specific technical insight or policy proposal, but rather his role in transforming AI safety from a fringe concern into a legitimate field of inquiry. He helped create space for researchers to work on long-term AI risks without being dismissed as alarmists. He built institutions that could fund research, convene experts, and engage with policymakers. He developed language and concepts that allowed people with different perspectives to have productive conversations.
In this sense, Tegmark is less a solver of the AI safety problem than an architect of the infrastructure needed to work on it. He has helped build the community, institutions, and intellectual frameworks that give humanity a better chance of navigating the AI transition successfully. Whether that will be enough remains the greatest open question of our time.
The physicist who spent his early career studying the largest questions in cosmology—the structure of the universe, the nature of reality, the possibility of parallel worlds—has spent the last decade on what may be an equally fundamental question: how to ensure that intelligence, in whatever form it takes, continues to preserve what matters about conscious experience and the capacity for understanding. The answer to that question will shape not just the coming decades, but potentially the entire future history of intelligence in the universe.