Over Optimization Is Killing Everything You Love

How optimization is destroying sports, art, your bank account, and more.

Recently I went to a museum in Abu Dhabi, where I made friends with a stray cat and had coffee with a pretty neat view.

The cat and the coffee.

Just five feet beyond the border of these photos, I saw a woman posing in front of the museum at the edge of the water, making her husband take no less than a thousand photos of her.

The woman went through a series of predetermined poses in rapid fire sequence. It looked like someone with Parkinson’s trying to do a Bollywood dance in fast-forward. She would go through this routine, walk to her husband, review the photos, and proceed to berate him for getting something wrong. She would then return to her starting position and go through the whole routine from the beginning, looking increasingly upset every time.

After about forty minutes she had all the pictures she wanted, so the two of them finally left, arguing and bickering all the while.

I was watching the woman out of the corner of my eye the entire time, curious as to whether she would ever actually turn around and look at the museum and the waters with her own eyes. She never did.

---

It seems like various aspects of our existence have cracked in some meaningful way, leaving us perpetually disaffected.

Despite the plethora of entertainment, we are more bored than ever. Despite the many communication channels with which we might reach out to each other, we are lonelier than ever. Despite advances in biology and physiology, we are more obese and more sleep deprived than ever. Despite having greater access to the world and all of its various cultures, we are becoming more homogenized than ever.

This essay examines all of these phenomena at a deeper level, showing how much of the malaise, boredom, and maladaptions throughout society can be neatly approximated to a single word: optimization.

Optimization is an oracle who gives with one hand and takes with another; usually the things that it provides are clear and tangible, with immediate effects, and the things that it takes are slow but long-lasting, usually of greater value than we had originally given them credit for.

Examples Of The Optimization Crisis

Basketball

Pictured below is an evolution of where NBA players take their shots.

Statisticians realized that three point shots have a much higher return on investment; and as a result, teams have effectively min/maxed their shooting strategies. Unfortunately, while this strategy maximizes the number of points that are scored, it leads to a much lower variety in gameplay, and a much more boring product overall.

When you couple this with byzantine and extractive subscription services, the net result becomes a substantial drop in viewership in recent years.¹

Baseball

Similarly, the MLB is suffering from a lack of viewership. Four out of the five lowest viewed World Series were within the last six years.

A few years ago, MLB rolled out a series of rule changes in order to make the game faster paced and more interesting. While it has worked to some extent, it’s clear that advanced statistical metrics reign supreme over traditional gameplay, which ultimately prioritizes boring yet reliable gameplay over highlights.

These two videos analyzing the gameplay of Luis Arraez and Derek Jeter are good examples of what I mean.

Arraez would have previously been hailed as a batting champion for his ability to hit the ball and get on base, although in the modern era he is seen as equivalent to someone who walks a lot. Derek Jeter is known for a number of iconic plays at shortstop, but the advanced metrics show that he is a subpar fielder.

Statistics now run the MLB, largely as a result of a new style of analysis made popular by the former Oakland Athletics—known as Moneyball. Ironically, the popularity of the 2011 movie of the same name starring Brad Pitt forced the majority of teams to adopt a similar style of analysis, which meant that the Oakland Athletics could no longer compete using this method. As of 2024, Oakland has lost their baseball team, as it is currently in transition to move over to Las Vegas.

Stock Investments

The United States, the top seven companies in the S&P 500—all of which are tech companies—comprise nearly 40% of the overall Market Capitalization. This means that the majority of stock portfolios—traditional index investments and pension funds—are disproportionately reliant on the performance of these tech companies. This leaves them open to sector volatility; as evidenced by the steep single day drop in value when the DeepSeek R1 model was released on the App Store.

With such concentrations of wealth, smaller family-owned businesses are forced to compete with much larger conglomerations such as Walmart and Amazon. Since they don't have economies of scale, they eventually lose out, creating a homogenized retail environment.

Book Publishing

Two works of literary fiction that I read recently were The Castle by Franz Kafka, as well as Moby Dick by Herman Melville. Aside from the language used and the nature of the plot, it is clear these works were written in a time when publishers didn’t impose an optimization paradigm on the writer.²

In Moby Dick, it takes over 20 chapters for Ishamel to actually hit the sea. In chapter 32, Melville lists off every different type of whale, explains the type of oil they produced, and how this oil is used in everyday life. As for The Castle, the final manuscript was published without a proper ending; the book literally stops mid sentence, because Franz Kafka gave up on the novel entirely!

“She held out her trembling hand to K. and had him sit down beside her, she spoke with great difficulty, it is hard to understand her, but what she said…” — The Castle by Franz Kafka

Even in genre fiction, such as science fiction and fantasy, we see the monoculture taking over. In previous decades, authors like Tolkien had the freedom to meander with his plot and include many verses of songs. Because of the relatively few major publishers, a small number of authors tend to be over-represented in the distribution cycle—authors who produce reliable yet relatively safe and formulaic content, such as Brandon Sanderson.

Considering Sanderson is a Mormon, there are very few instances in which any of his characters are morally complex, as most of them are prototypical heroes and villains. Additionally, there is a writing trope specifically named after Sanderson, known as the “Sanderlanche” where a number of the plot threads he has been building up predictably come together in the last hundred or so pages.

Movies/TV

Speaking of Tolkien, a fitting example of the degradation in cinema quality arises when comparing the original Lord of the Rings trilogy to The Hobbit.

The original trilogy is considered one of the greatest series of all time, largely due to the great respect Peter Jackson had for the original books, coupled with the use of practical special effects. Meanwhile, The Hobbit suffered from its over-reliance on CGI, as well as the unnecessary expansion of the book’s short plot into three separate films, requiring low quality filler and extra story lines. Ironically, the central theme of The Hobbit franchise was to show the pitfalls of prioritizing personal greed over everything else, while the movies themselves suffered from optimizing for what was thought would lead to box office success over artistic integrity.³

This particular clip of Ian McKellan breaking down while filming The Hobbit is moving, considering he had to act in a version of Bag End which was entirely green screen.

More generally, we can see a lack of originality in the film industry as movie studios tend to disproportionately rely on existing intellectual properties. In 2024, not one of the top 10 grossing movies was original, all of them being sequels, prequels, or adaptations of existing franchises. Even in the part of Hollywood that is supposed to prioritize artistic vision over commercial success, such as the Academy of Motion Pictures, we see that the opinions of these supposed cultural tastemakers have become uncoupled from the general audience, as evidenced by the disproportional degree of Oscar nominations Emelia Perez received despite the film being just plain bad.

Aesthetics

I’ve previously written about how everything has gotten uglier, and how this has a greater impact on society than we might expect as beauty serves as a gateway to meaning and transcendence. Many facets of our lives—from the architecture of the buildings around us, to the interfaces of the online landscape, to the beauty of people themselves—have degraded in comparison to previous generations.

Taking the example of architecture, buildings have now been prioritized to meet a functional goal, which leads many cities and towns to have a homogenized and drab look—a never ending series of strip malls with the same 5 or 10 fast food restaurants on every block.

Academia

Private higher education institutions are increasingly turning away from their original goal of producing an educated population. Rather, they have turned into proprietors of large endowment funds, and increasingly apply pressure on their professors to “publish or perish.”

This means that researchers are becoming risk averse in the type of papers they publish, prioritizing their H index over pursuing work that might have a more meaningful impact or lead to revolutionary insights. Pursuing something truly revolutionary means that there is a greater risk of outright failure, which means that their research will not be published, and it will generate less citations. Unfortunately, citations have become a common metric by which the credibility and reputation of academics are measured.

This prioritization can have lasting effects. For example, the man who was largely responsible for discovering the Higgs boson (a subatomic particle), Peter Higgs, admits that he wouldn’t have been considered productive by modern academia. Had he entered academia today, the world would have been deprived of his Nobel Prize-winning research. More generally, the optimization mentality is hypothesized to be a main factor contributing to the lack of innovation in the sciences.

Dating And Relationships

Friendships, though incredibly valuable, are difficult to quantify or monetize—and as a result it’s always easier to set aside these relationships in favor of career aspirations, or simply getting by.

Romantic relationships have been commodified as well. Dating apps take the various intricacies of our lives and personalities, and collapse them down into simple metrics, often extremely superficial in nature. Additionally, while dating apps don't necessarily have a direct incentive to keep you single, their business model can certainly benefit from this.

This deterioration of our social fabric is nothing new. It’s been taking place for several decades now, as discussed in the book Bowling Alone by Robert Putnam. But COVID added an accelerant to the phenomenon, as many of the lockdown measures proved deranging for various subsections of the population, especially the youth, who experienced a significant disruption in their education as they were forced to learn remotely. Additionally, many of the extracurriculars that would have put them into contact with their peers were canceled, leaving them alone for extended periods.

Social isolation has gotten so bad that the surgeon general of the United States, Vivek Murthy, declared loneliness to be a health crisis.

Optimization Is A Monkey's Paw

The monkey’s paw is a twist on the old “Genie gives you three wishes” story, where the main character gets what he wants, but ultimately in a way that makes him worse off. One of the examples described in the story is when the protagonist asks for a sum of money, and receives it as a settlement payment after his son dies in a horrific accident.

The optimization crisis is ultimately the monkey’s paw, applied at a societal level.

My fellow Substack writer Dmitry eloquently sums up the issue in his post, attributing this crisis to a phenomenon which he calls “spreadsheet brain”—that all the meaning can be adequately captured in an Excel spreadsheet, and that all of the biggest decisions in life can be made through a utilitarian framework—a carefully arranged list of pros and cons.

But at the risk of stating the obvious, when you optimize for something, you optimize for something.

As I’ve written about before, there is a god, and his name is "Trade-Off." Under this paradigm, optimization is nothing more than the ruthless prioritization of one thing over everything else. This ultimately serves to increase some metric or another, however, it leads to a total collapse of meaning, of beauty, and of all the other things that we hold dear—simply because they are difficult to collapse into a line item on a spreadsheet.

All the true intricacies of our nature gets stripped away, until we are left hollow and utterly alienated. We find ourselves lost, much like the protagonist in The Castle by Franz Kafka, when he is told: “You are not from the Castle, you are not from the village, you are nothing at all.”

Ultimately, we all turn into the woman standing in front of the museum in Abu Dhabi, yelling at our metaphorical partners to capture the perfect photo, though never turning to look around and appreciate the beauty right in front of us.

The Louvre Abu Dhabi art museum.

1. Another factor supposedly impacting NBA ratings is the explicit liberal bent that many of the owners/players exhibit, which has alienated conservative fans. This alienation is hard to quantify, so I have excluded it from my analysis

2. The publishing industry in those days had its own constraints. For example, one of the reasons why older books are so long is because they often got paid by the number of pages they wrote

3. With all credit to Peter Jackson, some of the issues were not his fault, as he suffered from health issues during production and had to take over the project at a late stage from its originally intended director Guillermo Del Toro.