With search getting worse by the day, maybe it’s time we rebounded in the other direction. The long forgotten directory.
Search is getting so much worse. AI hallucinations are becoming a major problem. Finding things is such a fraught and hostile experience. An unnerving journey of hacking through AI slop and low quality “optimized” text, funneled first into results and second into the content of the websites you might actually be able to see in the results, if you look long enough.
It’s gotten so bad that some people have tried clever hacks to get around it. Search engines don’t want you to visit websites anymore.
And that got me thinking about directories.
Directories go out of fashion
Jerry Yang and David Filo started Yahoo! with a web browser and a list. They’d surf the web endlessly, finding websites that brought them joy or piqued their interest. And they’d sort them into distinctive categories invented on the fly.
When you visited Yahoo!, you didn’t always know what to expect. That was part of the fun. You would come to the site and explore. Maybe you’d click from one category to another, or line up a dozen websites across your tabs. It was sometimes hard to find what you were looking for, but sometimes you happened upon a brand new discovery, or ventured down a path you hadn’t anticipated.
Eventually the duo needed help to keep up with a web exploding into the mainstream. They hired Srinija Srinivasan to put together a team of curators, which Yahoo! referred to as “surfers.”
At the time, searching the web was not a great experience. Search crawlers, like AltaVista or Excite, were too easy to manipulate. Spammers had already figured out how to fill results via hidden keywords, and artificial content, and dozens of other dark patterns. Results were too often low quality, as search engines tried to figure out how to rank things properly.
Surfers were a natural antidote to the problem. They were real people who could easily see through the bullshit. Yahoo! gave them a lot of leeway to organize things into collections and rank and organize results. And they were mostly up to the task. At Yahoo!’s peak, surfers were adding a thousand websites a day to the site. Each. Being powered by humans gave Yahoo! an edge. It made them a relatable path of discovery.
Even in those early days, maybe especially in those early days, the categories and selections carried with them a point of view. It’s what Filo would often refer to as the “voice of Yahoo!”. Surfers didn’t shy away from embracing that voice, as long as the criteria was fair. It’s actually what kept people bringing back, eager to discover and explore areas of interest being thoughtfully arranged by real human beings who seemed like they cared.
Eventually, the copy cats came in the form of portals. This led to a decade long war as company after company launched their own portals trying to grab their own slice of internet traffic. But unlike Yahoo!, this new brand of directory didn’t want people to leave. They stuffed their site with news and suggestions and videos. But without Yahoo!’s surfers, and without an actual tether to the web, most fell soulless. They lacked the voice that had made Yahoo famous.
Google developed Page-rank, and made ranking search results far more accurate. By then, directories like Yahoo! had been so stripped of personality and utility that it wasn’t even particularly hard to take them down. In the 2000’s, search won out.
Becoming Surfers Again
Twenty-five years later, search appears doomed to repeat the mistakes that had allowed for their success. Search has handed over their voice to toneless, AI-generated results. They have made it their explicit goal to ensure that people never leave, never actually explore the web. Websites are desperately trying to game the system again, so that they might be able to catch a sliver of traffic from these low quality results.
Search has bent in quality towards its earliest days, difficult to navigate and often unhelpful. And the remedy may be the same as it was a quarter century ago. It may be once again time for the surfers. Only different this time.
The only resistance to the current advance of AI is humanity. We are relearning that humans can see through the bullshit. There are examples everywhere.
There’s a reason, for instance, that the last refuge of actual information for search is on Reddit (though they are adding AI now as well). It’s a common pattern to simply add “reddit” to searches to find something that human beings are actually discussing. It’s the same reason that a new generation is turning to Tumblr, shielded for the most part from the lack of humanity seen everywhere else.
There’s an appetite for discovery of the web, driven by humans. This type of discovery is typically referred to as curation. But curators cull lists down to only the most essential finds. Surfers cast a wider net, and they may be better suited for the current moment. They find things, organize them, and share them with a world. They have a point of view.
On the modern web, a few dozen surfers in some office working day and night to find things and organize them won’t work anymore. The web is simply too big. But communities on the web are also embracing decentralization. It’s ok for surfers to be somewhat fragmented.
Instead of one directory, I think we will begin to see many. The excitement around a recent redesign of Ye Olde Blogroll at least begins to make that point. There is something positively thrilling about a list of websites assembled by actual people for your delight. Those with the will to do that are our surfers. And thanks to decentaralized tools, I think there can and will be many, operating independently from one another.
I think that directories can rise again, powered by surfers and backed by a point of view.
I’ve felt that we could benefit from the return of directories for a long time, but I would rather go with something like DMOZ than chance what happened to Yahoo before it died (it started charging for submissions and the quality fell of a cliff.)
I wouldn’t even mind seeing a revival of something like The Mining Company (which later became About) which could mix expertise and published content within directories/subdirectories.