Jason Calacanis: Hey everybody, happy Monday. Jason is still away. If you follow him on Instagram, you can see his ski adventures in Japan. It's incredible. I am back though today with another great interview. Today on the show, I'm joined by Sridhar Ramaswamy, the CEO and co-founder of Neva, an AI-powered search engine and chatbot. It's private, it's AI-powered search, but it's a very, very different model from OpenAI, ChatGPT, Google's Bard, and Sridhar himself is a long-time Googler who's trying to find a different way forward in the search world. It is a fascinating conversation. You do not want to miss it. It's going to be a great show. Stick with us. This Week in Startups is brought to you by Microsoft for Startup Founders Hub. It helps all founders build a better startup at a lower cost from day one. Startups get up to $150,000 in Azure credits, access to open AI APIs, free dev tools like GitHub, technical advisory, access to mentors and experts, and so much more. There is no funding requirement and it only takes minutes to join. Sign up today at aka.ms slash thisweekinstartups. LinkedIn Marketing. To redeem a $100 LinkedIn ad credit and launch your first campaign, go to linkedin.com slash thisweekinstartups. And Pilot. Grow your business sustainably and operate more effectively. Pilot provides the most reliable accounting, CFO, and tax services for startups and small businesses. Head to pilot.com slash twist and get 20% off the first six months. All right, everyone, I am having a great day. I am delighted to be joined by Sridhar Ramaswamy, the co-founder and CEO of Neva, the next generation search engine that is using the power of AI to just give us answers with maximum privacy and efficiency, and that I personally am already willing to pay for. Welcome to the show

SPEAKER_04: and thanks for coming on. Thank you, Molly. Thank you, Jason. Super excited to be here.

Jason Calacanis: So you probably heard us, everybody, audience, talk about Neva back on episode 1674. Sridhar is also a former SVP of Google's ad business, currently a VC at Greylock. And if you wouldn't mind, I guess, let's start for those who may have missed that episode. I hope no one did. Tell us what Neva and

SPEAKER_05: Neva AI is and what you're building. And also, are you building that while you are a VC at Greylock?

SPEAKER_06: Um, I am, I'm a venture partner. So it's very much a part time job. I'm on a board of like a synthetic data AI company on Greylock's behalf. Most of my time is spent at, at Neva. Yeah, just unwinding a little bit. I was at Google for close to 16 amazing years, early part of the search ads team, but sort of grew with Google to run the ads and commerce teams. Left about four years ago with my co founder, Vivek,

SPEAKER_09: um, and we embarked on this crazy mission of reimagining search. Um, you know, we love the problem of search, but we just felt like Google was trapped in its own success. And so Neva is about rethinking, you know, what is a core product, you know, core part of our daily lives, most of us search without even thinking about it. So we wanted to create a product that was all about the user, um, all about actually just, you know, creating a great product serving. So that's why we had an early focus on privacy on being ads free. Um, but just as importantly, we thought we could build a better product. Like privacy is not a product. Privacy is like, it's a feature of a product. So over the first three years of Neva, we've been busy building up a search stack, running a crawl at internet scale, building the search system widely acknowledged to be one of the hardest problems out there. And some nine, 10 months ago, we saw, you know, early versions of things like GPT three, and realized that this was a magic power that we could harness in the context of, of search. And so we've been working on deeply integrating language models into into search. Um, but, um, you know, while chat GPT took the world by storm, we again took a contrary in approach and said language models should, should enhance search things that you and I love about search, which is you want quick, authoritative information, you want timely information. He said, we want those characteristics to be present. So we've been working on this technique to basically run a search engine and language models in parallel. Um, and while there's work to do, we're pretty proud of what we have accomplished as a 50% team, um, being able to generate, um, answers, fluid responses to more than half the queries that people put into Neva. And we expect that number to keep growing with time, we are pretty confident about showing you answers for 75, 80% of, um, of queries. So fundamentally, Neva is now sort of, again, reinventing itself to be an answer engine, um, but provide you with reliable, cited, timely information about things that you might be interested in.

Jason Calacanis: So you were sort of unpack some parts of that that gets into my next question, which is sort of, what's the technological underpinning? Are you using the kind of large language learning models that we keep hearing about with GPT-3? It sounds like you're not using that exclusively, you're combining that with some special sauce to make sure that the answers are more than maybe what we have seen recently from being in chat GPT. Like, turns out, it's not all as accurate as one might have imagined.

SPEAKER_16: That's right. That's right. So, you know, chat GPT is what I would describe as like open loop, meaning that it is in purely

SPEAKER_09: generative mode. Now, these large language models have been trained pretty much on every document that exists in the world, but they don't understand things like provenance. When you and I take a course or when you and I want to learn about something, we pay attention to, well, who is authority on this topic? Who is saying what? Who should we believe? Like, you know, we go through a process of constructing our belief systems. Um, but what chat GPT doing is it's just like understood literally every sentence on the planet, but at a superficial level. Um, and if you ask it a question, it'll start saying things. Um, some things will be true, but some things will just not be. What we do is, um, you know, when you, when you put in a query, we run the search stack and what that gives us is, um, are things like, you know, we know what the authoritative sites are, you know, that the New York Times is more believable for news than someone's blog. Um, and, uh, we then take the, the top results for that query, look at the contents of those pages. Often it's a lot. We extract, summarize portions that are relevant to the query that you have. And then we do a second level of summarization, which is what generates the answers, but it's basically constrained generation. We try very hard to make sure that these models are not in this open loop mode where they are making up things. And we try to, um, you know, we would rather not give you an answer that's wrong. Um, then, you know, like, like, we don't want to be in the business of providing bad answers. So we refrain from answering questions that we are not really sure about. We do get things wrong. Jason pointed out earlier this morning that if you ask us about how the next are doing this year, um, we take old articles to a certain extent, this is like a problem with how search engines have operated, where they returned results for every query, no matter what, um, the fact of the matter is like, there are very few articles that are talking about how the next are doing this year. Um, but we just picked like the top ones that there are some of them are from last year. And we don't understand, like, in the, the temporal meaning of this query. And that's why we generate a bad answer. These are things that the team is busy fixing. But this combination of a search engine and a generative model, we think provides a good balance for making AI useful.

Jason Calacanis: Um, without exactly knowing the baseline for this question, how hard is this to build? If the baseline is, I don't know, right? Zero to 10, or if it's, if the baseline is Google to the moon, like how hard is it to integrate? I mean, search technology already, all of that signals processing, all of that determination of noise versus value, plus integrating these language models, which are somewhat new, at least

SPEAKER_22: this must be like, you must have a really good team. Well, we have an amazing team. It's very small. It's a

SPEAKER_09: little over 50 people. But, you know, we've been working at search for a while, we have many of, you know, the brilliant engineers that help build search that are part of our team. Search is widely acknowledged to be one of the hardest problems to crack. Because it's just runs at a scale that is like not really fathomable to most, to most people. And then integrating the large models. Um, also doing all of these on a budget that works for us. Now, don't get me wrong, we are a well funded startup, but we are not open AI, we are not being, we don't have, you know, hundreds of millions of dollars to throw out the problem, our annual infrastructure budget is less than $10 million. Um, so part of what we have had to do is be very inventive, um, about distilling models, about making them smaller, how do you get more bang for the buck? Um, so we pre train a lot of our own models, we fine tune them. And so a lot of effort and sweat has gone into making this work at scale, but within the constraints of how much budget we, we have. And a lot of it goes back many years, you know, from early on, we've been building a search stack, what we release would not really have been possible. If you are just like a thin shim on top of someone else's API, at this point for pretty much any meaningful page on the planet, we can generate summaries for you. And we will soon have some of these pre computed. So it's, it's, it's a very technical problem. But obviously, the result is a simple, easy to understand, hopefully brief answer that gets to the heart of what you're looking for.

SPEAKER_26: Doing more with less is more important than ever, you know, this, especially for startup founders,

SPEAKER_28: we all have to be efficient. And if you're running a startup, I want you to know about the Microsoft for startups founders hub. It's a no brainer, they're going to help you scale efficiently while preserving your runway, how are they going to do that? Well, they're going to do it with the best startup program I think we've ever seen up to $150,000 in Azure credits plus access to open AI APIs. Think about that, as well as the new Azure open AI service. And listen, there's other things that Microsoft has available to you as part of this program. How about free access to GitHub and Visual Studio? How about one to one technical advisory and expert help on topics like product fundraising and go to market? How about access to a network of mentors that are plugged into the startup world? Plus free access to partners like LinkedIn and bubble, which can help you build your MVP even quicker. Microsoft so generous, the Microsoft for startups founders hub is open to everyone. Whether you're in the idea phase, or you're further along, there's no funding or requirement. They want any founder to be able to get access because they want all founders to succeed. It's really a no brainer takes five minutes to apply. And startups can get massive benefits immediately. Go to sign up right now. Let's take a pause and write this down. Everybody aka.ms slash this week in startups. Like I said, most generous program I think we've ever seen on this

SPEAKER_32: podcast, aka.ms slash this week in startups. Thanks again, Microsoft.

SPEAKER_33: Um, let's break down your we're talking about sort of parallel technologies working together and then

Jason Calacanis: parallel sets of problems to solve with each one of those technologies. So let's go back to Google for a minute and talk about the problems with search that you wanted to solve and dive into those a little bit more. We can start with privacy, but it's a longer list than that. Yeah, so I would broadly put

SPEAKER_16: them into two buckets. And the irony does not escape me that I certainly had a large role to play in one of

SPEAKER_22: those buckets. One is that ad load just keeps going up. Text ads are one of you know, it's pretty much the

SPEAKER_09: most incredible business invented. No one in their right mind for like the first 10 years of this decade thought that Google would make more than 100 billion dollars of revenue just in search ads. It's remarkable. But that's the business it is. Um, and it's also a reflection of our times that, you know,

SPEAKER_22: there's no limit to expectation while Google is a very successful business, it is still judged by how

SPEAKER_09: much it can grow. And so, um, as I said, I approved many of the decisions that increased ad load. But I also felt ultimately that there was no limit to it. It would just keep going up and up unless some external externality like the one we are, you know, going through happens. And then on the organic side, um, you know, they are hemmed in their own way by the ads model. Um, the, the dirty secret behind the ads model, and this actually goes back to even TV is that ads have to stand out. And so broadcasters, for example, would increase the volume of the ads as they started playing, because they knew that that would get a little bit more attention. Similarly, you can't make organic search too attractive, because that's going to kill how much money you make on the text ads. It's the same reason why on your Facebook feed or Instagram feed, you're going to see video ads interspersed with mostly pictures, because moving video gets more attention than a static picture. It's the same theme. And so it's hard for the organic team to say, we can create the best product that there is, while they're constrained by but don't kill ads monetization too much, right? Um, these are some of the problems the ads ecosystem, you know, with things like the the double click acquisition, essentially Google became like this purveyor of ads for the entire internet. The other side of ads is measurement. His conversion tracking is basically measuring to see which ads are successful in getting the customer to take the action that you want. And that sort of unleashed a whole ecosystem of 1000s of companies keeping track of every single thing that you and I do. And Google's a big part of that. And that's what people talk about when they talk about this rampant loss of privacy. Every single thing we do goes into all of these databases to be used for sort of monetizing at a letter at a latter time. And so you know, a lot of Neva is really about, okay, start with a clean slate, start with no constraints, focus on the product, what can you create?

Jason Calacanis: Um, how let's there in terms of Google, there are also sort of other maybe complaints about search, like the idea of a filter bubble that is created almost as a result of that data collection. Is that something that Neva has an opportunity to avoid? I mean, there still is even some controversy around ranking and ranking pages and the idea like there is somebody out there who just heard you say that the New York Times is a trusted source who is losing his mind, which haven't helped us, but

SPEAKER_22: still it's happening. Yeah, I think honestly, that filter bubbles are less of an issue in a search based system, because it's not a recommendation system, right? YouTube definitely has filter bubble

SPEAKER_09: issues. But if you start on a topic, I'm sure you'll run into this, all of a sudden, there'll be a whole bunch of exciting material about that topic that start appearing in your feed. And it can be a vicious cycle. My mom, who is 80 years old, got super energized about one of our last two

SPEAKER_22: presidential candidates for reasons I simply did not understand. I'm like, how do you know this? And

SPEAKER_09: how do you have these opinions? You barely watch anything outside of religious videos. It turns out that she was in a loop that fed her a bunch of videos from one candidate. And so that obviously is extreme. But a lot of us fall victim to these subtle ways in which content that we see is, is engineered. And the way we think about this is we give control back to the user. So as part of Neva's design early on, we said, you can express your preferences for which sources you prefer, we don't think we should be in the business of deciding that one newspaper is better or more trustworthy than others, if it roughly looks like, you know, they are, you know, they are high quality. We've also launched features like bias buster, which is a fun little feature that lets you see viewpoints from different ends of the spectrum. So you can, you know, if you search for something political, you'll see a slider, and you can decide, ah, I want more, you know, right wing opinions on this, or I want more left wing opinions on this. And we change the news results based on based on what you pick. So our approach to a lot of this is to return choice back to you. People will, you know, did ask us for features like not having Amazon or not having Walmart in their search results, because they wanted to shop from the little retailers out there. And ad supported search engine cannot do these things. On the other hand, a customer supported search engine, you know, we are all about serving customers. So we are like, Oh, you don't want that particular site, that's fine, we'll give you all the others. And so very much an aspect of Neva is this personalization is giving that control back to you. But doing that again, with with privacy in mind, we don't track your search queries by default, there's like, you know, there's no list of queries that you have issued. Search is a deeply personal product for most people. And we want to respect that.

SPEAKER_28: Let's talk about marketing to senior level executives, you know, the ones that make the purchasing decisions, the ones that listen to this podcast, well, when you're selling business to business solutions, you want to market to decision makers, it's really hard to find these executives

SPEAKER_57: on social platforms where people are dancing and arguing about politics or movies. I get it, right? Social media is like, it's a whole spectrum of conversation. But I have a solution for you. I think you know what

SPEAKER_60: it is. LinkedIn have 180 million senior level executives, and 10 million C level executives on the

SPEAKER_28: platform. That's out of the 875 million people, right? These are the creme de la creme. That's a ton of purchasing power waiting for you to use LinkedIn ads. And LinkedIn ads is built specifically for B2B marketers, that's you. They know what you're trying to do, trying to get a demo trying to get a meeting trying to get a white paper out there. It's all built around the business to business use case and no other platform in the world can offer these kind of eyeballs. LinkedIn is going to help you reach them in a very respectful environment. Context matters, right? And when people are on LinkedIn,

SPEAKER_60: they're doing business. LinkedIn equals business. Business equals LinkedIn. Let's back that up with

SPEAKER_63: some data. Audiences that are exposed to brand messages on LinkedIn, six times more likely to

SPEAKER_28: convert above average, okay? Make B2B marketing everything it can be and get a $100 credit on your next campaign. Go to LinkedIn.com slash This Week in Startups to claim your credit. That's

SPEAKER_63: LinkedIn.com slash This Week in Startups. Terms and conditions do apply.

Jason Calacanis: When people use Neva and when you interact with your customers, what do they tell you is kind of the number one reason? Is it privacy? Is it customization? Is it just simplicity? I mean, I feel like just the idea of an ad free page that gives you what you want all by itself. If I didn't care about privacy, it might be enough. But I'm curious why people come to Neva.

SPEAKER_09: It's changed over time. A set of people definitely were attracted by the customer focus by how we actively prevent tracking by third parties. People have said things like after a few days of using Neva, the world just feels like a quieter place. It's just there's not, you know, stuff chasing you all across and demanding, um, your attention. Um, Andrew Olofsky, who's a journalist for the Telegraph. Um, he once wrote an article about us. He's like, ah, this feels like breathing clean Alpine air after you have lived in the smog of a city, um, for, you know, four years. Um, but, you know, privacy by itself, um, doesn't in my mind attract enough of a mainstream audience. Privacy is a little bit, um, like being fit or eating clean. All of us want it, but you know, not so many people are actually going to do it. Um, part of what is really exciting about the current moment for us with AI is we are able to harness its power to truly create better experiences. I know we are going to get into things like, um, you know, copyright and fair use, um, but these short answers are super attractive to people, especially in this day of not that much attention. And so these answers are a big step forward in the experience of search. Um, and in a bizarre way, this, we are able to do this quickly, um, much faster than others because we've been at this problem for a long time. And we have a business model, but I don't have to worry about, oh, is this going to lose 5% of revenue because this is all about making the product, uh, product better. Um, so it becomes neither the original part was about ads free and private, but it is very rapidly transitioning into it's just a superior product experience. In addition, it's just efficient. You can also

Jason Calacanis: side note, there's all kinds of things you can search, right? You can attach your email to it and different accounts and actually be able to search things that have otherwise been a little bit hard to, it's almost like organized instead of the world's information, like your life.

SPEAKER_16: That's right. That's right. Um, so we built these, these apps, these connectors to various things. Uh, we have not pushed those, especially in a work context as much

SPEAKER_09: as we should be. Um, we're looking into things like how do we get more companies to adopt these things? As you can imagine, especially with AI thrown in, there's just a lot of time efficiency to be had. Um, and their stock of things like enterprise language models, but part of what we have already figured out is how to pay attention to permissions, how to make sure that like your docs don't get mixed up ever with anyone else's doc. How do we make sure that that stuff is kept strictly segregated? Um, and then this retrieval augmented generation, this search and large language models working in parallel are also very privacy safe because we don't have to ship information from your private documents out over the API or have a language model that that mingles these things. So there's a lot of other exciting stuff that we can

Jason Calacanis: add on top of the base product. Yeah. Once you have like local indexing and you can just sort of index all the 75 sticky notes that I have all over my desktop for various things and I can search those,

SPEAKER_33: please. I'll pay extra for that. Um, let's talk about the business model. And then I do want to move

Jason Calacanis: into some of the kind of more controversial aspects of large learning. Um, how do you make money?

SPEAKER_09: So we are a freemium product. We, you know, the base product is free to use. It's, it's a, it's a pretty generous, uh, you know, basic product that is, uh, that is free. Uh, for much of Neva's existence, we've been focused on user growth. Um, we have a, we have a paid subscription product. It's roughly $50 a year, um, about $6 a month. And, uh, so in addition to essentially getting unlimited Neva, unlimited number of connectors, unlimited searches, um, and so on, we also work with partners to give you additional benefits like a VPN and a password manager, but it's really, it's a freemium model. And the people that end up converting are the ones that, um, use search a lot, um, use things like our personalization features, like setting preferred providers for news, for example. Um, but, uh, yeah, there's, there's a lot of focus on getting people to try the product and we have a pretty decent conversion rate, um, from people that try the product or two people that become paid, uh, paid

Jason Calacanis: subscribers. How does that, I mean, knowing that we live in a world where Google, as you mentioned has nearly unlimited growth potential and revenue potential, um, how do you pitch to VCs that this is a Google S hundred X investment? Well, as you know, uh, you know,

SPEAKER_22: multiples for companies and expectations, uh, have changed dramatically over the last year. And, uh, yeah, it is a very different, you know, environment. And the important thing to remember though,

SPEAKER_09: is that Google is so large that our search is so large because it's everyone on the planet that even a small fraction of people becoming Neva subscribers is enough to make us a self-sustaining company. Um, and so, you know, I've, I've done these estimates, something on the order of five to 10 million subscribers. I mean, now that's, that's a lot of ARR mind you are talking 250 to 500 million dollars a year. Um, by our estimates would be enough for us to run, um, a, a search engine sort of for the whole world. So to say, you know, like being available in all, uh, in, in all countries. Um, and that is enough to get going. So even a few percentage points of market share is still very large simply because, um, you know, the population of the world is very large. Um, in addition, we are also looking into, we have active conversations about taking our technology, um, and licensing it, whether it is the search APIs that sites like Reddit or others can use, um, or being a search provider for large language models, because other people have realized that, uh, doing the kind of constraint generation is actually very useful, um, for generating authentic, uh, output that users will like. Uh, so there's a lot of basically business to business opportunities that have also, um, you know, come about just within the last three months, given how much excitement that there is in the, um, in the area. A direct answer to your question is like, we see scare, um, because the potential market is so gargantuan.

SPEAKER_48: Everyone needs search at the end of the day. Right. And people want better search. Well,

Jason Calacanis: you also, I think it's also a reasonable question about whether the ad supported model, which obviously makes insane amounts of money now, but it's like under attack from a million different directions. Like one assumes that you just cannot continue to make infinite money forever on a model that may be outlawed in entire countries.

SPEAKER_22: Uh, more than that, right. It's, um, you know, if you get into sort of the

SPEAKER_09: excitement of last week with Microsoft and, uh, and, and Google is sufficiently deep pocketed competitive pocketed competitor is basically saying, I am going the state of the art for search is no longer going to be a wall of links. Many of which are ads, the state of the art is going to be this succinct answer that people can consume and they'll be much happier with it than a wall of links. Um, I think that is something that is true. If they can, if somebody can do a good job, um, that's, that's number one. They also have the clout then to be able to take large amounts of, uh, of market share. This is a problem with very successful business models. When there is a fundamental change in the assumptions of the model, uh, you are very, very vulnerable. And that's the place Google is at. Um, Google is amazing as a search engine, as a search company, but if you rapidly move to a world in which you want a paragraph of text, and that is the state of the art, it's not really clear where the wall of ads fit into that. And that, as much as anything else, um, might be the thing that drives change into this whole ecosystem.

SPEAKER_33: Right. And this gets back to your tagline and our kind of accidental tagline. We're moving from a paradigm of search to a paradigm of answers. That's right. That's fundamentally. Yeah. At what

Jason Calacanis: point did you realize that that could be the disruption? Like at what point were you like, nobody wants to search. Everyone wants answers. That's the key.

SPEAKER_16: This is actually, this has been known for some time. Yeah. Google has this feature called they're called featured snippets, right? It's an edgy feature where they would pull out a snippet of

SPEAKER_09: text from a site and show it directly on the search result. Um, and every experiment that's been done to raise coverage of that feature would have wildly user positive. But I mean, this is like users always love it. And it's very, you know, to a certain extent, this is like instinctive, um, which is, would you, um, like, do you want to read three sentences and have your question be answered? Or do you want to tap on a site and go to that site and try and figure out where that answer is? Um, you know, a few years into Google, um, I sort of realized, um, it's dumb, but I was like, yeah, no one wakes up and says they want to click on an ad. Similarly, no one wakes up and says, yeah, I want to click on a link to find out something. If you can just give it to them, they're much happier. Yeah. Um, we've always known this, but the question, the real breakthrough is, oh, wait, we can do this at scale for billions of web pages. Um, and essentially assemble a single, um, answer on the fly. We are like constructing mini Wikipedia pages on the fly for every query. Um, that required a lot of things to come through. Um, and that happened over the course of like Q3 and Q4. We had a lot of stumbles just trying to figure out how to make this thing work at scale or for systems would take eight seconds to, you know, do like the first level of summaries or like, ah, that's never going to work. So a lot of things had to come together, but the basic insight still is answers trump links all the time.

SPEAKER_33: Yeah. And that gets us, I think, directly to this question about publishers, even when

Jason Calacanis: Google introduced those snippets and the summaries publishers, you know, somewhat rightly. I mean, I have worked at these publishers plus freaked out. I mean, it was a massive drop because the link to an answer ecosystem created its own secondary and tertiary ad cottage industry that couldn't have it. I mean, I'm thinking of all of the recipe pages right now that are about to die because it's like, 1500 words, 5000 words of personal story with an ad every paragraph. And then finally, the recipe at the bottom that only exists because of the search paradigm that we live in now. Um, what happens when even when you cite your sources, no one clicks on those links anymore,

SPEAKER_22: they have their answer. It's a, it's, it's, it's a real question. We try to be very thoughtful

SPEAKER_09: with how we do these snippets. We want to keep the length short. Um, we want to convey the answer. And clearly people want to take action. They're going to go, um, you know, click on the link and

SPEAKER_22: actually go to the site. Um, but I do think that we are at a moment, um, where the traditional search engine publisher model, um, is just going to go through a lot of, uh, a lot of stress. Um,

SPEAKER_09: the second order and many ways that you can think of this as like the ads ecosystem sort of coming back full circle. What I mean by that is that the contract between search engines and sites was that they would make their content available to search engines and search engines would deliver traffic. Now for the past 20 years, um, Google slowly, but surely has been lowering the amount of organic traffic that goes out, um, especially for commercial clicks. You show more and more ads, then fewer and fewer people are clicking on those organic links. So this is a, um, this is a slow, but steady, uh, degradation. And similarly, because of the ads model to, and the kind of things that search engines have prioritized, which is like how much time do people spend on site bloggers, for example, rather than just show you the recipe, they have made them long because you want more space to put in more blocks of ads because you know that that drives more click through. Um, I think we are headed to a world in which, uh, a lot of these things are going to be short circuited. Um, and I won't pretend that I know exactly what the consequence is going to be, but I think one important consequence will be that a lot of sites that simply thought of their role as just creating lots of content and traffic would just roll in from search engines and ads would monetize are going to think long and hard about what does it mean to actually attract, acquire, keep customers and have them coming back. So for example, one of our predictions is that the same technology that's used to generate answers on Neva can be used, um, to, you know, as a publisher product, um, so that a publisher like Vox Media can essentially offer a conversational interface to content that's on Vox Media. We have the tech to already do this. So that's a much more, um, engaging experience, right? On the site. But right now, most of these sites don't even bother to put a search box because, uh, no, one's going to search here. They're just going to search on Google and that's how we get people back. So I think this is a little bit of back to basics of every customer that shows up at our site is precious. We have to figure out how to convert that into a meaningful long-term relationship. And what are the tools that we need to have in order to, in order to do that? And I, for one, you know, have to say, like, I, I will not be sad to see the anonymous internet and pages full, um, of ads designed to grab your attention. Go, um, I'm old enough that I remember the time when I used to like subscribe, not just to magazines, but also newsletters that I paid for. Um, and you know, I, I think there'll be quite a bit of back to basics, plus also consolidation. I think that is something else that will definitely happen.

Jason Calacanis: It's, it's, I, it's great to have this honest conversation about it. Cause I, it's sort of, it reminds me of the conversations about something like minimum wage. Like we always say, okay, if, you know, if we want people in America to be able to afford houses, we have to raise a minimum wage to, you know, $15 minimum, maybe $20 and companies will go out of business. And that is just a fact. And it is a hard fact. And we are sorry that that will happen, but it's very interesting because the Google ad supported search model created in some, I mean, the ad supporting publishing model, obviously predated Google, but because you had this huge ecosystem that got even bigger, exponentially bigger under Google, it sounds like what you're saying is that a lot of the ad supported publishing ecosystem, as we know it now only exists because of the Google ad supported search ecosystem. That's right. That's right. And by the way,

SPEAKER_16: the thing that I'll also, I don't think anyone planned this. The thing that I'll also say is that

SPEAKER_09: the ultimately, I don't think that the ad supported ecosystem turned out to be a great deal for publishers, for creators of great content. I think it's the platforms. If you think about it, it's the Facebooks, it's the Googles of the world that are the largest media companies on the planet, because they became aggregators of all of our collective attention. So, you know, again, this is me being a little philosophical, but I think that is something fundamentally unjust, um, about, you know, one company, like spending a billion or so to run a product and making 120 billion dollars of money on it. And essentially advertising became a winner take all sort of game. And there are only three winners. They're called Amazon, Google, and Facebook. Um, and, um, you know, and I don't think, as I said, anyone predicted, uh, predicted this, but definitely in an answer world, I think one of these players is going to be under heavy stress. All right, everybody, I'm here with the seem daughter,

SPEAKER_102: he is the CEO and founder of pilot. You guys know pilot. They help everybody with their accounting,

SPEAKER_28: CFO and tax services. Welcome to the program. Thanks for having me. Let's talk about one or two

SPEAKER_102: of the serious mess ups that founders make when it comes to doing their taxes and their books

SPEAKER_104: properly. Lots of good stories here. The first is simply just not doing your tax return at all. I think a lot of people think if the business didn't make any money, I don't know any tax, which is true. And therefore that I don't need to file a tax return. That part is false. You still have to do the return no matter what. This is a classic place people get bitten. Yeah. And if you don't

SPEAKER_106: do your tax returns, what about R and D credits? Right. You're, you're missing out on a bunch of

SPEAKER_104: good stuff, including the R and D credit. The, the second one that people really get burned on is a big no, no is paying people without using a payroll system. Like you can't just write someone a check.

SPEAKER_108: You got to do payroll withholding. You got to pay payroll tax. You got to set up one of these systems.

SPEAKER_106: All right. Pay your taxes. Even if you lost money this year and make sure you use a payroll system,

SPEAKER_28: critically important. All right, everybody. Here's your call to action. Twist listeners can get 20% off their first six months at pilot.com slash twist. That's pilot.com slash twist for 20%

SPEAKER_64: off your first six months. It's so, it is so interesting to be you. Like how, what has it been

Jason Calacanis: like to sort of undergo this transition? And you must've been realizing this at, at Google for all of those years that there could be this better way that it would be massive. Like it's very brave to want, you know, I, I'm not trying to blow smoke. Like it's scary. It's a scary thing to be the person at the center. And now you've got some cover because other companies are headed in this direction also, but to be saying, this is going to be a huge disruption. This is going to be a massive change, right? Billions of dollars of value will be at best redirected under this new regime. And then even though it will be better,

SPEAKER_66: revolutions are always painful. Solutions are very painful. I think.

SPEAKER_22: Yeah. I mean, leaving, leaving Google itself to start Neva was, was, and is a scary thing. You know, you realize I'm, I'm often reminded by life that a 50% company is nothing. And it's very hard to get

SPEAKER_09: the browsers to work with you. It's very hard to like, you know, stop Chrome, um, from putting up these obnoxious screens to get people that get, try to get people to convert back to Google after they have made Neva their search engine. Um, but, uh, you know, in many ways, I think of like what we have done with Neva AI and answers as the ultimate vindication that like competition matters, um, that a small team can create a product, um, that is state of the art in some way in my mind, um, tells me that the more competition that there is, the better that we'll all be. And I'm not just saying it in a trite fashion. As I said, it's very hard for the organic team at Google to launch a lot of things because they will have a big impact on ads revenue. Also like to remind people that, uh, AT&T invented the answering machine in the 1930s, but basically canned it for a long time because they thought it would reduce the number of phone calls that people made, um, successful people always prevent things that are going to, you know, undermine that model. What is really interesting about this moment, um, is this dam is about to like break wide open. Um, yeah, let's talk a little

Jason Calacanis: bit more about the revolution that we're in because one of the things that Jason has said many times on the show, and I'm sure you've heard is that publishers are lawyering up that the lawsuits are going to come flying, that the copyright questions are intense. We've had a little bit of an argument about how citations can even work in a world of neural networks where in theory, they're learning and ingesting information and repackaging it. Where, where do you sit there? What do you think that the,

SPEAKER_36: you know, next couple of years of this upheaval look like?

SPEAKER_09: So I think we, you know, there's going to be all kinds of lawsuits. Jason's absolutely right.

SPEAKER_22: Yeah. And, uh, I do disagree that, you know, language models, um, cannot cite. If you run them open loop,

SPEAKER_09: yes, they cannot cite, but there are other ways to run them. As I said, we provide context for the generation, um, that also protects us because we are less likely to hallucinate and just make up, uh, you know, make up something. Um, so this is how we are able to provide citations for, um, you know, each, each sentence that, uh, that we write. We went to a lot of trouble to make sure, um, that we were able to do that rather than say, provide like, you know, for four links at the, um, at the bottom. And, um, we also try to keep our summaries, uh, short. Uh, we've been playing around with length. Um, you know, sometimes it's gone to like 300 words, but we're trying to reduce that back to 80 to 80, 200 words. Um, you know, but fair use is a complicated legal concept, not just involving things like the nature of the work or the purpose of the use, um, but also things like the effect of the use on the potential market. Now Neva is so small that I can tell you that we are not going to have an impact on the potential market, but Google doing this at a large scale, um, clearly is not going to be that. Um, so I do think that there is going to be litigation on, you know, on this topic and I don't, I, I, I agree with Jason. I don't think it's okay, um, for a model to just ingest all of somebody else's content, um, and have it be duplicated very easily. So for example, image models, I think are particularly vulnerable if they're going to ingest every painting that someone's made so that you can easily say, ah, you know, draw a sunset in the style of Molly and off it goes and creates a painting. That's just like one that you have created. I would say that looks the weakest. Um, you know, on the other hand, if sort of open loop generation becomes a big business, um, I think that can be, um, that can also be subject to attack. Um, we are trying to, for our own reasons, we are trying to create a space in which a, we are being thoughtful. Citations are mutually beneficial. Um, but there's obviously a lot more to come as the big players get up, um, to change their products.

SPEAKER_128: Are you compensating publishers?

SPEAKER_09: Uh, so we made an early commitment to compensate publishers. That was part of the whole subscription, um, model. So, you know, we have, uh, arrangements with people like Cora and, uh, Medium where we show their content on the search result, uh, search result page. So we committed to sharing 20% of our subscription revenue. Now we are a pretty small company. Um, we are going to, you know, we have millions of users, but in terms of subscription dollars, we're going to hit like a million in air or shortly. So we are pretty small. Um, and, but there are, um, informal arrangements already with publishers where we pay them a fixed token amount of money, um, every month in exchange for the privilege to be able to show, um, you know, essentially these are right hand side pains, um, that show more content than what would be deemed fair use. Um, so yes, we, we've made a commitment. We are paying out small amounts of money. If we grow, this will absolutely become a more

Jason Calacanis: meaningful thing. But will that be sustainable for every publisher or will you have chosen partners? And then will that sort of distort what gets presented? Seems like it could get messy in a

SPEAKER_09: hurry, huh? It 100% can and will get messy in a hurry. Um, again, the rule that, um, we have, we've actually talked about this. There's even an internal doc that we have, um, about this. The nice thing is like, you know, part of the benefit you get from running ads for a decade is you sort of run into every policy problem that there is on, on, on the planet. Um, and, uh, so we have a document internally, for example, that says that, uh, we are not allowed to take into account, um, whether we have a partnership with somebody, um, when it comes to ranking their content, the ranking has to be organic and things like, uh, and answer pain on the right hand side is more, that's the consequence of the partnership, but it's not that we are going to, that we are going to change the ranking.

SPEAKER_132: Um, and, uh, but I agree with you that this is going to become a, uh, this is going to become

Jason Calacanis: a complicated area. And then other hard problems you have decided to tackle because you do not make your own life easy, uh, involve content moderation. Now we're starting to have this question about what should be allowed to show up from these models, how, you know, to what extent content is moderated or filtered or censored, depending on what kind of word you want to use. I, obviously that has been a

SPEAKER_131: problem, uh, certainly a policy to grapple with at Google for a really long time. How are you approaching

Chamath Palihapitiya: that? So roughly the way a search engine thinks about, uh, content is that a search engine's job

SPEAKER_09: is to present you with all the legal content in response to like a query from you. We represent the internet. We don't pretend, um, that, um, you know, we know what is right or wrong or that we know who is, um, you know, like whose position is, is better. So in that sense, um, it is, um, it's, it's a little bit different from say a hosted platform like Twitter and YouTube are often held to a different standard because they also host the videos that express these opinions. So search engine in some sense is, um, is very much off show all the content that's out there. And by the way, European people, um, who think about things like content moderation and free speech differently, um, would be livid with us, um, for showing response. This is Google for showing responses to things like how to make a bomb. They're like, you're helping terrorists on the answer from Google would always be like, Hey, listen, as long as the page that tells you how to make a bomb is legal and can exist, it's our job to help you find it. We are not in the business of saying these queries are wrong. Obviously this does not apply to legal things. Um, but you know, hopefully that gives you an idea of like, how does a search engine think about content moderation? We do not do, um, open loop content generation. So none of the stuff that, for example, you've seen an EYI answer is, is something from a language model that is not constrained to, okay, please focus on the content from these sites. This is what they say. Um, now, now summarize it for, um, the Neva user.

Jason Calacanis: Can you give a, this seems like a good place and probably that good place was 42 minutes ago, but to give us a, a baseline definition of open loop versus closed loop, is that the difference

SPEAKER_09: between generative and not? Um, so generative AI is a broad term. It is used to, um, you know, it's used to refer to models that can like synthesize, generate content that has not existed before. Okay. I think the primary, um, breakthrough in these models, um, is that they understand language really, really well. I think a few years from now, while there is hype, um, about chat GPT and Sydney and other things. I think what we are also going to realize is that these are amazing, like interfaces between us and the world of computers. So traditionally, like things like websites, they're hard to deal with. They want information in a very specific format. God help you. If you like, you know, change the order of months and days when you're entering a birthday on some site, like they're very, I remember the days of Boolean. Yeah. Um, they're very finicky. These models are much more flexible, um, and they are able to understand our input much better. They're also able to generate output that we can consume much better. They can also be used in a way where you just query them. And they will tell you about whatever topic that you ask them. Um, what we do is basically constrained generation. That's what I mean by open loop versus, um, constrained. Open loop is the model has learned once, um, and you're treating it like an Oracle and asking it questions. While in our case, as I said, we do constrained generation because we essentially fix the belief system of the model to content from a small set of pages and say, answer this question within this context. If you can't answer, like, say, so don't try to make up facts.

Jason Calacanis: I think it is a revelation to people that these models can make up facts. That seems like a thing that maybe not quite enough. People realize that open loop means, you know, people are encountering

SPEAKER_36: these hallucinations where they're getting these kinds of rants or machine freak outs.

SPEAKER_09: Well, these models do not understand, as I said, they don't understand provenance. They don't understand, you know, what is right and wrong. They don't understand which site is more trustworthy than others. They don't really understand. Like which author wrote what, um, what book they might, they might understand a little bit off it. So there are language summons. Um, I think it is pretty amusing what they can do if you just ask them questions. Um, but it's like someone gave you a box that like knew all of the books on in the world, but didn't like really understand it at any deep level. You're asking it questions. It's giving you answers. Sometimes it's right. Sometimes it's amusing.

Jason Calacanis: Sometimes it's horrifying. Um, and sometimes it's subjective. It feels like that's sort of the important difference to constrained models are going to give you facts. They cannot be subjective.

SPEAKER_09: That's correct. That's correct. They're subjective. There are also all of these artifacts in how these models are trained that we don't even begin to understand. Now we used to run in my ads team, we run these massive machine learning models, like in 2006, and it would freak out everybody that came into contact with that team. Um, like how little we understood about what the models did. They were right on average, but it is impossible to predict any particular action that's taken. Um, so it won't surprise you, for example, that chat GPT will not write a poem about Donald Trump's accomplishments, but will happily write a poem about Joe Biden's accomplishments. Um, and I don't think like the people that created the model understand why it, it does that. Um, so these models are pretty

SPEAKER_33: mysterious, but so they don't know why I want to, this is a Jason point that I need to clip and take back to them. They don't know why that's happening. That was not like a decision by

SPEAKER_09: some engineer to say, don't, no, no, no. Um, it's, it's like, it's, it's, it could, it could have been what they trained on. It could have been feedback data that they got, like, man, you know, maybe there were poems of Donald, I don't know the poems generated and somebody press like the, um, you know, the thumbs down button lots of times and that made the model realize maybe it should not be generating those. We don't really understand. We don't really understand that. Um, but again, I think our obsession on sort of the, the strange things that these models can create, I mean, it's fine, but there are lots of really good practical uses for these, for these models. Um, is a little bit like if you, you know, if you go back and think about evolution, clearly language was a big breakthrough for, you know, for humans, all of a sudden you can express concepts. Similarly, you know, tool usage was a big breakthrough. Um, I think what you're seeing with Neva, for example, is very much just like, oh, a language model. Oh, a tool. That's a search engine. Let's see what they can do together. But these models are now going to be integrated with browsers with different APIs that can get you facts, um, with structured information. So we are very much at the beginning of what these can accomplish. Um, as I said, it's fun to look at them, um, and see, oh my God, they're saying crazy things. But I think it takes away from all of the practical uses that you can put these models to and, you know, as, as, as a business, that's the thing that we focus most on.

Jason Calacanis: I mean, I wonder, does it start to create a marketing problem for you if everything that you see, you know, an engine that looks a little bit like Neva produced is made up or somehow biased in someone's eyes? Or do you just think this is a long game and we will win it with accuracy in the end?

SPEAKER_22: Yeah, it's, it's a long game and, uh, it, it, it feels very much like, you know, so much can happen in one week. Um, you know, all of this was super quiet and then Microsoft came out with all guns blazing last week and they were like, they could do no wrong. And today Kevin Roos writes an article about how deeply disturbing, um, interacting with Sydney is, um, it's, it's a wild ride. But as I said,

SPEAKER_09: there is so much raw power in these models that like for Vivek and me, um, every week we are like, we can get more done in one week than we could have in like three months of sweating it with the search engine even last year. Um, so there's a lot of benefit that you can get from it. Um, and you know, this, you know, as a startup, you have to roll with the punches. Um, if answers are a problem, then, you know, we're going to rebrand as believable answers, um, and work hard to earn that trust. Um, these are things that you just have to be, you have to be reactive. Um, we stopped, we used to make quarterly like goals early this year. We said like, ah, no more than six weeks. Um, the world is just like got a whole lot more unpredictable. Um, you know, as a technologist, it's exciting. It's exhausting. Um, but as I said, all technologies have positive and negative sides.

SPEAKER_156: We are focused on how do we actually use them to create better products. And you're going to see

SPEAKER_36: a lot of people do that. What a fascinating time. You're right at the eye of the storm. Shradar

SPEAKER_33: Ramaswamy is CEO and co-founder of Neva found at neva.ai. I think you should all check it out. Yeah. Thank you so much, Molly. Thanks for your time. Thank you.