SPEAKER_00: a startup should be looking at like well what's different and it's like well part of what's

SPEAKER_02: differing is we have reasoning as a utility think of every role in the economy as a bundle of tasks and those tasks some of them fall to ai and some of them fall to to humans so i i do think what

Chamath Palihapitiya: startups should be looking at is like how can i change the way like if you look at workflow the way that it's been built over the last 10 years it's been built with humans at the center yeah and that's the actual workflow so if you were to zoom all the way out and say oh okay well i'm going to have a workflow that's going to be shared some of it's going to be through an algorithm and

SPEAKER_01: some of it's going to be through a human what would that allow me to do this week in startups is brought

SPEAKER_04: to you by linkedin ads to redeem a 100 linkedin ad credit and launch your first campaign go to linkedin.com this week in startups zendesk the best customer experiences are built with zendesk qualifying startups can join their startup program and get zendesk products for free for six months visit zendesk.com twist today to get started and beehive power your newsletters with ai tools referral programs and ad network features all in one platform get 30 days free and 20 off your first three months at beehive.com twist hey everybody welcome back to this week in startups this is alex and

SPEAKER_06: i have a special show for you today there's a theme going around the world of startups which is ai hype ai fundraising ai applications ai agents ai chatbots ai job loss ai this ai that what i've always been

SPEAKER_07: very curious about is behind the headlines how is enterprise generative ai spend going and thankfully for me a venture capital firm that i've known for a long time menlo dropped a very long and detailed report digging into precisely that how is generative ai doing in the enterprise so i decided to have a couple of the authors of that report come on the show to answer my questions and then in a stroke of fortuitous luck menlo is perhaps best known in the ai game for being one of the backers of anthropic and this morning the company announced that it raised four billion dollars more from amazon so now we have the vcs from the report trapped under our thumb so we can hit them with all sorts of questions

SPEAKER_08: as we would like but just to welcome them to the show first of all we have joff redfern joff how are you i'm fantastic alex thanks for having me on the show absolutely and what i like the most about you

SPEAKER_10: apart from the fact that you're part of this report is that you were um at yahoo back from 2003 to 2009

SPEAKER_02: right i was i was i was early yahoo prior to uh i've been a product leader my entire career prior to joining benlo about a year ago i was the chief product officer at atlassian yeah i was there for six years and then before that i was an early vice president of product at living din when it was a tiny company pre-ipo and then helped grow that up into uh 10 000 employees so i was there for seven

SPEAKER_12: years yeah i love building things uh started companies sold the company helped take a company public that's a bit about me yeah yeah but the coolest thing is that you worked at yahoo where

SPEAKER_11: i've also worked and it's really yeah it's like of all the things alex that we can point out it's like

SPEAKER_02: i don't know if yahoo was the one but i yeah i know yahoo in the early days and you'll remember this was just super fun um it was like uh it was really the google of the day as the internet was emerging back in uh in the late 90s so it's it's always a tricky one because i i think you know younger folks and uh that are listening won't really know yahoo as the the amazing company that that we were a part of in the early days it'd be uh like if an amazing institute like uh harvard was all of a sudden you know not harvard and it was at a different tier so yeah it's changed quite a bit over the years it's

SPEAKER_16: like if harvard became san jose state was purchased by private equity and i was gonna i was gonna go

SPEAKER_10: with kate cod community college but kate cod community college um yeah that's a deep cut

SPEAKER_06: uh i'm in rhode island which is why all northeast jokes work but we also have derek shaw with us derek hey you my friend i was prepping for the show and the thing about your background that i love the

SPEAKER_07: most is that you were president of the harvard crimson that's right i started in journalism

SPEAKER_19: i know a lot of overlap i think with venture capital but uh trying to yeah it's an interesting start and um a lot of admire a lot of the work that you do and excited to be on i appreciate that

SPEAKER_06: so um also derek on your menlo page you are tagged in on the anthropic side of things now i know it's matt murphy who's the lead partner over at menlo for the anthropic relationship but today the company announced that it raised four billion more from amazon bringing it to a total of eight billion and i i guess the question i want to know is why why is it still so expensive for these ai companies because that's a lot more money not that far afterwards and to me sitting here where i am apart from the numbers and so forth it feels a little terrifying but i was hoping you could assuage my uh my oh my god at that number yeah i mean i think anthropic was uh has a special

SPEAKER_24: relationship with amazon i feel like they have a very close partnership um and this is just a

SPEAKER_26: furtherance of that but the other side of the coin is that anthropic is one of the foundation models right this is when we made our first investment back in 2023 the thesis was that this was going to be one of the you know companies that will matter the most in the ai revolution and we've seen

SPEAKER_27: that play out that's why we have been getting closer and closer to the company and you know the announcement today it's probably just a furtherance of that and deepening of the relationship with

SPEAKER_07: amazon so joff you guys took part in the series c and then if i recall correctly led the series d is

SPEAKER_02: that right yeah we uh through an spv we were lead on 500 million of the billion dollar raise that

SPEAKER_10: happened on the d side and i take it now feeling pretty smart given how ai has gone since that deal

SPEAKER_30: was put together yeah well we're uh we're excited to share with you some of the findings that we've

SPEAKER_02: we've learned on the llm side uh as it pertains to the enterprise right there's obviously two large markets for llms we have the consumer-based market and then we have the enterprise-based market and it it's um you know it's starting to shake out that these are different markets and and how competition is approaching and attacking those are quite different so very very happy to see some of the data uh behind anthropic's progress on the on the uh on the enterprise side i know that i'm slightly harassing

SPEAKER_07: you with the anthropic news when i asked you to come on to talk about the actual enterprise ai report

SPEAKER_06: but i'm just curious one more question about that does having a company like anthropic in the broader you know family portfolio does that really bring in a lot of information to the investment team that you guys can then i don't know learn from and apply directly to making new investment decisions or is that information segregated from the firm and so you guys can't go fishing in that pond to get

SPEAKER_02: learnings for we are we are very um you know church and state when it comes to that we're we're anthropic and in both menlo are um you know we're not in there deep looking at uh people's usage metrics or anything like that but i will say one benefit of the many that comes from it is just being able to see what actually is coming down the the pipeline when it comes to new models we forced a relationship a special relationship with anthropic five months ago we announced our anthology fund that's a hundred billion dollar fund and that's really looking for um who are some of the most pioneering ai founders out there and being part of that uh fund gives gives founders a number of benefit early access to models uh 25 000 and uh anthropic credits access to some of the dev rel teams over there and some of the expertise you'd be part of a network of fellow founders and builders in the ai space and then every so often once or twice a year we we run a builder say our first builder day with anthropic was uh the first of november and that was just that was super fun so it was at the anthropic offices we had a bunch of the experts uh from anthropic coming out meeting with uh companies that were building on the building on the platform and yeah so we were able to have a really successful builder day so i would say you know that part of the relationship is really helpful for both of us so we're we really cherish the uh the team over there and they're they're just amazing i mean the the the team and the the the caliber of the talent that has been brought to anthropic i just i i admire every

SPEAKER_07: day it's it's really quite talented well with four billion dollars more they can certainly keep hiring and i know one of your predictions in the report was a continuance of the ai talent drop so

SPEAKER_40: we'll get to that in a minute let's start with what everyone wants to know which is the the high level numbers so you guys wrote in this 2024 the state of generative ai and the enterprise that ai

SPEAKER_06: spending surged to 13.8 billion this year up 6x from 2023 now before we get into categories derek did that match your expectations is it a faster pace of growth than anticipated to me big number big jump no idea if it was bullish or bearish compared to your projections yeah i mean i think that when we did

SPEAKER_24: this report last year we thought that this would take a little while to uh ramp up and that's what

SPEAKER_26: you saw in previous technological transformations like this right you benchmark to cloud you'd benchmark to mobile and these are all you know trillion dollar kind of uh markets now that took a little while to get started and so i think when one of our predictions from last year was that this would similarly take a little longer to um to ramp up despite all the spending that's going to it

SPEAKER_45: have you built something awesome are you ready to bring it to market well that's great but you're probably asking yourself how do i get to my target customers well here's an easy solution for you it's called linkedin ads and with linkedin you can precisely target the professionals most likely to care about your product you can use linkedin to reach people by job title industry company location and more you know all the stuff that you have on your profile page imagine you're looking to build your b2b audience maybe you want mid-sized companies maybe you want large ones maybe you want emerging small businesses right small and medium-sized businesses smbs well you got to build those key relationships and you want to drive results well you just do that with linkedin ads and you're going to gain access to over a billion members of linkedin of which 13 130 million of them are decision makers that means they can pull the trigger on buying your product or service and 10 million of them one percent of the linkedin audience are those elite c-level executives you know what i'm talking about chief technology officer chief strategy officer chief revenue officer chief executive officer chief financial officer they're all there on linkedin and you're going to get two to five times the return on your ad spend 79 of b2b marketers say linkedin delivers the best paid media results for a reason so here's your call to action start converting your b2b audience into high quality leads today and we're going to give you a hundred dollar credit to start your next campaign linkedin.com this week in startups to claim that credit that's linkedin.com this week in startups

SPEAKER_26: terms and conditions do apply i think what we've seen this year actually is this a surprise in terms of how fast that the the revenue is actually materializing um and so you look at areas like healthcare or legal which are actually traditional laggards in terms of technological adoption and these are actually the verticals that are leading the uh the ai revolution in many ways on the app layer

SPEAKER_27: um there's you know 100 million dollar plus companies now in both of these verticals and um i think that that part was surprising is just like how much that app layer has taken it has uh really taken off the

SPEAKER_07: reason why i'm actually less shocked by that than i thought i would be is that my spouse works in medicine so i'm actually viscerally aware of that world and its inefficiencies but if there are places where

SPEAKER_06: you could apply ai to a voluminous amount of you know written information legal and healthcare are gotta be two of the ripest you know apples on that tree so even if they are historical laggards doesn't that actually mean that they have more accumulated equivalent of like technical debt in their operations and therefore they're the best place to deploy so actually i can kind of see that derek i guess i'm less surprised by that than i thought that i would be yeah absolutely and and the

SPEAKER_26: other way that we look at it too is the comparison between the size of the total market versus the size of technology spent there so if you think about somewhere yeah i mean this this goes towards you know the silicon valley uh kind of phrase now is services software and i think that it rings true it's a cliche because of a reason right because these are massive markets that tech used to not

SPEAKER_59: be able to touch and now uh with ai they can now automate a lot of that you know one thing that does

SPEAKER_00: surprise me and going back to this comment that derek made is just the the steepness of the curve on ai

SPEAKER_02: right having lived through a number of these different waves over the years you know the internet came and then cloud computing came and then mobile came they tend to be much slower you you lose fact that at the enterprise level it was really this is the gen ai movement started two years ago and i would peg that at january 2023 it was january 2023 the chat gpt came out and said hey we got to 100 million mao faster than any other application out there but it was a really telling story i was i was still at atlassian at that point i was the chief product officer and in january like that first uh earnings call there was like one mention of ai but if you looked at all of 2022 there wasn't a single branch of generative ai so it went from the first earnings call by the second earning call there were 18 mentions in the transcript so like a fun activity is go grab any company right and pull all of the transcripts from 2022 to to now and load them up into your favorite llm and just say build me a chart of the number of mentions on ai or artificial intelligence and what you see is it's like it goes like crickets to hey something's happening here to all of a sudden that is the conversation yes um so i would say that is is definitely one of the things that played out and that's one of the findings that we have in the report if you look at 2023 that would be the year really of the pilot so the the ceo gets off the tr uh the earnings call with the cfo goes over to the r d group and says to the cto and the cpo like hey what are we doing with generative ai and very quickly what happens is teams get pulled together it's like how are we going to use this new technology and that's really one of the findings from the survey is that last year was a lot about experimentation this year it's really about moving stuff into production and that's where we get the the the 6x increase in the overall uh spend at the enterprise level coming up to close to 14 billion

SPEAKER_07: uh by by our marks i want to talk about uh basically data sourcing for the stuff because i love this particular chart but to me when i see this i go how confident are you guys in these numbers because

SPEAKER_06: you could easily categorize things in different pockets i presume there's some bleed between them and also it's a growing industry so can you just talk me through the how this chart was put together and if you're listening to the audio this shows um year-over-year changes in generative ai spend for foundation models training employment data vertical departmental and horizontal let me start with a

SPEAKER_02: high level and then i'm going to let derek talk through the how we've actually calculated numbers in this chart really there's two big buckets that you should be focusing on one is around the llm and the infrastructure needed to bring ai into the organization and then the second bucket these four are actually three the three bars on the right are really talking about the application layer so what we can see is that two-thirds of the spend that goes on at the enterprise 9.2 ish billion is sitting in the llms in the infrastructure bucket and for those that can't see by far

SPEAKER_30: the largest spend inside of the enterprise is against the foundation models themselves 6.5 billion of

SPEAKER_02: that is spent there so two-thirds is happening at the infrastructure layer and then we have another uh third which is being spent at the application layer the main thing about the application layer the big story is there is that that's an 8x increase from from the prior year so that's sitting at 4.6 billion then what alex is is asking one of the questions he's asking is like you can categorize your application layers a lot of different ways in this case we've broken it down into vertical departmental and horizontal ai and and certainly we can squabble about like what belongs into what bucket but i would say the higher order story is that applications are coming they're alive and well and then for a

SPEAKER_26: variety of reasons we've chosen to break it out this way in terms of data sources we went out and asked 600 it decision makers um so these who are you know budget owners that have purview over their organizations generative ai spend you know this very lengthy survey but basically how much are you spending on generative ai what is that relative to your overall spend um and then very specifically what tools are you spending it on um i think that one of the motivations behind the survey was that there's just a lack of good data out there of like what are actually it decision makers really kind of looking at what are the use cases that they have um and where is their money going towards and so this is um you know we spoke to 600 we didn't go get to all you know the fortune 2000 um and so there's obviously you know this is directionally our answer um but i think that it's

SPEAKER_27: pretty informative to see some of the insights of like what folks are actually you know not just playing around with anymore but actually adopting so just to summarize that um the 4.6 billion that

SPEAKER_06: we see for the different types of ai applications you guys surveyed on i should probably pay a little bit more attention to the year of your growth in that number than to sit here and go why is it 4.6 and

SPEAKER_02: not 4.7 billion yeah exactly and then the other thing you know the next question you're gonna ask i was like okay great what are they doing at that application layer or you know and the thing that i would say is fascinating about that is that there's a real broad uh set of geni use cases in the enterprise so sometimes you see things get very concentrated around one department or one use case but i think it speaks to the the usefulness of the technology so when we break down if on average an enterprise has 10 geni uh use cases amongst it i'd say which ones are most popular yeah and that would be a that'd be an obvious question so when we look at it code generation was number one 51 of the folks on the survey uh said that they're using ai for code gen they're building software for it joff just to be

SPEAKER_07: clear not 51 of the 600 who are paying for generative ai services but of all the 600 51 companies are paying for um code completion tools yes okay just want to make sure so this is as big of a number of the

SPEAKER_60: 600 as we could imagine okay i appreciate that keep going that's not surprising we've seen um github

SPEAKER_02: uh copilot is one of the you know fastest growing revenue products out there and in the application layer of ai they're north of 300 million and uh in annual revenue we've seen players like cognition codium all hands come about um after code gen is support chat bots probably not surprising there 31 are saying they're deploying chat bots we've got products like sierra decagon uh acera which is uh from an itsm it service management use case is a is a big one there hey startups when you're a business

SPEAKER_45: you gotta treat your customers right unreasonable hospitality is the standard today but you're going to need tools you're going to need a platform to help you do this and that platform is the zendesk suite the zendesk suite is going to give your startup all the tools you need to deliver exceptional customer experiences so you can build stronger relationships without growing your headcount that's key right every dollar matters you gotta control headcount you gotta control spend so use the tool that shopify squarespace uber and instacart all rely on it's called zendesk let's take a look at another customer unity very famous company they saved 1.3 million dollars with zendesk automations and self-service and they saw an 83 percent increase in their first response time these companies love zendesk because it's so easy to set up and it scales with you as you grow they'll also give you all the metrics to make your reporting easy keeping you and your business agile and investor ready and that's because you're their esteemed customer and they've created the zendesk for startups program just for you where you get unlimited access to all the zendesk products expert insights all the best practices and entry into their amazing community of founders all at no cost for the first six months that's right they want to support you zendesk.com twist get ready to

SPEAKER_44: scale with the best in customer support with six months free nothing to lose then you move to

SPEAKER_00: enterprise search and retrieval 28 percent um you know these products like a product like glean has

SPEAKER_02: been around for a while but with the emergence of genii what they've been able to actually accomplish on the search front has been dramatically better than maybe what they were able to do in their early years another product and there would be something like sauna yeah then we move to um you know things like meaning summarization uh you know so many of us are remote workers and we're so used to seeing that ai agent has been added to either pull a transcript or summarization of our conversation yeah um things like firefy and and otter and then the the last one i would mention that kind of round out the top five would be um would be copywriting and this goes back to the comment that you were making out it's like i you know llms are like calculators for words they're like really good at authoring content so i you know you could certainly see a world in which everyone um would be using uh or having a copyright or an editor you know grammar checker just part of whatever they're communicating to help them be more precise and concise products like writer typeface where we have an investment uh copy ai they tend to be you know some of the leaders in that space but that gives you a feel for like i could get what is going on at the application layer of the enterprise

SPEAKER_06: so a lot of words is my read of that i think the calculator for words is a great way to think about this code generation is creating characters on a screen support chatbots create characters on a screen you know meeting summarization characters on the screen copywriting characters on a screen the thing that surprised me is that workflow automation was so low in this chart to date because if you go back five years we're thinking about ui path and you know rpa robotic process automation and there was a lot of enthusiasm pre-generative ai pre-llms that rpa was going to remove a lot of the human drudgery from digital work then we got much better tooling but now that we're you know looking back at 2024 i'm not seeing as much of that show up as i kind of expected and this is a long way of saying how much progress have we actually made on agents i suppose that can take some of the work from us and do it versus being more assistive in our day-to-day work

SPEAKER_24: and derek i see you blinking at me so go ahead yeah i i think that we think about it in terms of waves and so the first waves of gen ai apps were what we called rag apps or retrieval augmented

SPEAKER_26: generation based um and so usually you know you you have an external knowledge store and you uh you use it for things like synthesis so eve is one of our portfolio companies uh it's a legal copilot and what it does is it takes a lot of long dense legal texts and makes it generates reports it generates legal briefings things like that so that the lawyer on the other end doesn't have to go through the drudgery of all that work and so that's kind of the first generation and then as you get more advanced we move into agentic architectures which you know today our survey found that it is a minority but if you ask a year ago if you look at our previous report it didn't exist a year ago and this idea of you had like baby agi and autogen and some of these like you know open source projects back then but it didn't exist as an enterprise idea of something that you know when i think about traditional rpa like ui path um the idea of applying it to the enterprise hadn't really existed and now it does

SPEAKER_06: so so derek it sounds like what i was doing was just being impatient and now it is showing up and so what i expected to happen is i just had my timelines off mentally compared to the market i think the

SPEAKER_26: technology is now there and if you look at things that we're really excited for in 2025 this is one of the things that we think will explode is moving from retrieval based architectures to more agents and things that can work automate workflows across horizontal areas so like you think ui path but also verticals you know healthcare there's solutions like tenor that are doing uh kind of ingest automation and um you know a lot of different uh for domain specific applications i think you'll you'll also see

SPEAKER_36: and i'm literally right now pulling up a um the open ai 01 preview blog post because i forgot the term

SPEAKER_07: that i need here it's like time series thinking when uh models take a little more time before they make a decision or return a prompt is that underpinning the improvement in the technology that is making the

SPEAKER_26: agentic approach more feasible today derek yeah um test time inference and so i think that i i think that there's a couple layers to this right um so you can have stuff like o1 which is uh kind of formalizing a designed um uh kind of pattern at the model layer and so as the model gets smarter it makes fewer errors i think the problem with agents that you had traditionally is that it's kind of run in a recursive loop so if you think about it you know the agent will the agent which is the llm will think okay in order to accomplish this like task that requires 10 steps what are those 10 steps and then i'll go out and do it and then i'll think i'm at step one okay check what is step two and then if you think about error rates there right llms hallucinate um that that's now well established if you have a 99 error rate or a 99 accuracy rate in step one if you haven't that for all 10 steps by the time you get to step 10 you know the error rate is something that is unacceptable for enterprises and so that's traditionally been the problem um and so llm's getting smarter is part of the answer but also when you apply it to specific domains you have data scaffolding around it we we like to use the term agent on rails which is basically you need to hard code it or harness kind of the domain of like all the actions that the agent can take you need to kind of set guard rails on it with hard with code in order to point it and get you know higher levels of accuracy so that that recursive error rate

SPEAKER_06: doesn't compound too much do those guard rails have to be programmed on a per use case per industry or per company basis i'm trying to figure out how hard it is to hard code those because to me that could be

SPEAKER_07: incredibly complicated or relatively easy i just don't know where it lands yeah i think it um people

SPEAKER_26: are trying to figure that out right um and i think that obviously the more specific to a particular use case the higher accuracy and more robust it is and but the thing you're trading off really is degrees of freedom which like you know all the way at this very end is agi you know no guard rails just the model just like put it in a for loop and it runs and then on the other end is um what we have today which is computers that are 100 hard-coded application logic is determined by uh the computer or by you know first of all some programmer who sat down and was like okay i'm thinking in the shoes of the user what do i need and so i think the answer will probably be somewhere in between we're trying to move towards agi eventually but yeah i think right today we need uh stricter guardrails i think you know there's um

SPEAKER_02: as i'm hearing you talk to eric there's a there's a good example in the software agentic space right okay so let's take code generation and we can say well how much progress are we making in code generation one way we can look at progress is we can look at the score called the sweet bench right and we can look at that over the last what's happened there in the last 11 months so sweet bench for those that don't know it's um it's testing real world tasks faced by software developers and the benchmark was based on things like pull requests and issues from open source github repositories and i think there's something like 2200 tests in there so if we go back to january this year about four percent of those tests were completed by the best software agentic system out there in march there was a company called cognition which got a lot of a lot of traction you might remember that by march it was 14 now if you look at it the number one score on sweet bench belongs to one of our portfolio companies it's an open source software company called all hands and they can

SPEAKER_45: solve 53 of those cases hey founders let's talk about a super weapon that your startup might be missing killer newsletters newsletters aren't just a great way for you to stay relevant they're also an incredible growth engine for your startup social platforms and search traffic those things are losing their edge but your email list that remains your direct line of communication to your customers you don't have to pay a middleman you don't have to get caught up and try to trick the algorithm into getting your message through but in order to have a successful newsletter you need a world-class platform and that's why we moved the this week in startups newsletter to beehive beehive is a newsletter platform that's built to help you grow your community and turn your newsletter into an absolute powerhouse it's packed with amazing features like an ai driven post builder it's also got this really clever built in referral program and a top-notch ad network if you want to turn on monetization plus they have really easy to use tools if you want to charge access fees like subscriptions so here's your call to action beehive plans start at just 39 a month super affordable but we've got an exclusive deal for you here at this week in startups a 30-day free trial followed by 20 off your first three months that's right go to beehive.com twist to claim that 20 off beehive.com twist to start growing your newsletter

SPEAKER_02: today so if you kind of look over the 11 months right it's gone from four percent to 53 so that's that's like uh you know 13 improvement from the beginning of of the year it's dramatic and i guess

SPEAKER_07: you know joff how confident are you personally not speaking for any other company that that rate of progress can be kept up for the next 12 or 18 months because if you run the numbers out long enough

Chamath Palihapitiya: eventually we get damn close to a hundred percent yeah look i think it's going to be uh invariant it

SPEAKER_00: will it will depend on what department and what use case that you're moving against will it move from

SPEAKER_02: 53 percent to a hundred percent next year on the software agentic yeah you know i do think it attenuates out you know the stuff is fairly unoptimized right now so i think even with the models that we have let alone new advancements in the and the underlying foundation model i think there's tons that can still be done from an accuracy perspective okay so thinking about the

SPEAKER_06: categories of enterprise gen ai spend that we started with discussions about improvements to agentic ai and the approach thereof and also improvements to existing leading categories this all sounds very bullish on the technology side my question is how much of the revenue that we're talking about is spread out amongst the startups because you know joff you said that you know six and a half billion other 13.8 is foundation model spend open ai anthropic a couple of names

SPEAKER_07: when we get to the 4.6 billion from the app side the number of companies that are nibbling at that spend is enormous so is is there enough enterprise generative ai spend for apps today to support the

SPEAKER_06: number of startups that are going after those dollars or are we like is there half as much spend as we need i i just don't know if this is too much brad not enough butter i guess you know

SPEAKER_02: it's a merchant if we look at the the software budgets at the enterprise for ai in 2023 they were going towards something called an innovation budget which a lot of times that's like hey we just need to be investing in here we don't have a permanent budget to pull from um so i i do what i do see happen is you're going to pull from the permanent budgets in time as the roi uh improves the you know another point that i would make is that historically a lot of the budgets that were being spent by both the business unit and i.t were around tooling and back to this comment that derek had made earlier it's um you know now we have services as software so it's not just the tooling budget but it's the human capital budget that we start to move into as well so once you can i can explain that to people because i think

SPEAKER_06: it's a good point but i want to double click on it so essentially if you can reduce head count and replace that with software spend you can often get quite a lot of bang for your buck and so budgets for software for ai that might replace some human activity can be relatively uh rich because humans

SPEAKER_113: are costly and in health insurance and travel stipends and office space and yeah and i know i don't

Chamath Palihapitiya: think it's just about human reduction right like i do believe that there's their companies are also

SPEAKER_02: faced with like a lack of having enough people like i don't have enough software engineers so it allows me to you know expand and continue to grow that department but i can do that in a different way so it's not just about reducing it's also about expansion but but that is sure is part of the story so i i think there's plenty of wood to to chop in there when in the report itself you'll see the ai spend by department um and you know you'll see departments like the legal department is historically uh like not spending money on technology they're they're much more of a late adopter and here they are being um more of an early adopter so i think the budgets are very promising and um you know i think the they'll continue to grow as the roi continues to prove itself out so from an roi perspective last year it was less clear what the roi was this year as we move from our pilots into production there there's greater clarity um but still some question marks right like it's not all figured out the number of folks that were in the survey i want to say it was something like uh you know a third of the survey respondents are saying well we're still figuring out our exact implementation on ai strategy but derek picking up on

SPEAKER_06: that point when we think about ai generative ai spend in the enterprise we're essentially asking what what is the tam for that today and i was just talking about uh on the show the other day about how uber when it was very young people were accompanying it against the taxi market turned out that was bs because the market got much larger it it seems that if we can unlock spend from departments that didn't have technology budgets to begin with like legal for example the tam for ai generative ai and the enterprise is huge but also it makes the overall tan for software itself

SPEAKER_07: larger and i don't think i actually thought that was going to be the case but it does seem very bullish if i'm understanding this correctly yeah i think that generative ai as far as adoption

SPEAKER_26: today is more expansive than replacement and so you have like innovation budgets last year but it's actually really interesting we asked where are you pulling uh uh budget from and of the permanent budget types um a lot of it or over half of the permanent budget for generative ai is coming from new budget so it is not you know i am either you know replacing spend for my system of record or some other software it is i'm creating a new line item for this and i think that you're seeing that across departments which is it is both expansive as well as across a lot of different places that used to not spend a lot on technology and we're seeing you know from the startup side which is where joff and i spend a lot of our time you know we're seeing companies pop up all over the map whether it's verticals uh we mentioned healthcare legal financial services or departments sales and marketing uh data science uh human resources like it's it's really all over the map which is why we're quite

SPEAKER_07: excited about you know generative ai as a category okay while i would love to just vamp about ai for another like three hours i want to narrow this down to some startup focused stuff so you guys had a great chart that's called selection criteria for generative ai tools and to my surprise uh the highest line item wasn't cost instead it was roi and so it seems that when people are approaching ai software what they want is not to spend as little money as possible but to have the biggest bang as possible and so i'm curious what should startups take away from this particular chart joff as they approach the market so that way they can see the most success you know things that are incredibly useful

SPEAKER_20: no no one's ever said that before there's no accelerator that says build things people want

Chamath Palihapitiya: yeah you know you will it's interesting and just sticking with like what would be good advice on the

SPEAKER_02: startup front i do find that um you know you want to look at what can you do now that you couldn't do yesterday and that's really a thing to focus on i i remember when i was at linkedin one of the things that i did was i started the mobile group there a lot of my peers were saying hey look we've got

Chamath Palihapitiya: 2 000 pages on the website let's like jam it into this mobile device and my point was like no that's exactly what we shouldn't do like the mobile device has something that is unique and it's special and

SPEAKER_02: therefore what we should be doing is focusing on highlighting those use cases so that it you know in time i believe that linkedin would become a mobile company and indeed it it did wind up coming to become a mobile company and the product that we built was quite different there was only 17 screens compared to the the 2000 that existed on on the website so i i think in this moment you know you have to look at

Chamath Palihapitiya: a startup should be looking at like well what's different and it's like well part of what's differing is we have reasoning as a utility so you know if i look at um there's a there's a guy

SPEAKER_02: daniel rock who's a professor at upenny um he i've had a number of conversations with him and he was telling me he's like look think of every role in the economy as a bundle of tasks and those tasks some of them fall to ai and some of them fall to to humans so i i do think what startups should be

Chamath Palihapitiya: looking at is like how can i change the way like if you look at workflow the way that it's been built over the last 10 years it's been built with humans at the center and that's the actual workflow so if you were to zoom all the way out and say oh okay well i'm going to have a workflow that's going to be shared some of it's going to be through an algorithm and some of it's going to be through a human what would that allow me to do because a lot of these workflows that we see at the enterprise level are very complicated and very tool rich you know if you look at a an average

SPEAKER_02: sales team they have like 12 different pools hanging off of salesforce or hubspot if you look at software development there's over 12 different tools that are hanging off of the software development flow so you know you have an opportunity to reimagine what that workflow should be so you're you're either going to enter in and say hey i'm just going to try to aiify the existing workflow or i'm going to think about it at a more first principles level and think about what could be possible given what we know now and and that would probably be my number one uh my number one suggestion is really get

SPEAKER_07: back to the first principles of it okay so i want to pick up on that because you guys wrote uh last year incumbents dominated the enterprise market with bolt-on strategies that layered gen ai capabilities onto

SPEAKER_06: existing products and joff it sounds much more like you're saying look don't do that rip the page out start blank and build from the ground up so derek i think we call this an ai first approach just to building software um what fraction of software that exists today needs to be ripped out and started over again because people joke like i don't know what salesforce does we all kind of do but not really and you know not to make fun of uh atlassian but jira is hell so um i'm curious you know how much

SPEAKER_20: i i will not sir i have filed tickets and i you owe me lunch for the pain you put me through don't

SPEAKER_06: worry people the people who make concur owe me a house so it's fine i'm just curious derek like uh how much of software needs to be rethought in this way yeah i don't know that i have a very clean

SPEAKER_26: answer for you know x percent of software needs to be completely rethought but it's very across the board i think that one of the things that we realized this year versus last year is that it's actually not that easy to build ai that works you know a lot of people thought last year like oh if i'm a sales force if i'm an adobe i can just tack on something ai on top of my existing system of

SPEAKER_36: record and that will be my ai solution data already because if you are a system of record you have the bucket of data to make your own tuned models with the data the distribution the trust with customers

SPEAKER_26: all of that like and that is why a lot of people thought like is ai really a net new category or is it just a feature on top of existing software and i think what we've realized is actually it is its own independent category and why it's really exciting for us is because that gives the advantage to startups rather than you know salesforce coming out with agent force and everybody just being like this is the greatest thing on earth and so everybody will adopt it there is opportunity for startups because people enterprises we talk to try out features like agent force and realize wait this is not what was promised and so there's opportunities through domain specific workflows you know an ai native approach to actually make that promise work but if we think about the companies that exist today

SPEAKER_06: that might struggle to move from you know their kind of legacy software offerings that make them all their money to an ai future that means that there is trillions of dollars in market cap out there for startups that are being born now with the blank sheet of paper and the modern tools we're discussing to go after so in a sense i kind of like think we should short the nasdaq and like double our investment in dc and that would be the best way to get kind of both sides of the back you know it's

SPEAKER_24: different um domain by domain right like it's hard to predict and that's what makes investing fun like adobe for example they started as an on-prem company and then when the cloud revolution came

SPEAKER_26: they actually took a tremendous bet it's actually really interesting if you look at their quarterly revenue when they decided to go from on-prem to subscription they were able to convert that and you know you could adobe is one of the companies that i personally admire a lot will i bet for them or against them in this gen ai revolution i i personally think adobe is is a really interesting company firefly is a great product but not all companies will do that right and so when you look at the uh startups but the startups tackling it and what do they offer why are they different from the incumbent as well as who is the incumbent in that solution are they well positioned to move with the uh currents versus against them um you really have to take it domain by domain and it is like it's

SPEAKER_30: a very visceral thing when you see a company that gets caught up and left behind in that right you can look at a company like check 85 percent of their market cap is disappeared with ai or stack overflow a

SPEAKER_02: site that i used to use quite a bit like their traffic is halved as people go directly into the

SPEAKER_06: llm to get their their coding guidance yeah i wonder how many of those we're going to have by this time next year because chegg made money off helping people cheat on homework and they're going to get mad at me about that so prg please don't email me again that's what people are using check for we all know it now people use open ai to cheat it's great i wish i had when i was in middle school what a crushed chemistry i did not because i didn't but i wonder how many other companies are kind of on that list and if that will be a good barometer for how fast ai first companies can kind of uproot

SPEAKER_07: legacy companies that are i guess cloud and sas are now legacy and kind of they almost feel outmoded

SPEAKER_06: like do you guys remember when salesforce invented sass and you're like oh my god this is the future it's weird now to be seen here going oh my god is that the past we you know it's it's interesting

SPEAKER_00: because i i feel like these are all um things that were building blocks that were needed to get us

SPEAKER_02: to where we are today we actually needed the internet to get the world's information digitized we needed to have cloud computing in order to unleash vast vast amounts of of gpus and cpus to do training and inference and things like that so it it feels like it was more of a lineage building up

SPEAKER_137: to the moment that we are today yeah but how much credit do we give yahoo for inventing the internet

SPEAKER_141: portal none today we don't we don't even think about it yeah well what's that saying or xerox in

SPEAKER_02: the gui right i mean it's like you're you know you're uh my company was built on the shoulders of

SPEAKER_30: giants or ai is built on the shoulders of giants i mean there there's a lot of things that that needed to happen to get us to the state that we're in today yeah i think there's building blocks and there's

SPEAKER_26: value capture and i think that a lot of the new value to be captured will be by startups and new companies um but obviously building on the the shoulder of the giants so glad we ended up here

SPEAKER_06: because i have a question about this i know we're talking about how startups can disrupt incumbents but there are some incumbent ai companies and i'm thinking about very clearly open ai and anthropic

SPEAKER_07: being two of the largest players in this kind of neo space and open ai recently put out a search product which is frankly pretty good i use it on a regular basis as a testing tool and i also think that perplexity will see some of its momentum cut because there's now a competing product from a larger better finance company and so when you guys are talking to startup founders how do you help them navigate building something that won't get stomped on by a foundation model company releasing

SPEAKER_24: something that can be considered to be competing i think one of the things to think about is what is on

SPEAKER_26: their what is on their near-term roadmap right and for a company like open ai and anthropic their ultimate goal is to achieve agi for anthropic to achieve agi safely um and what are the things that are going to be you know the things that they'll tackle next and so they've anthropic has already mentioned you know we won't pursue you know side quests such as image generation or things like that because it is not necessarily the thing that will bring them to agi better reasoning on the other hand or using tools like web browsers and computer use on computer use those are right um and so a lot of the you know when you are an ai app today there are two things that you really need to get right one is can i make the base technology work and a lot of the things that actually have happened to date are just how do i make the llm reliable how do i connect it into enterprise systems how do i give it a tool such as a web browser and how do i you know when i was talking about the agents before how do i make sure that the reasoning is reliable but you can think about these as data scaffolding things that um you know actually are needed today to help uh act as crutches for the llm to actually work in an enterprise application those i think will one by one fall away over time as anthropic makes advancements in the intelligence of the base llm but what won't change and what anthropic won't get to because quite frankly it's not important for agi is how do i apply this to my you know health care domain specific workflow if i need to you know uh do like clinical documentation integrity if i need to do rcm on the back end revenue cycle management anthropic's not really

SPEAKER_06: interested in that and so that's an opportunity for outplay or startups okay but here's my question because all that tracks with me derek i i agree with you wholeheartedly but as we get closer to agi and as we actually maybe reach it in the near-ish future doesn't that obviate a lot of the work that's being done to make ai apply to specific categories or market verticals because as the ai

SPEAKER_07: brain gets smarter it probably needs less help to do more and so i wonder if we'll see a dilution of the power of going vertical in ai as we get closer or reach agi i think the the thing that maybe that

SPEAKER_24: discounts a little bit or takes for granted a little bit is the training needed to actually make

SPEAKER_26: the solution work you know if you let's say that you have a super intelligent phd level human right if you have to if you take that person um and apply them to let's say the health care example of like how do you do rcm and work with payers to make sure that your claims get paid you still have to teach them how to do it you still have to you know there's a learning curve as they get used to the workflows there and you know obviously if you take that person versus somebody um you know who may be less intelligent there's a shorter learning curve there but there's still a learning curve there and how you actually get that intelligence and apply it to the specific workflow on the other end that is

SPEAKER_06: the area where application layer companies can add value okay but you're making a a a a a jump there that i i wouldn't make which is you're saying that if you took a phd level agi my hope is that by the time we reach agi we're no longer using postgraduate benchmarks to determine intelligence or expertise we're so far above that that those those analogies don't hold and then we won't have to have so much verticalization guardrails built and so forth because in theory this should be able to be a bit

SPEAKER_02: like a magic box i'm hoping there's a question too if there's like well one algo to rule them all or if that algo is a collection of millions and millions of algos right you know think about my phone like the power of my phone is actually sitting in the ecosystem it's that there's an app for that so there's millions and millions of apps that make that product more productive so you know one version of the world is like i got an algo and it does everything another version might be that i have an orchestration algo that knows when to ask other algos like for their expertise in a given area or science so part of that is still unknown um as we move into the future yeah i think we joke about

SPEAKER_07: how everything is bundling and unbundling but i kind of wonder if when we get to agi if we're going to have agis orchestrating agis in a bundle or kind of a singular brain mother of all bundles right yeah kind of like if you got you know netflix and espn the same thing can you imagine that would be

SPEAKER_161: crazy that would be nuts alex i know i know so to summarize though just kind of taking all this in

SPEAKER_06: one in one bucket enterprise generative ai spend grew very quickly and perhaps even faster than expected we are seeing increased spend on foundation models but also on the application side and you guys in the menlo perspective is that the upstarts are going to knock off more incumbents and probably we're going to see more agentic ai progress next year that's going to be very exciting is there anything else at the the highest level from this report that i have not brought up because i want to make sure we get all the all the key bits of meat out if that makes sense joff yeah i think you know there's one

SPEAKER_02: point that i found really fascinating that we didn't punch into too much is that we went into the enterprise and we said well what is and this is at the infrastructure layer we said well what is the llm that you're using uh what they came back and said is that they're leveraging multi-models so i i thought that they would try to like get behind a single model um whether it's for hedging or for performance or cost reasons they're they're typically on average using three different models which i thought was really fascinating and then the other thing is um you know just the shift in the enterprise around what is who is using what from a foundation model perspective open ai clearly had the first mover advantage in the enterprise but what we saw on the survey is that they moved from 50 down to 34 so that lost 16 yeah 16 points here every year you can see that anthropic doubling coming up closed source models are clearly you know the majority of what's being used um meadow was flat mistral

SPEAKER_06: uh you know off a little bit um but you know poor europe look at that look at their one ai champion like fourth in line we gotta maybe everyone should spend 10 of their ai spend on mistral just to help france because you know new administration ukraine okay just me all right

SPEAKER_166: keep going job man we're investors over here i love how uncomfortable he got right there it's like

SPEAKER_160: exactly no don't touch exactly it's like stand clear stand down you're this is where like that pr training comes in play it's like wait is alex baiting us is this gonna be the one quote that comes out after all this like wonderful great conversation you know alex and team are gonna pick on that one time

SPEAKER_08: that we blinked joff redfern just endorsed all trump administration policies i'm kidding i'm kidding

SPEAKER_06: i'm kidding exactly but i i am i think just very net optimistic and that's a good place to end a year i think a year of so much change it does all feel to me very exciting like we are really making progress and you know the computer use stuff from anthropic not to give your portfolio company extra ups but like when that dropped i was as excited about that as i was about chat gpt the first time i used it you know and that that's a great place to be going into a new year of a lot of investment in progress so it's all very encouraging now just before i let you guys go i do have one lightning round ish question so sorry for this but i i can't help myself so derek we'll start with you

SPEAKER_07: um what percentage of your net worth is in crypto or bitcoin i am actually outside of venture a very

SPEAKER_176: unsophisticated investor so uh next to none fortunately uh so a little bit via etfs sounds like all index funds uh you know bet on the us economy bro you should see my family's portfolio exactly the

SPEAKER_07: same and now 0.1 bitcoin via fidelity etfs joff over to you same queue curious how well crypto exposed you are going into 25 zero zero zero yeah this was supposed to be the freebie not the not the alex

SPEAKER_40: does pick out the quote at the very end and make that into the headline i mean jesus yeah no i um

SPEAKER_02: you know i i focus on things where i think i have like better understanding or a competitive advantage so similar to derek it's like passive on the economy and then i am way over indexed on uh on on venture investing through a variety of different funds and angel investing so it's like yeah if you and if you look at that it's like why is he doing that it's because i feel like i have a like that's all i've done i built software my entire life like startups that's like that's the thing i know super well it has no and it's like not a reflection at all on crypto or my beliefs there it's mostly just about what i i mean like i have so much interest in in startup land i i appreciate the

SPEAKER_08: honesty there because i feel like there's always a pressure whenever crypto does crypto things to

SPEAKER_07: appear sophisticated about it for 18 months until it goes away for two years again yeah but it has been quite loud and the uh the crypto fans have been making noise but uh anyways

SPEAKER_06: guys thank you so much and we'll come back and do this again next year for the 2025 generative ai and the enterprise reports i'm sure by then we'll have even bigger numbers to uh dig through but before

SPEAKER_30: you go drop your twitter handles and then we'll say goodbye yeah i uh might use my linkedin handle go to go to linkedin man that's where i'm posting that's my activity that's the thing i built so i stick with

SPEAKER_40: it fair enough derek uh what is your preferred social media handle i apologize i am on i am on

SPEAKER_188: both um but on twitter i am at derek gsha all right thank you guys very both very much and uh twist

SPEAKER_07: is back next week for a couple of new shows and then it's thanksgiving time y'all we'll see you then bye thanks tal it's super fun