SPEAKER_00: You know, what engineers do is troubleshoot, fix problems, they create problems, and then they fix the problems they created. There's a lot of interest now in how to use AI, especially Gen AI, and how that can ease the burden of IT operations.

SPEAKER_01: The moment you get on a phone and you're looking at a TikTok video, I mean, the number of layers that are involved to deliver that, notwithstanding the physical infrastructure to deliver the audio and video content, but the infrastructure software and physical infrastructure that it takes to store the media, it's a very complex environment, and people want more of this faster.

SPEAKER_02: Just for fun, can you walk me through some of the software that goes into making a TikTok appear on my phone, for example?

SPEAKER_01: The videos have to be stored somewhere, so there's some quality control. So there's software associated with that, and then when the video gets edited and uploaded and stored, you want to make sure that there's no data loss, no integrity compromise on the video, and then the distribution aspect of it. Then there's edge caching. You don't want somebody looking at the video to hit the backend server each time, and so

SPEAKER_05: there's what's called content distribution networks involved. This Week in Startups is brought to you by Uber.

SPEAKER_07: Bad data equals bad AI. Your AI is only as good as the data it learns from. Uber, that's right. Uber AI Solutions now works with enterprises around the world to source, label, evaluate, and scale real-world high-quality data for every industry, everywhere, so that you can focus on building the next big thing. High-quality data equals smarter and faster AI. Uber.com slash twist. Perspective AI. Surveys? They never capture what customers are really thinking. That's why we use Perspective AI. Real insights straight from your customers. And the first two months are on us. Just go to getperspective.ai slash twist. And CLA, innovation takes balance. Our CPAs, consultants, and wealth advisors can help you get from startup to where you want to end up. Get started now at claconnect.com slash tech.

SPEAKER_09: Hey, everybody. Welcome back to Twist. And if you are in the United States, tomorrow is Thanksgiving, so I presume you are listening

SPEAKER_11: to us on the plane, from the car, wherever you are. We're glad you're spending the day before the great holiday with us. Now, on the show today, NewBird, a Twist 500 company that is building what it calls an agentic AI SRE. What does that mean? Well, it means they're applying modern AI technology to helping companies ensure that their cloud setup actually works. As the cloud becomes more complicated, there's more wires and tubes to connect, and this company wants to make sure that you never get them crossed. Then, Jason talks to Subject AI. It is a really cool company in the ed tech space. Like some others, it's helping students have more personalized educational tools and also helping teachers get rid of the busy work so that way they can spend more time one-on-one with the kids. Then, to close off, we're doing a little bit of a flashback all the way back to November of 2019 when Jason sat down with Alexander Wang, then the CEO of Scale AI, talking about the AI market and more. It is a fantastic time capsule and look into the prior era of AI.

SPEAKER_07: Let's have some fun.

SPEAKER_11: I am paying a lot of attention to applied AI. Sure, it's great to keep up with the newest models, but if you really want to understand the health of the AI market, I think you need to go and look where the rubber actually meets the road. So that's why we've talked to startups like Harvey, which is bringing AI into the legal realm. And today, we're talking to another startup working in applying AI, albeit in a very different realm. Now, NewBird AI, that's N-E-U-Bird, asked the question, what if we took modern AI tools and applied them to IT ops? And if you want to know why that matters, well, just think back to the recent AWS outage. Well, maybe these things are very important to the running of the economy. Anyways, let's learn some more. Please join me in welcoming to the show. It's Gu Rao, the CEO and co-founder of NewBird. Hey, how are you doing?

SPEAKER_14: I'm doing well, Alex. Thanks for having me.

SPEAKER_11: Oh, my absolute pleasure. I have to say that when it comes to hands-on experience in IT ops, I have precisely exactly zero. And I think a lot of folks out there, founders and all, probably are in the same boat. So maybe the best place to start, Gu, is just why do we need an AI, IT operations engineer?

SPEAKER_00: Allow me to give you just a little bit of background. I'm an engineer. You know, what engineers do is troubleshoot, fix problems. They create problems, and then they fix the problems they created. And creating and fixing the problems for some way, shape, or form typically happens at two in the morning. And you don't want to wake up at two in the morning, waking up to an IT outage saying

SPEAKER_01: that the website's not working or the payment processing system is down and you can't take transactions. So this is IT computers are something that have moved us into the new frontier over the past 20 years. And it's going to be more and more essential and critical to all of our lives. And so getting ahead of this problem and having infrastructure in place that can keep the operations running smoothly is of the utmost importance.

SPEAKER_00: So there's a lot of interest now in how to use AI, especially Gen AI, and how that can ease the burden of IT operations.

SPEAKER_11: Before we get into how you guys are doing that, though, is it correct that the state of IT operations is one of increasing ramping complexity? Because it feels like with the advent of multi-cloud setups and so forth, there's just a lot more pipes to keep connected.

SPEAKER_20: And so my impression as a layman in this context is that this is probably a problem that's getting harder and harder as time goes along.

SPEAKER_01: That's exactly what's happening. As things mature and there's more demand for new and faster software, faster products, better features, complexity increases. And people are moving at such a high velocity now, engineers, in creating product that there are many layers of technology, many components that interact with each other to deliver the features that you like. The moment you get on a phone and you're looking at a TikTok video, I mean, the number of layers that are involved to deliver that, notwithstanding the physical infrastructure to deliver the audio and video content, but the infrastructure software and physical infrastructure that it takes to store the media. It's a very complex environment and people want more of this faster.

SPEAKER_02: Just for fun, can you walk me through some of the software that goes into making a TikTok

SPEAKER_03: appear on my phone, for example? Because I think that's an interesting thought experiment to kind of talk through.

SPEAKER_00: First of all, there's the content creation and distribution aspect of things.

SPEAKER_01: The videos have to be stored somewhere. A lot of times this content is being curated by people that are brand specific. They have a tight control on their brand. They're not just putting any video up there. So there's some quality control. So there's software associated with that. And then when the video gets edited and uploaded and stored, you want to make sure that there's no data loss, no integrity compromise on the video. And then the distribution aspect of it, this now goes to people are watching this on their Verizon or T-Mobile and audio and video format doesn't need to get delivered in. And then there's edge caching. You don't want somebody looking at the video to hit the backend server each time. And so there's what's called content distribution networks involved. You guys are all familiar with the Cloudflare outage that happened a couple of months ago. And so these things, there's many layers involved.

SPEAKER_00: A video blogger goes out there and says, I put this content out there, but my customers can't reach it is the problem with the cloud distribution network is the problem with you running out of space on your AWS account.

SPEAKER_01: Like, how do you even go troubleshooting these things? It's a complex problem.

SPEAKER_23: Takes many engineers to get involved and troubleshoot these things, whether you're an engineer at TikTok or you're supporting the blogger's website.

SPEAKER_03: Is it more surprising that the internet as we know it works today, or is it more surprising when it breaks? Because I feel like given what you just described, I'm kind of shocked that everything stays

SPEAKER_05: online most of the time, Gu. It takes a lot of hard work to keep things running.

SPEAKER_00: It's great that the infrastructure works. Look, I'm surprised that any part of infrastructure that works, like when my train comes in on time, I'm really happy with it. Kudos to the people that are managing that.

SPEAKER_01: And so it's taken many years for us to perfect how to get trains running on time. And if you were even look at the aircraft industry, right, or the airline industry, that's a huge logistics issue. And so people have perfected that. And now we're getting into moving away from manually keeping things online to using AI

SPEAKER_17: to keep these kinds of systems working.

SPEAKER_11: But the complexity that we described generates so much information, logs, essentially, that I presume that AI is a good tool to be a kind of a first set of eyes on all the information that's coming in. Because me, the individual human, even if I replicate myself and sit myself at 12 different desks, probably can't look at all the logs that are coming in from even a fraction of

SPEAKER_20: TikTok's overall digital footprint to stick with that one example.

SPEAKER_00: 100%. That's where agentic systems come in. Let me define what an agentic system is.

SPEAKER_01: An agentic system is something that leverages gen AI to do what I think you called it applied AI, which is a really good term. So an agentic system takes generative AI and applies it to a very specific task domain, a problem domain. And in this case, we're talking about IT ops. And so to your point, in IT ops, what an engineer has to go through when they're debugging an issue is they're looking at a complex array of data sources. They're looking at logs. They're looking at metrics. They're looking at traces. They're looking at alerts. And each one of these have their own complex data structure complexities. And there's a lot of this data.

SPEAKER_00: Data has always been the Achilles heel for an enterprise. So yes, while people can do this, it is tedious, takes time and effort for people to go through logs, alerts, traces. AI can do this very quickly. All we do here at Newbird AI is exactly that.

SPEAKER_01: We leverage large language models, extract the reasoning capabilities, and interface them with the complex array of metrics, logs, alerts, traces, so that they can solve an IT

SPEAKER_23: issue much faster than humans can.

SPEAKER_11: And this product is called Hawkeye. You guys call it an AI IT ops engineer. I think you've also described it as an SRE in some capacity. Can you just explain to me the difference between IT ops and site reliability engineers? Why you, I think in one release said one and one release the other, this might be a small

SPEAKER_09: point, but I'm just intellectually curious.

SPEAKER_29: One of the themes we talk about over and over again on this week in startups is making sure you do your chores. I'm no expert on these things. I have some experience. Steven Estes from CLA is an expert. Let's talk about being cash efficient. Tell us about efficiency and what you see in the top tier startups in your practice.

SPEAKER_32: We're seeing kind of an interesting trend out there where companies aren't needing to

SPEAKER_33: raise quite as much as they had in the past. You really have to be careful as a founder to only take on as much money as you really need. You've got to do the forecasting. You've got to do the modeling and you've got to dial it in and get it right. Otherwise, you're going to end up either not raising enough capital to get to where you're going and you're going to have to go get venture debt or go back, have an extender to the round, or you're going to give up too much of the company because you just didn't recognize how much money you actually needed.

SPEAKER_34: Yeah. Very important to get this stuff right, folks. And that's really a bummer when startups don't do things in a button-up way. Always have a great partner, a good partner to have on this adventure.

SPEAKER_37: While things change, my friend Steven over at CLA, visit claconnect.com slash tech. And don't forget to mention that your boy Jake Al sent you. That's claconnect.com slash tech. Start today.

SPEAKER_23: Site reliability engineering was a coin turned by Google for very large enterprises where

SPEAKER_01: there are specific engineers focused on maintaining site operations. Now, depending on the size of the company you talk to, there are dedicated SREs that are completely focused on making sure that the IT operations are running up smooth. These are guys that respond to things like pager duty alerts, like it goes down, they get an alert, they have to go in and resolve the issue. Now, as you get into more mid-market, there's a blend. A lot of times engineers that are working on product or in DevOps share the role of maintaining their site operations.

SPEAKER_00: So they're doing double duty, so to speak. These products, agentic systems, address both use cases. So to talk about agentic systems in the context of site reliability engineering appeals to very large enterprises. But at the same time, we want to make sure that the same technology is helping mid-market as well.

SPEAKER_11: So essentially, I should think about SREs as very, very specialized IT ops engineers,

SPEAKER_09: normally the largest enterprises. But if you're going to sell to the mid-market and up, you're going to want to speak to both the IT ops and SRE community.

SPEAKER_01: A hundred percent. And a lot of times, you know, the engineering team that's writing code, they're on call duty. Like if the site goes down, this engineer, you have your project to do, you're working on your coding, but you're also on-call support if something happens.

SPEAKER_05: And so you want this product to be able to address the needs for both. You guys describe Hawkeye as an engineer versus an agent. Is an engineer in this context kind of like three agents in one trench coat?

SPEAKER_01: The terms and how these things resonate with people, it's still evolving. Everybody has a different expectation on what the term is called, what it does. Is it a co-pilot? Is it an agent? Is it something autonomously? Am I interacting with it? These are all things for people to figure out how they're going to live with agentic systems. We call it an engineer because Hawkeye, initially when it's deployed, it is autonomously and asynchronously figuring out what's going wrong with your IT ops and then figuring out what the issues are and creating remediation solutions. If somebody wants to interact with it and say, look, there's nothing going wrong now, but what if I were to do this? Can you go figure this out? People can do that too. Do you call that an engineer? Do you call it an agent?

SPEAKER_11: Engineer is a specific term for a general category that I would say is distinct from agents. I think co-pilots, if you go back to the early days of the post-ChatGPT era when Microsoft was talking about co-pilots, they're little assistants. They help you. I think agents to me feel more like single shot tools that go out and do a thing like, I'm going to send my agent to Amazon to buy myself candles. The reason why I kind of like that you guys are not calling this an agent is that it does feel to be like a discrete set of systems that I can work with, but also work for me, even when I'm not directly tasking them with a, please go do A on site B, right?

SPEAKER_17: So it does feel different to me in a positive way.

SPEAKER_05: You hit the nail on the head, Alex.

SPEAKER_01: And look, this is a little fluid, but that's the feedback we're getting to. You're right. A co-pilot is like, you have to distinctly carve out a body of work for it to go in and do, or you're asking it to cross-check the work that you've done, which is kind of not what's happening here.

SPEAKER_00: What we're really trying to do is streamline yours as an enterprise's IT operations, where they're getting far fewer alerts or like 80% of the alerts are being handled by Hawkeye and the remediation is done. It's automatically resolved. I'll give you a simplest use case. If somebody's wondering, like a very large company here in the Bay Area that operates in the cloud, they run an AWS, obviously cost operations is a big issue for them. All they want Hawkeye to do is see if their today's spend is projected to be greater than 5% of yesterday's baseline. And if so, why?

SPEAKER_43: You mean greater than 5% differential?

SPEAKER_00: Differential from yesterday's baseline. That's it. Simplest of use cases. You would think like, hmm, that sounds easy, but then you realize what engineers have to do.

SPEAKER_01: They have to log in, look at every instance that's launched, which instance changed size, or maybe it's even instances. Maybe somebody's consuming more storage capacity, or maybe somebody's doing a lot of egress traffic and downloading a lot of files. Like it is not that easy. And most importantly, it's kind of boring at the end of the day when you find the issue.

SPEAKER_05: So you just kind of let these agents do this on their own.

SPEAKER_11: So this seems to me expensive, because when I think about using AI to do a lot of stuff, I just begin to think about inference costs. And you guys talk about having integrations with Elasticsearch, Prometheus, IBM, PagerDuty, MongoDB, Redis, Snowflake, everybody. Which means that if I'm going to send out Hawkeye, and I'm a big cloud user who has a lot of different things to keep an eye on, it feels like that Hawkeye would need to ingest so much information and process it that it would be relatively heavy or expensive to run Am I over-indexing on gross margin fears here, or is it actually as expensive as I think?

SPEAKER_01: You are dialing in on one of the core problems that we need to solve for. Anybody can go in and download a whole bunch of logs, paste it into ChatGPT, and say what's wrong over here. What you've actually done is you've isolated the problem down to a certain area, and you're asking for a solution. I'll answer your question. The term for that is called context engineering. Quite honestly, more context, while these large language models have very large context windows, then garbage out. To answer your question on cost, if a doctor were to charge you by minute, okay, just assume, and you go in unprepared, and you say, I don't know what my problem is. It could be in my head. It could be in my back. It could be in my legs. Well, you're paying a lot of money. So what you will do then is before you go to the doctor, you're going to say, I'm going to ask myself, what is my biggest issue, isolate it, go to the doctor and say, I have

SPEAKER_00: this problem in my knee right here when I touch it like this, and you'll get a very good answer from the doctor. That's called context engineering. That's all we do at NewBird.

SPEAKER_01: We specialize in context engineering because of two reasons. We don't want the customer to incur very large inference costs. More importantly, we don't want the LLMs to come up with garbage answers or what's called

SPEAKER_00: hallucinations. When you give a marketing document to a chat GPT and say, I did this campaign last week.

SPEAKER_01: Next week, I'm going to reinvent, create a new campaign. It'll do a great job. But you can't do that with IT systems because you want Hawkeye to give you exactly what's going

SPEAKER_49: wrong with your website and why it's failing. Not a what if, not a possibility, but exact.

SPEAKER_11: Does the system have to be trained on a specific company's setup, or is it kind of something

SPEAKER_51: that you can plug into any cloud or IT environment, and it will automatically know how to look around and observe?

SPEAKER_01: The latter. It's plug and play. And the reason for this is we believe these foundational models have seen so many different IT scenarios that the specific ailment you're facing is not something that hasn't ever happened before. We're not solving for that minutia of possibilities. So then all we have to do is take all this knowledge that exists in the LLMs and apply it to your context, your data. That's what we do here at Newbird. We are context engineers. We pick the right alerts, logs, metrics, traces, so that the LLMs can diagnose the problem using reasoning. And we apply a different kind of reasoning methodologies. We have these models argue with each other, cross-check each other, consult knowledge basis to see if the reason that it's coming up with is in the realm of something that's happened before.

SPEAKER_00: It's a complex system to build agentic systems, but it all relies on good context engineering.

SPEAKER_11: I've always felt that once we could get AI models to kind of fight amongst themselves as discrete entities, we would have some really positive friction that would yield better end results. And I think that's one reason why I'm so bullish on building up more compute capacity, because I think it would be great to be able to be wasteful with inference, because then I think we can do quite a lot, but that's down the road. You guys launched Hawkeye, general availability like a year ago? Yep. A year ago. Yeah. So sitting here 12 months later, Gu, how's it going?

SPEAKER_01: Pretty good. Look, we intentionally launched this with limited availability so we could learn with key design partners. We picked enterprises of all shapes and sizes, some very large logistic companies, some very large financial institutions, pharmaceutical companies, manufacturing to all the way down to low end of the mid market. Why? Because we want to see if the product resonates well with the whole spectrum. What type of problems do you have? At the very large enterprise, a lot of noisy alerts at the very low end. They want even more surgical because they're more tied into DevOps. What part of my code change is actually causing this issue? And so we've learned a lot. We've had a lot of good success. What do I mean by that? We're achieving close to 90% plus reduction in the meantime to incident response and resolution

SPEAKER_05: from the RCAs that we're creating. A meantime of incident to response.

SPEAKER_11: So that means that if it used to take me 10 hours to get from problem to solution, it now takes me one.

SPEAKER_57: You can't make your product better without listening to your customers.

David Friedberg: But how are you supposed to actually figure out what your customers want? You can send out a survey, of course, but people just want to give the right answer in that case. They're not really giving you their honest opinion.

SPEAKER_60: Well, at This Week in Startups, we actually found the perfect answer. It's called Perspective AI. Their expertly trained AI conducts one-on-one interviews with your users based on your prompts and questions. Just tell their system whatever you want feedback on in simple language. And they take care of the rest. We've learned so much about our audience since teaming up with Perspective AI. Reginald, for example, is a founder who listens and wants more quick hit, faster paced content. And an anonymous guy from Canada gave us tons of helpful feedback about some audio syncing issues we've been having. And we fixed them. This is the kind of feedback that's invaluable when you're a founder who is product obsessed like you should be. Sign up today and get started in just a few minutes at getperspective.ai slash twist. And you'll get your first two months free. That's getperspective.ai slash twist.

SPEAKER_23: Exactly. And by the way, that's not uncommon too. And it's not just time.

SPEAKER_01: It takes an army of people. Look, you can imagine that any one engineer is an expert in networking and Kubernetes and storage. That's not possible. People have expertise in different areas. And thankfully, these LLMs sort of have cross domain expertise. They have a lot of knowledge, but they need to be told what to do. And that's where the context engineering comes from. And the hardest of problems we've been told that Hawkeye is able to save people a lot of time and at minimum tell them all the things that it did look at, which are something that the engineer, the on-call engineer shouldn't look at and dial in to where the problem could be. It doesn't have access to all the information.

SPEAKER_00: Sometimes it doesn't. Looks like I've taken care of all of this. This area, I don't have visibility. Focus on this.

SPEAKER_03: Does Hawkeye recommend methods of resolving the issue or can it actually go in there and

SPEAKER_40: make the changes or updates, corrections that I need to resolve it autonomously?

SPEAKER_01: Both. So it depends on the customer's comfort level. Typically, this has been the case when people deploy Hawkeye, they're looking at it from a consultation perspective. Tell me what the remediation is. Give me a root cause. It will create the remediation report, code change or corrective actions, but people may not give it right access to take the action and it can submit a PR and somebody would have to hit approve. That's normally the case.

SPEAKER_00: In some areas where feature flags and things like that are involved, because that's low hanging fruit, people will just, Hawkeye, go take the feature flag, enable or disable it.

SPEAKER_11: But it sounds like maybe a good way to track how smart our overall AI intelligence is becoming is as more and more Newberg customers allow Hawkeye to make those changes, it'll imply that there's increasing base model intelligence and also increasingly sophisticated applied AI techniques that you guys decide to make this entire thing feel more like magic and less like integrated systems. 100%, Alex.

SPEAKER_01: And the other thing that I'll say here for your audience, and I'm not talking specifically about Newberg, I would imagine that this is in all facets, because I think agentic systems are going to be a thing in the enterprise, not just in IT, but in all facets, marketing, sales, and people should treat agentic systems as not, you know, off the shelf software where you have a known outcome. You should treat it like it has a persona because these things are built out of human knowledge. So what do I mean by this? You have a chance to interact with it. So it's not a binary answer. You're not evaluating an agentic system for saying it's good or bad. But there's a range in between. So a lot of times these agentic systems can come back with an answer. You have time to interact with it, give it feedback and say, go do this. I like what you did, but you fell short here. Go make these changes. It is like another employee. You asked me how I have things been going. I've had evaluations come in where an SRE manager came to me and said, your product is doing really good. Hawkeye came up with an answer, but my SRE disagreed with it. Okay.

SPEAKER_00: So both were correct, Hawkeye found an issue, which was correct. SRE found an issue and the SRE had never thought of that. So it offers you different insights into your environment. It is bringing diversity. It's bringing a different chain of thought because it's been trained from so many different scenarios. And, you know, the manager was pleasantly, really happy with that.

SPEAKER_65: And ultimately solving both issues improved the site ops multiple fold.

SPEAKER_11: I think it's funny how we go from, you know, deterministic computing systems to more probabilistic. We do have to do more talking to it. Like, great. Try again. I spend a lot of time with my child GPT instance being like, excellent. Now please do it without being lazy this time. Thank you. And I think it's very, very fundamentally, are we still seeing the pace of AI model improvement that you were hoping to see? Because it does feel a little bit like from where I sit that we are seeing improvements in Chinese open source models. We're seeing some improvements in American and European closed source models, but it doesn't feel as fast goo as it used to. And I'm curious if I'm just wrong about that, or if you guys are also kind of noticing a bit

SPEAKER_00: of a. It's a, at some point, everything starts to plateau.

SPEAKER_01: And so I think you're acknowledging the fact that the rate of new features is starting to reach limits. And I think this is expected in any new system you're evolving so fast and you're putting out features so quickly and you're training these models. And when you have billions of parameters that these models have, it takes time to train them. But then the number of parameters are limited. And at some point also having more parameters may not even make sense.

SPEAKER_00: I mean, you may just start getting weird answers. So in that sense, the knowledge in the models is reaching at this point, some sort of convergence

SPEAKER_41: or, and it'll keep increasing, but the rate of increase, I think it's expected to not be at that same angle of trajectory that it was.

SPEAKER_03: Well, that's too bad. I was really hoping it was going to keep going straight up for a while there. Cause I'm, I could use some more free intelligence to kind of layer onto myself.

SPEAKER_05: You know, I would love to be smarter, but see the intelligence is going to come from external systems is what I'm saying. I think there's enough knowledge in these models to do a lot of good work. Now it's on external tools and external context. We have these things called MCP model context protocol. It's an anthropic project. Yeah.

SPEAKER_01: The core of the brain, if it's steady, then augmenting its intelligence or sort of call it, if it has enough IQ, let's get it EQ, which is the external knowledge that it would need to do awesome tasks. And that's where I think the next two or three years in AI is at least as it comes to, as you put it, applied AI is going to be around building the context and the support system

SPEAKER_11: around this really great set of brains that we have. One last question, Gu, before I let you go, we talked about there being a problem. Hawkeye notices it, figures out what's going wrong and maybe either suggests or executes a remediation. Are you guys going to eventually go in front of the issues and begin to do like proactive scanning and checking to see what might break in someone's IT infrastructure?

SPEAKER_20: Cause I feel like that would be a natural next step, but I may again, not be, not be correct.

SPEAKER_01: No, we actually do that. When somebody interfaces with Hawkeye, there's three phases to their journey. The first phase actually is retrospective, the opposite of what you said. Hey, I want to try Hawkeye. Last week I had an issue. I did what every person on the planet does. I rebooted my cluster because I don't know what the issue is. I want to get out of it. I got out of the issue by rebooting. What was the actual issue? So retrospective analysis. And then Hawkeye will say, I'm glad you got out of the issue, but this is what you should have done. Number two, real-time analysis. Your, how is my environment running? I have an alarm. What's the issue? And that's the primary use case. Big use case for Hawkeye is what you said, which is trends. What could happen if things are progressing the way it's progressing today? What will happen next week? A very good use case for that. A customer will say, I'm planning to roll out new software. Here are my changes.

SPEAKER_05: Is my environment ready to absorb these changes? Hawkeye will tell you, hell no, it's not. You got a lot of work to do. Yeah, it could be that, look, you have this much reserved memory capacity, but your pod spec is requesting more capacity than you have allocated.

SPEAKER_01: Your pods will fail to start.

SPEAKER_05: Sounds like a reasonable answer to me. Sounds like a very reasonable answer to me.

SPEAKER_11: It's newbird.ai. If you want to go check it out. N-E-U-bird.ai. And who you promised to tell me when you're closing your next major round of capital before we left.

SPEAKER_20: So I'll leave that to you.

SPEAKER_00: We are very well funded.

SPEAKER_01: Fortunate to have great investors. We're funded by Mayfield, Microsoft. We have enough money in the bank that we don't need to, but then, you know, we're always talking to investors and there are strategic relationships that we're forging without disclosing anything. I think there'll be some good news down the line soon.

SPEAKER_11: All right. Well, we'll see you on January 15th, give or take. That's what that sounds like to me. Go an absolute treat. We'll talk to you soon. Thank you so much, Alex.

SPEAKER_09: All right. Next up on the show, we have Mike Vlardo from Subject AI. Jason, this is an AI education company started off by making courses for online consumption, but has since pivoted into more tools for the classroom. We've talked to other companies in the space like Magic School, but this one is blowing up. So let's talk to Mike. Mike, welcome to the show.

SPEAKER_80: Hey, thank you so much for having me. Long time listener. First time caller is excited to be here today.

David Friedberg: Great note for startup founders. Michael, for some really intelligent reason, decided to put me on his updates to investors. This is a really great technique for founders. We train people to do this at the launch accelerator, which is you make two lists, your investors who you want to keep updated. So that's your investor update. And then your dream investors. These can be the same email or slightly different. With your existing investors, you might say, Alex, hey, here's where we're going. And here's some challenges. This is the problem. Here's the legal issue we're dealing with. Okay, right. We're getting sued by a former employee. We got a patent dispute. Okay, great. That's for the internal like 20 investors, right? Then there's the prospective investors. Now, this is just where you shine. Right. Hey, this is the third quarter in a row that we've had 20% on average growth. We are now on pace to hit 5 million in AR, 10 million in AR by this year. And we have cash in the bank. And we've just reached infinite runway because we're profitable. And then somebody like me gets this email. Now I'm BCC'd on it. And I'm like, wait a second. I've invested in 600 companies. Did I invest in Michael's company? Oh, no. Then I do a search. And there's like an email from Michael pitching me that I never got back to from four years ago or a DM. But anyway, well done, Michael. You've been doing this technique for some time, or is this the first time you did it? With AI, like everything else, the old saying is so true. Garbage in, garbage out. Without the right training data, you're not going to get great results. But my pals at Uber, they're now working with enterprises all over the world to source, label, evaluate, and scale high quality real data for your startup. Uber is one of my favorite companies in the planet because they demonstrated their ability to scale and to build great products. First in the rideshare market, then they figured out Uber Eats, and then they moved on to autonomous driving. And all of this was only possible because of a deep understanding of how to collect, label, understand, and analyze data. Now with Uber AI solutions, you can put the team and tools that turned Uber into one of the world's

SPEAKER_60: best companies to work for you. Partnering with top talent and experts from Uber's global network. What better partner to help you scale your business than the company that organizes over 36 million rides every single day? Book a demo right now by going to uber.com slash twist. That's uber.com slash twist.

SPEAKER_80: I'm doing it a little bit more often, but I do have to give you a shout out. I believe we were mentioned on a previous twist episode when we rebranded from a meal learning

SPEAKER_89: to subject and got subject.com and you mentioned it as startup of the week and seven figure domain. Now we didn't pay that much, but I do believe on the market, it could be worth that. So I always appreciated the way you gave us the nod then. And that's when I actually reached out via Twitter or X.

SPEAKER_90: And we're really excited that finally we're here together. How much did you pay?

SPEAKER_77: How much? You can't, you can't say not.

SPEAKER_90: Undisclosed amount, but, uh, you know, uh, low six figures. So we're really excited about that.

SPEAKER_92: Three or 400 grand for that is a pretty good deal. I have to begin right now.

SPEAKER_80: It's less than that. I can confirm. So less than that. And we also have subject.ai now as well.

SPEAKER_89: So we really take a lot of pride in our branding and name recognition. And, you know, we want to be synonymous with elite premium education for any age in any part of the world.

SPEAKER_95: Hey, so why don't you show us what you're building? You can share your screen if you like, and just walk us through the website or the product. I don't know if we, uh, give you a heads up to do that, but it might be nice to just show the product a little bit and what's working. Cause I know you started pre AI and then you wound up here with building AI tools, right? So yeah, tell us what you're working on.

SPEAKER_90: I could quickly voice over and really give you up to speed.

SPEAKER_89: So we were really the Netflix of education subject. I started this during my MBA experience. It was 2019. I was at UCLA. It brought me to Los Angeles 2020. I got thrown into zoom university, the worst experience of my life. 70 grand a year to sit on a zoom lecture like this. Everyone, uh, you know, had their screen blacked out, slacking each other, the answers. And, you know, in Los Angeles, it was pretty extreme lockdown. Couldn't even eat outside of the restaurant for some period. So what did you do after six, seven hours of zoom? Well, you watched tiger King last dance, all these Netflix shows. That was the saving grace and got us through that time period. So my co-founder and I really said, Hey, how can we make education feel like Netflix? How can we help students be engaged? I'm a millennial. Imagine a gen Z learner or now gen alpha in the middle of North Dakota, Nebraska. They might not have nearly the sports that UCLA was providing me. And so that was our original passion premium cinematic quality. I'm in our office in Los Angeles. We have four production studios here at 10,000 square feet where we do all our filming. But then since the launch of AI, we've really graduated past that and really providing much more adept, personalized tools for students and teachers, save them thousands of hours a year and allow all of them to feel like they're learning from their favorite teacher every single day in the classroom.

SPEAKER_95: So in the same way, Waymo and Uber and Neuro are trying to make like the perfect driver,

SPEAKER_102: you're trying to make the perfect teacher with AI, yeah?

SPEAKER_80: A hundred percent. I'm really here to amplify the heroes of society, which are teachers and coaches.

SPEAKER_89: You know, I had a lot of great teachers and coaches growing up. And our goal is to give teachers the time to spend with their students in person, making magical moments happen. And, you know, we're seeing that firsthand now with subject AI, they're able to service three times the amount of students on a daily basis by using our product. And that allows for them to have more one-on-one interactions like this, while subject AI can be the core experience in the classroom.

SPEAKER_95: This used to be called adaptive learning in the industry. It was like really hard to code, et cetera, but it's gotten easier, yeah, to do adaptive learning?

SPEAKER_90: Yeah, adaptive learning or commonly referred to also as flipped classroom,

SPEAKER_89: where you walk into a classroom and every student is engaging with subject videography, personalized AI tooling. And then the teacher is doing small group or one-on-one instruction with students who are behind or way ahead, making it much more impactful. And actually on the Andreessen Horowitz podcast, Mark and Ben have a really great episode on education, and they talk about this is very commonly known in the industry. The highest impact on a student's journey is how much one-on-one tutoring time they get. The problem is most people can't afford one-on-one tutoring, but when you work within the public school system and you allow subject to be that core methodology in the classroom, now every student has a chance of getting one-on-one time with every one of their teachers.

SPEAKER_95: So flip classroom is the technical term. You learn the repetitive stuff on your laptop,

SPEAKER_106: and then when you're stuck or you're excelling, then the teacher comes and says, oh yeah, hey, you're way far ahead. Here's some additional coursework for you to go on. Or if you're behind, let's get, let's break you through whatever the blocker is.

SPEAKER_89: Yeah. Oh, 100%. I mean, think about it from a first principle standpoint. Why should a teacher be doing the same lecture three times a day, year over year? And how much variance is there? You know, depending on your previous day, anything going on in your personal life, the teacher may be proaring at their best or worst on that given day. How can we give the teacher the time to focus on that one-on-one instruction where they love it? They get so much energy from talking to students in person, and the mundane as well as the grading and feedback is all taken care of by AI. And there's no excuse in this world of AI now to not be able to provide these teachers much more support so they can have a much better work experience as well.

SPEAKER_110: Can you just show us? Because I got, yeah, I want to see it.

SPEAKER_89: You can see it's very much like a Spotify, Netflix-esque vibe when you go through the LMS here. We really want it to always feel like premium consumer technology or social media. Our goal is to bring the best trends from those industries into education. And, you know, we did the shout out earlier about you being an Uber investor, me being an Uber, ex-Uber founder. You know, that's what Travis and Emil, who are huge, huge inspirations for me, Emil is actually one of our investors. They brought the best people to work on transportation across all industries. Our goal is to do that in education, and we're super excited about that. You could also turn into light mode, which a lot of the teachers like when they put on a projection screen, but we'll go into dark mode here. So we're super excited about the product right now. So you can quickly see here how we're able to engage students like never before. This is all premium cinematography filmed in our LA studios. You know, most of the legacy players in our space are either no videography or leveraging 30 plus minute videos. Everything we do is short form, you know, especially Gen Z and Gen Alpha. They're looking for 30 seconds to a minute clip. All of our clips are around five minutes or less now, and you can see a quick 10 second clip here.

SPEAKER_113: Hello again. I'm Miss Holly, and I'm back to discuss two important ideas to help us understand stories better. The central idea?

SPEAKER_80: Really engaging teachers and videography, but videos are only a small part of the course. As you can see on our left hand panel, this is what a typical course will look like.

SPEAKER_89: We'll have one video and then a variety of quizzes, check for understanding and exams. That way you could be able to take the entire course asynchronous and use it in a live classroom setting to be able to get through the curriculum. One of the most exciting pieces though with AI now is now we have all of our courses have video games, which really engages the students. Right now, 96% of our students complete at least one course on subject. That's record high engagement in the industry. And so that means when a student starts at subject.com or subject.ai, 96% of the time they complete at least one course. And these new AI video games are a huge driver of this.

SPEAKER_95: All right. So you're doing a video game of some type to teach storytelling and fiction.

SPEAKER_57: Yes. Got it. Wow. And these are all made with AI, I take it, or you just hire?

SPEAKER_117: Everything with AI. It's wild. Oh, I got it wrong. So that's not a good look for me.

SPEAKER_89: And I'll quickly show you one last teacher tool. So that way you can get a good snippet of that before we jump back in. You know, the educator portal and everything we do for educators is absolutely critical for our success. We live to serve teachers. We live to serve teachers and students, but teachers are the core methodology to be able to get through to these students. When you work with a district, it's very uncommon that a student would, you know, revolt to a school and say, Hey, if we don't get subject, I'm dropping out. Typically not a lot of agency around the student voice. The key is to have teachers say, Hey, my 20 students are crushing on subject. You need to get subject memberships for your students as well. And so we're really big on empowering teachers, but this is all done with AI. And this is why we're saving so much time for teachers today. We do all the grading, lesson planning and scoring and feedback for them in our console here. And so quickly you could see which students aren't passing their course.

SPEAKER_119: Within seconds, you're able to see, Oh, wow. I have five pages of students not passing. How can I do something in class today to help?

SPEAKER_95: So it's really got the whole classroom information data across all of their project progress in the adaptive learning. And you're just using a chat interface for the teacher to

SPEAKER_89: solve these problems pretty quickly. 100%. And this is what's really powerful as well, because, you know, we are in a highly regulated industry, every state has its own nuance, you could be able to go into our product and get state specific suggestions, scoring and feedback, lesson planning and worksheets within seconds. And so we're really excited about the advent of AI there and how we've been able to really accelerate our growth. So started with the premium videos, made a more short form every single year. And now everything we're really doing from here on out has been AI native, we really made the transition in early 25. We launched subject AI June of this year,

SPEAKER_126: big inflection point for subject. It's really, really, really interesting. I'm kind of blown

SPEAKER_128: away how in depth it is, it feels like you've replaced the entire school framework with a series of videos, quizzes, and so forth. That's highly automated. Does this mean that individual teachers can therefore better serve more students at once? Because I feel like if they're just focusing on one to one, the actual kind of quote, quote, quote, class size becomes less, less meaningful.

SPEAKER_80: Well, so we're really excited about and we see this in the data one, we want our students to spend more

SPEAKER_89: and more time on the product because that shows they love it. And right now our students are spending around three hours a day on the product. They love subject. But teachers, we want them to spend less time on the product. We want them to be with the students in person. We want them to be hanging out, making those magical in-person experience moments happen. And so while we've seen teacher usage stay under an hour over the last two years, we've seen triple the amount of students served. That's exactly what we're going for. Higher leverage, more opportunity for teachers to have better access to

SPEAKER_80: student in-person experience. And let us do all the mundane admin work. Let us do the grading. Let us do the scoring and feedback. Let you do what you love. That's actually the only thing you've said

SPEAKER_09: thus far that I, that I've kind of like freaked out about a little bit is the grading element of this

SPEAKER_07: because, well, actually maybe a better question to ask is what are the underlying models powering

SPEAKER_80: the teacher side of Subject AI? We've used a variety of models right now. We're leveraging Claude, which has been fantastic for us, but you know, we've used ChatGBT, Gemini. We're constantly

SPEAKER_89: looking to be able to see what's the best. And we have easily, you know, in our code base being the ability to swap in and swap out. I think the big thing for us and, you know, especially when you look at legacy competitors in our space, you know, and courseware and such, a lot of them have had the thing of being clicked for credit. You know, a lot of multiple choice, true, false answers. We never wanted that adage with Subject. And so previously we were having, you know, a teacher having to grade so many of these open form essay responses. Now with AI, this is speeding things up and allows for us to have a lot of authentic assignments while keeping up math speed and delivery to these students to get instantaneous feedback. Is there anything lost though when Claude

SPEAKER_02: grades my assignment versus the teacher who knows me and perhaps even the... I can imagine a teacher's

SPEAKER_09: syllabus being partially verbal and not entirely written down and therefore ingestible into the AI model itself. And so like, therefore the grading system might not entirely match up with what the

SPEAKER_137: teacher wants. It just seems like a place where I personally, I wouldn't put as much AI versus on the

SPEAKER_80: teaching side. Yeah. So when we do digital curriculum, everything we do is a recommended grade. And so the teacher is the final say, no matter what, when we do teacher of record,

SPEAKER_89: we typically are the final grade, but there's still always opportunity for them to amend, edit the grade if they have some feedback for us. And so we're always working with our districts. At the end of the day, the customer is always right. And so whatever the district wants to be able to do for

SPEAKER_137: their students, we're here to serve them. Can you actually explain teacher of record to folks? Because I don't think that's actually a term of art that most of our listeners know. Totally.

SPEAKER_80: And, you know, especially when any industry jargon, acronyms, et cetera, it's always like, okay, let's explain it to the greater masses. You know, when I first started in this industry,

SPEAKER_89: I had no idea what these terms meant. So basically we're accredited by two major governing bodies, WASC, Western Association of Schools and Colleges. This is very common, you know, Stanford, USC, Cal, they're all WASC accredited. There's six regional bodies across the US. And then we're also Cognia accredited, which is the international accreditation that took, you know, many years to get, and we're really proud of that. With those accreditations, we have the ability to service as teacher record for our districts. And that means that without a credentialed teacher in the classroom, we could step in on the backend and provide a credentialed experience through subject.com or subject.ai. And so with AI, that's made it a lot more scalable and high leverage. And so when they do teacher record, it's typically not a credentialed teacher in the district. We're providing the teacher of record credentialed instructor on the backend.

SPEAKER_128: And this allows for AP classes and such that might have a low student count, but then they can be brought to schools that wouldn't otherwise be able to support them was my understanding of it.

SPEAKER_80: Yeah. I mean, even though we were the first cryptocurrency accredited class in the country, that's exciting. Taught by Chad Copeland, Brandon Copeland. Financial literacy is a big one as well.

SPEAKER_89: It's a high usage for us. Brandon Copeland teaches the class. He teaches that same class at UPenn. So you have an Ivy league instructor teaching the class for high school students across the US. We're super excited. But then you also see, and this is, I'm a really passionate and, you know, I came from a small town, Kerry, Illinois, a lot of small town schools. It's hard to fill all, especially STEM courses, science and math. And so when you lose a science teacher in a rural community,

SPEAKER_80: it's very hard to backfill that. And so we can step in and help support that district by providing a four credit course with our teacher record AI. What's the impact on student ability to learn and

SPEAKER_09: be college ready? There's been a lot of reporting lately about how people get into pick your brand name school with the unable to like divide fractions and very basic things that I thought we'd handled.

SPEAKER_143: How much can you guys help get our graduating high school students up to snuff?

SPEAKER_80: I feel very confident in our ability. I think that's the big transition we're going through

SPEAKER_89: with education technology in general is how can we be able to be more part of the solution and making sure all the rigor is behind it and not allowing students to easily cheat their way through platforms. And so that's why we take academic integrity to the most seriousness and make sure that students aren't just passing. We don't want to be a credit mill. And then, you know, that's again, one of the negative terminologies in our industry. We want to make sure that students are actually learning and prepare for the next step. You know, there's over 40 million adults in the US who have college debt, but no degree. We never want to be a part of that opportunity. That is super aggravating to me, motivates me every day to make sure we're helping students either go to top colleges or go direct into the workforce. And that's what we're doing now with the career technical education product that we're debuting in 2026. So, hey, you're hearing it here first.

SPEAKER_09: So you're going to help people be essentially STEM ready out the gate. Wow. That's fantastic.

SPEAKER_95: How do you make the decision to go sell into schools, which is a slow, arduous process versus just

SPEAKER_98: doing what masterclass did or just go direct, like just go direct to consumers. How do you make that

SPEAKER_88: decision? So in 2021, we were all direct to consumer and we're only AP classes. And we got a lot of great momentum, raised a great round from Kleiner Perkins, Annie Case, our partner there,

SPEAKER_89: love working with her. Then over the summer, we saw a huge dip in signups because naturally direct to consumer and K-12 education, a lot of folks aren't active in the summer. And so we were like, hey, do we want to always engage with this, especially your part of K-12? B2B is going to be the stickier business model, but also, which is really important from the mission standpoint, it's not just affecting affluent families. Whereas B2C AP was almost all affluent family. When you're selling into districts, you know, we work with some of the most poverty stricken areas of the country, which means the world to me, you know, like we're in some incredible districts.

SPEAKER_124: How long do they take to like actually get on board when we sell the product?

SPEAKER_89: Is it a year sales cycle, two years? Because there are, you know, in the beginning. And so we really struggled when we pivoted and, you know, it was a couple of tough years, but those years made us stronger. And now we're seeing sales cycles of under six months, which is really great. But in the beginning, learning to sell to schools, especially being outside the industry, there was so much to learn and we readily acknowledge that. And we definitely had to really learn the hard way. But now the last couple of years, we have an incredible acceleration momentum. It just took us, you know, all of really 22 and 23 to understand the sales cycle and understand who even the buyer was. What were the key stakeholders? What was the ICP? And now we have great alignment on it and we're full scale mode.

SPEAKER_100: Awesome. Listen, continued success. Thanks for coming on the program. Everybody go check out subject.com or subject.ai, either one, any positions you're hiring for.

SPEAKER_89: So really talented folks who are willing to be in office five days a week in either Los Angeles or Austin. We have two offices now and we're super excited about that. And the biggest thing for us, I would say as top level engineers and data folks, we really want to continue to improve the efficacy of our data. And we believe that's the best for unlocking more sales and more product delivery. So data and engineers, please find our way. We have a lot of job openings ready to rock. We'd love to work with you. You're an in-person guy. All about in-person. Tell people why. We launched in 2020 is a pretty sensitive HR environment, say the least with COVID going on and whatnot. And so we were very flexible for the first few years and, you know, we struggled. We didn't, we didn't have a lot of momentum. We weren't scaling quick. And so I really said, Hey, why don't we try just running the company our way rather than trying to please anyone? And I come from a sports background. And so, you know, Hey, when you're playing high level sports, everyone's training together in person. So we went full hardcore five days a week in office. And the number skyrocket. And it's a hundred percent, not the right fit for everyone. I'm totally empathetic of that. But for subject, if you want to work with us, we're really passionate about building in person as a team. And, you know, one of our core values is championship sports team mentality. And so we love being able to build together.

SPEAKER_100: Love it. Awesome. Continued success. And we'll see you next time.

SPEAKER_09: One of my favorite parts of being here at twist is that we get to look back at figures that are currently making news back when we had them in the studio before they were famous,

SPEAKER_128: before the tens of billions of dollars. One of those people is Alexander Wang. Now he was in charge of scale AI, and then eventually sold about half that company to meta and became the new God of its super intelligence team. But back in 2019, on episode 1005, Jason had Alexander in the studio to talk about all things AI. And keep in mind, this is several years before ChatGPT came out. Please welcome back to the show. It's Lon Harris. Lon, we're going to do another

SPEAKER_81: flashback. And this interview is a straight banger. Yeah, this one was great. I think what's so

SPEAKER_167: interesting about this is 2020, it was like, I think spring 2020 was when all of these apps like ChatGPT and all the like stable diffusion, like the first generation of image to text ones were blowing up. To me, that feels like when all of us on like the ground level became aware of the AI revolution is real and it's happening. And these apps are like, they exist now and you can use them. So it's fascinating to go back to a time before that and hear people talk about what we were all going to be talking about within a few years, but really like none of us understood at all. Like Alexander, at some point said, this is sort of explaining to Jason, like what is training and how do you train a machine learning system? And like, how do they learn? And all of this information is now embedded in all of our minds because we've been thinking about it every day for years. But back then it was like very theoretical. You could see Jason still like wrapping his mind

SPEAKER_63: around it in some ways. Oh, absolutely. I mean, asking the difference between AI and ML and where

SPEAKER_81: does the data come from and what do you need to do with the data? Can you share data between providers?

SPEAKER_165: Alex was like, absolutely not. But let's jump in, Lon. Where do you want to start? Right away, they talk about, you know, Alexander being only 22 years old, already raising 100 million.

SPEAKER_167: I like that Jason felt a little bit of kinship with him as a person who had also been a young Buck founder who got asked a lot about his age and being so precocious. So let's take a look at that.

SPEAKER_171: I guess the thing that most people would think is remarkable is candidly that you've raised over

SPEAKER_173: a hundred million dollars in your last round of funding. That's a lot of money. Quite a bit. That's quite a bit of money from Founders Fund that I think you're 22 years old. 22. Yeah. 22. That's annoying to be young and successful because then every interview starts with your age. A little bit. It's annoying. I had it happen to me. People were like 23 year old publisher of Cyber Server, 25 year old publisher of Silicon. And I was like, why does my age matter? Now, I tell you, when you hit about 35, 40, they don't mention it anymore because they're like, well, you're 40. You should be doing interesting things or be successful in the world. But you've been running this company since you were how old? 19. 19. How did you get into the game?

SPEAKER_179: I have a fun little history. I grew up in Los Alamos, New Mexico. So both my parents are physicists and they worked at the national lab in Los Alamos. Yeah. Tell people about that lab.

SPEAKER_182: It was the lab where the atomic bomb was originally built. So the Manhattan project

SPEAKER_185: started in Los Alamos. It was very secretive at that time. Yeah. What do they call that lab? Los Alamos National Lab. Los Alamos National Lab. It's pretty boring.

SPEAKER_186: It's pretty boring. It's a government-sponsored lab.

SPEAKER_179: Exactly. Yeah. Totally government-funded. And then growing up in high school, I did a bunch of programming. I did all these coding competitions. I was getting recruiter inbounds in high school. So after high school, I actually came out here to work. I worked at this company, Quora, for a couple of years. Yeah, we know it. They do a Q&A site.

SPEAKER_191: Yeah. Q&A site. How'd you get that job? You just applied and they saw your code and they were like, okay. They are recruiter inbounds because

SPEAKER_182: I was an anonymous person on these coding competitions. And then-

SPEAKER_180: So you could just go into a coding competition. Nobody knows your age. Did you teach yourself how to code? I guess the internet. You just looked it up. You found courses online on YouTube or-

SPEAKER_179: It's hard to remember. I think I just Googled around. Anyway, I worked at Quora for a couple of years doing engineering, infrastructure, et cetera. No college. No. Well, then I went to college after that. Ah. I went to MIT and then got basically bored after a year and started scale. So you left. I left. Yeah.

SPEAKER_198: I remember, Lon, back when I was young enough to be considered a rising something or other.

SPEAKER_128: And now I'm just a middle, middle, middle, middle person. But Alexander Wang is still quite young. So his career trajectory kept going straight up. Lon, if I had given you $100 million when you were 22, how well would you have spent that money?

SPEAKER_167: Oh, not at all. I mean, this is always a fascinating thing for me, looking into the tech business where everybody gets started young. And it's all these like prodigal talents who started coding when they were 13. Like, I had no idea who I wanted to be or what I wanted to do when I was Alexander Wang's age in this video. Probably still not even Alexander Wang's age today, but definitely not when he made this video. I was like, I just graduated college. I was floating around. Maybe I want to go to grad school. Maybe I want to be a journalist. Maybe I want to write screenplays. I don't know. I was working in post-production in Hollywood doing subtitles for HBO DVDs when I was age, age. So no, I don't think I would have been in a good position to raise $100 million and decide what to do with it. One more thing I wanted to pull up there from from this opening, though, I thought a really fun little tidbit was Alexander comes by being a prodigally gifted STEM guy, pretty naturally. Both of his parents are physicists who work together at the Los Alamos Laboratory that, of course, famous if you've seen the film Oppenheimer is where the Manhattan Project was and they were doing the Trinity test. So, you know, he grew up in that kind of like a hard science environment around people who were doing like fascinating high level research.

SPEAKER_207: And next up, Alexander explains how scale provides extremely accurate data

SPEAKER_128: to the autonomous vehicle companies. This interview, Lon, really spent a lot of time talking about self-driving because, you know, in the pre-job GPT era, we didn't think a lot about, you know, chat bots and that sort of thing. AI agents weren't even a phrase. We were trying to get

SPEAKER_182: cars to stay on the road. Take a listen. The one that has really captivated the world's attention is autonomous vehicles, right? It's a compelling example because first nobody likes driving, but also driving is very unsafe. There's a lot of risk in driving. Sure. And so the- Lot at stake. Yeah, exactly. And so the captivating sort of machine learning model is one that can take in all of the camera data and other sensor data from the vehicle, understand everything that's going on around it, something that's very easy for you or I, but currently, or at least before machine

SPEAKER_214: learning was very difficult for machines, and then can determine the best path to take and figure out

SPEAKER_171: how to drive on its own, basically. Got it. So we see the lane markers, double yellow markers,

SPEAKER_173: double white markers on the highway. We know keep the car between those two lines. Exactly. As smoothly as possible. We see somebody deviate from their lane into ours. We know to slow down, give them some room, maybe they're drunk. Machine doesn't know that inherently. We have to teach it that. Exactly. What does scale.com do that Tesla and Waymo don't already do? Because they're solving that problem. Do they use your software and do they need to? The core problem, as you just laid out,

SPEAKER_182: is that machines don't know what to do unless they have data that actually tells them what they're supposed to be doing, right? And so what that means is one of the huge bottlenecks for machine learning is data. It ends up being like data that tells these algorithms, tells these models what they're supposed to be doing. That's where scale comes in. What we are is sort of this data refinery, if you will. We accept a bunch of raw data from our customers. We go through and process it, and we tell the machine what it should be doing. For example, given an image taken by a self-driving car, we would outline, these are where the people are, these are where the cars are, these are the lane

SPEAKER_185: markings, et cetera, so that over time, these algorithms can learn those things.

SPEAKER_191: And I see you are highlighting cars, you're highlighting people, and the machine is figuring

SPEAKER_173: out, okay, that's the approximate shape of a Dodge pickup truck, that's a Toyota Prius,

SPEAKER_191: and these look like the silhouettes of people. But that's a human telling the machine that's what it is

SPEAKER_182: for now. Yeah, exactly. So the core way that our whole pipeline works is that a lot of work is done behind the scenes by machines and our own AI models originally. And then humans basically give input and correct mistakes to make sure that the end data is extremely accurate. Because that ultimately is what's important for the safety of these systems and for low bias, et cetera. All these things that

SPEAKER_195: are needed, imperative for machine learning to be performing. So you would go to a customer, so they would give you videos of their cars driving, and then you would annotate it for them and put that

SPEAKER_192: data into a database somehow? That's exactly right. So for example, if they gave us a video like this,

SPEAKER_179: you'll see originally, the first step was a human drawing a box, and then a machine learning model that's already pre-processed through all this data has determined the path of that vehicle over time. Right. And then we confirm that all this is correct, and then send that data over to the customer, and they train machine learning models on top of it. Got it. And this is how, I guess,

SPEAKER_191: one of the cars got fooled. Somebody drew on the ground like an arrow turning, and a car followed the

SPEAKER_173: arrow, which a human would do too. But they basically drew a turning signal to see if like, it would fool a self-driving car. And of course it did. I didn't see this news. I would believe that

SPEAKER_228: that is how, that's what would happen basically. Yeah. Yeah. And that's what would happen to a

SPEAKER_173: human, by the way. So I thought that was the stupidest prank ever. They're like, look, we can fool a machine that's driving cars to make a wrong turn. In a lot of ways, they will have some of the

SPEAKER_182: same challenges that humans have in driving. I was wondering, when do you think we'll have a self-driving

SPEAKER_173: car in a major city like San Francisco driving a major route? What year would this be possible? 2030 and over, or under 2030? When do you think we'd first see this?

SPEAKER_182: This is the million dollar question, which is when- It's actually a trillion dollar question.

SPEAKER_233: A trillion dollar question. Let's be real. Where we're at, like, fundamentally,

SPEAKER_182: the technology is getting better and better every year. The algorithms that perceive the environments are getting a lot better, like, asymptotically better every year. And then the algorithms that figure out what the car's supposed to do, the planning algorithms, et cetera, are also getting better. So it really is, it's sort of only a matter of time before we get to the point where these kinds of routes are possible and we'll live in a safer world. So that seems to me that you're thinking

SPEAKER_171: less than 10 years from now, this will be happening with regulation and all that counted in.

SPEAKER_182: I don't think regulation is necessarily going to be that tricky. Why? There is precedent for, like, if you think of when autopilots first came about, I think there's precedent for how to think about a lot of these things.

SPEAKER_191: You mean autopilot in the airplane sense or in the Elon sense? In the airplane sense. So the FAA and whatever regulatory bodies were like, okay, we get it. Autopilot works better

SPEAKER_173: than a human. It's pretty obvious. Plenty of room up there to operate when you're up at 30,000 feet,

SPEAKER_238: a lot less room to operate when you're going through the Tenderloin. There's six people in

SPEAKER_182: the middle of the street, though. The challenges are a bit different, but I think there's precedent for how to think about these things. I think the technology, once it gets good enough, will be clearly extremely good. And so I don't think it'll be that big of a deal.

SPEAKER_240: Andrew, so you think 10 years, maybe less?

SPEAKER_182: I'm excited for the self-driving future.

SPEAKER_167: What's so interesting here, I mean, obviously, Jason having to be sort of told or instructed about here's how we're training these AI models, you know, like the cars record the video and then they send it to us. And then we annotate it with all the like helpful things that the computers need to learn and then send it back. It's interesting just that Jason needs to be sort of like, this was probably the first time he encountered, you know, the differences between what Tesla's doing with their full self-driving and what Waymo is doing with their LiDAR. And like, just thinking all this stuff out again, it's become so second nature for us today. We've all been in Waymo's. We've all, you know, tried this sort of out, or at least everybody in the tech industry sort of been thinking about it. Whereas it was still so new and they were on this, you know, horizon of this really exciting new time, which they then go to talk about. Like Jason asks him what Alexander says is the trillion dollar question, which is basically predict the future

SPEAKER_128: of self-driving cars. In terms of the future is many players, several years down the road, all doing incredibly well. Waymo expanded their geographic footprint recently. The Tesla robot taxi project is doing well. Zoox is in Vegas. Everyone's working with everybody and it's going

SPEAKER_81: very quickly. So this AI thing, I think worked out, Lon, all this hype from 2019. There is one other very

SPEAKER_167: prescient moment where they're talking about over the next 10 years. Of course, this was already six years ago. How often are we going to be in self-driving cars in 10 years? Alexander didn't want to like set an exact, he didn't want to be nailed down. But one thing that he does say is that he predicts that regulation is not going to be that tricky. That like, we have some precedent for the idea of self-driving vehicles. He mentions autopilot and planes, but you could also think of like trains or other things that are on tracks or monorails or whatever. At the time, that must've seemed like a very bold prediction. Like you think cars are going to drive themselves around cities and cities aren't going to want to like obsessively monitor that for safety. And they're not going to like regulate the hell out of that. And I think he's been sort of proved right. We're doing these pilot programs. People sort of roughly believe in this technology and trust it. And they're, they're able to sort of roll out these tests to more and more cities all the time and expand it. And there's some pushback, but I really don't feel like the regulation of self-driving cars has been super onerous and it's stopping them from collecting the data that they need.

SPEAKER_09: To the contrary, regulations for self-driving cars in the U.S. are so lax that many Chinese self-driving companies are training here. So there you go.

SPEAKER_167: That was a pretty remarkable thing to predict in 2019, when I think most of us would have expected cities to crack down very hard on this sort of thing.

SPEAKER_128: Yeah. It turns out that humans are terrible drivers. And if we're going to allow 16-year-olds to drive tanks down the highways at 100 miles an hour, maybe letting the smart computers do it, it's a better idea. Lon, the next thing I want to grab is speaking of pressions and kind of getting things right. The two of them discussed the importance of helping humans do higher value work. But now when we talk today about AI agents, increasing model intelligence, we're always thinking about what's this going to replace in the human labor pool? Is it blue collar work, white collar work? But often we're seeing AI today turn up in very high economic value areas, coding, legal work, that sort of thing. And so when Alexander Wang discusses where he sees the AI market going, it's really interesting. Take a

SPEAKER_182: listen. I think there's like helping people focus on higher and higher value work. I mean, that's sort of like the core of human progress in some sense, I strongly believe will be the actual story of AI and machine learning. And it'll have to happen more and more and more for us to be comfortable with it. A great example is like truck driving. So there's, there's a lot, all these automated truck driving companies. Yeah, lots. We work with a lot of them and Bark, Ike, etc. The naive view is that, Hey, they're just, they're going to automate truck drivers. And like, if you look at the map of the States, like truck driving is a top profession in a lot of States. And so it seems really bad, but actually we look at the system as a whole, there's a shortage. There's a national shortage of truck drivers in the United States. And the median age is like 50 or something crazy. There's this kind of like instability in the market because of all of this stuff, the automated truck driving systems, actually what they would do is automate the, the long haul middles. Yeah. Of these truck driving, boring parts, which are the boring parts, arduous, that displace people from wherever their homes are, et cetera. Yeah. And allow the current truck drivers to focus on these like higher value trips that are sort of like warehouse to a meeting point or drayage to the factory or even the last

SPEAKER_173: mile. I mean, who knows? Like maybe these trucks will change their form factor and be half the size, be automated. And when the truck gets off the road, the same truck, instead of using 18 wheelers, we might just use smaller midsize trucks. That'll be electric and solar powered. So you have more

SPEAKER_260: of them when they get off, they become the delivery truck and they just automatically start delivering. Exactly. Could be a much better model.

SPEAKER_221: These sort of like the introduction of machine learning to improve the efficiency of the economy.

SPEAKER_182: It'll be slow because of how in general free market economics work, it'll take effect in areas where there's an acute problem today that it'll happen in those places first. And it'll allow the current jobs that exist to become higher value, more impactful.

SPEAKER_167: Yeah. I feel like his truck driving example here is really good. It's a very smart illustration of the larger point that I think makes it easy to sort of visualize. Because what he's arguing is, you know, they're not going to fire all the truck drivers overnight. It's not like, AI is here. AI could drive the trucks. We don't need you anymore. It's like, there are different kinds of truck driving. There's the kind of truck driving where you're in a straight line down the highway for like three days in a row. And then there's the more intricate, you're going through a neighborhood, you're going around some sharp turns. You're the last mile of where the truck needs to go to actually make the delivery or what have you. And so he's saying, you know, like, well, AI would take part of that, the like long haul boring part, that's sort of uneventful. You don't really need a human for it. And that's the part that makes a human drive three days away from their house and then be away from their family for the whole week to get that haul done. And so he's like, well, you could have the AI do that part. And then local truck drivers take care of the more intricate parts where it helps to have a human and they get to stay in the same place. So it's like, AI makes everything better and takes care of the stuff that we don't want human staff to do. And it's like, I don't, I don't know if that's actually how all of this stuff is going to play out, but that strikes me as like the best case scenario, like the utopian vision for how AI would integrate into our lives. I'm a little more pessimistic about human labor than that,

SPEAKER_128: but I'll take the bullish scenario. Why not? I want to grab one more thing from this, from my end line, and then if you have anything else, we can go to that, but they discussed the dangers of AI. And what I found very interesting in this was just how much unbothered Alexander is. I mean that really in a completely non-joking way as a real positive. I mean, I, I do think we went through a period of AI excitement, a period of AI concern, and now mostly we're talking about economic concerns, you know, is the data center build out a bubble, but we're not talking as much about like, is AI being used an existential risk to humanity? And so to hear Jason, this is back, you know, five, six years ago, taking the tack that he did

SPEAKER_210: surprised me because this is not how he talks today. So take a listen to this.

SPEAKER_182: Well, we believe the true narrative will be extremely positive actually, versus the current narrative, which is like AI and AI, et cetera, are going to take over the world.

SPEAKER_173: There is a possibility that AI could get out of control at a certain point with exponential computing. That's not far-fetched that it could do something crazy and stupid.

SPEAKER_213: You only think that's not far-fetched because you watched a lot of these sci-fi movies.

SPEAKER_173: If you were to train an AI to work on a drug to kill cancer and you didn't program it properly, it could create a drug that was too aggressive because you didn't tell it, well, in the process of killing cancer, please don't make the person blind or all these other things. So you could just forget some edge case and some general AI might think, if you said to the general AI, you should work on things that make the human species better. He goes, okay, yeah, let's kill cancer. And then it's like, oh yeah, let's cure this communicable disease. Great. The best way to cure communicable disease is to kill everybody who have the disease currency so it can't be communicated. This sounds far-fetched, but there will be instances where they will make the wrong decision, right? Or it will be just too slow of a ramp up for us not to catch it.

SPEAKER_185: The thought experience always go like, oh, you'll make an errant comment to an AI and all of a sudden

SPEAKER_182: it'll take over the world and do something that you really don't want it to do. I mean, in reality, like there's a lot of oversight over these machine learning systems right now. There's like tens, hundreds of people who look at these models. They look at all the data that comes in and out. They like analyze everything and they try to figure out, okay, what is this model doing well? Was it doing poorly? And how do we adjust to that, et cetera? So I think that could happen in a world where it's like we have low oversight of these systems. So oversight is always important in any new technology, right? It's like when we started having airplane autopilot, for example, it would be crazy to just say, okay, we have airplane autopilot, just let it fly. Do we put any oversights over the Facebook

SPEAKER_276: and social media companies? They have, to be clear, they do- They didn't have oversight. The FCC, like giving a fine and the review mirrors oversight. It's not oversight. They had no oversight. What

SPEAKER_277: regulation is there of AI right now? There's none. You're acting under zero regulatory environment right now, and China's got a negative regulatory environment. It's true that like- So you should be regulated to your admission. No, no, no, no. That's not what I'm saying. Well, wait, wait. You just said that you should be regulated so that we don't have problems. So

SPEAKER_182: which isn't? I do think that there are a lot of important issues about how we deem what AI systems are appropriate, how we look at what they're supposed to be doing, et cetera. I do think governing bodies, the US government in particular, for example, has to take a deep look, understand the

SPEAKER_192: technology, determine what is reasonable, what is not reasonable. But even in their case,

SPEAKER_171: they're looking at the miles driven and the accidents, but they're not looking at the code that you guys are writing. They're not looking at anybody's code. They're not looking at the AI

SPEAKER_180: systems. They don't even have anybody on staff who could even write an algorithm, right? That's also changing to be clear. Is it? In general- Do you think they're looking at any lines of code at any of these systems?

SPEAKER_182: I'm not sure about the answer to that, but I do think they look at a large amount of data. So for example, these-

SPEAKER_285: Okay, they do, yeah.

SPEAKER_182: Yeah. In Europe, for example, there are all these ADAS systems, right? So there are these driver assistance programs or driver assistance systems in a lot of high-end vehicles that you buy today, right?

SPEAKER_171: Keep you in the lane.

SPEAKER_185: Yeah, exactly. They keep you in the lane. If you have stopping or traffic, you don't need to do anything.

SPEAKER_171: Yeah. Adaptive, cruise control, lane change warning.

SPEAKER_182: These systems exist. People buy these systems. People rely on these systems. And in the EU, for example, where a lot of these car makers are, where BMW, Audi, VW, et cetera, are, they have a responsibility to actually both have a large dataset they have collected themselves that is able to validate that these systems are performing well, as well as pass a series of trials and actual- Oh, really? Yeah, yeah. Different forms of data that governing bodies place in front of them.

SPEAKER_171: Well, that would be very interesting. Now that I think about it, we do crash tests for cars. You're required to give three cars or something to the government for them to just

SPEAKER_173: destroy in their crash tests. But we don't require those cars to go into a lab, get taken over by the governing body, and force them to go into real-world testing environments. Because there's

SPEAKER_191: some real-world testing environment where you do self-driving up north here, I think.

SPEAKER_192: A lot of these companies buy cheap real estate. They outfit them into these mini towns so they

SPEAKER_185: can create these funny scenarios.

SPEAKER_191: Have you ever been to one of those?

SPEAKER_185: I've never been, but I've definitely seen the- The video from there? I've seen the videos, yeah.

SPEAKER_191: Yeah, it's pretty cool. They have children come darting out,

SPEAKER_171: like little cardboard cutouts of children to see if it hits it. This way they can do that in private. But it's interesting that at some point, the government's going to have to have people who are developers and coders actually getting into the data and understanding

SPEAKER_173: some portion of this.

SPEAKER_182: At the very least, they'll have to create the driver's test, for example, the driver's license test for a self-driving car. I mean, that will exist.

SPEAKER_128: Interesting. And I think we've all just become a little bit less concerned about that because when's the last time you heard Jason talk

SPEAKER_81: about, I mean, Terminator?

SPEAKER_268: Or like they're going to use the AI to make like a biological weapon.

SPEAKER_296: That's going to blind you, yeah.

SPEAKER_167: We are definitely less concerned about all that today. And I think, yeah, Alexander was, he was a bit ahead of the time on that. I enjoy the fact that he is a little bit skeptical of science fiction. I feel like so many of the tech visionaries of the moment love science fiction and obsess over it and try to make science fiction reality. I mean, I think Bezos and Elon Musk are great examples of guys who were like, you know, they grew up reading a lot of this sci-fi and a lot of their ideas about space and robots and AI and brain implants are like, I want to bring about Star Trek and make Star Trek's future happen. And I feel like a lot of what Alexander was saying was more like aware of the sci-fi element of it, but a little bit skeptical, like he's a little bit skeptical about AGI. And like, we're, we think we're going to get there faster than we do. And, and he sort of put forth a, a very fascinating, very sci-fi feeling, but he feels the most plausible path to AGI, which is you would create an artificial life form by simulating evolution in a computer. We can't just design the final form. We would design the, like the cells or the, whatever was in the pond for AGI. And then the computer would like naturally select that up to be like a life form, like a human that was sufficiently evolved. And like, you could tell a guy who's thrown around ideas like that has read his sci-fi, but hasn't let it like, I'm going to cloud, like this is, I'm going to just try to make what Robert Heinlein was envisioning. Like he's really thinking about it in a practical way and trying to integrate it into what we really know about science and technology. I thought that was

SPEAKER_210: fascinating. If you want more about this, the life cycle of software objects, a short story by Ted Chiang,

SPEAKER_128: which was put together in his latest collection of short stories is really well worth your time. It's funny how much there is a shared language long, thanks to our science fiction heritage. And I think it's fun for me because I've read all the authors that are kind of in question here. So when they make allusions and references, it's kind of like my personal hobby. So that's a real banger.

SPEAKER_167: For the sci-fi nerds, we should point out Ted Liang also wrote the story that became that movie Arrival.

SPEAKER_128: Yes. Stories of our life. But the thing that I want to just double click on before we play this general AI clip is that he talks about how just more compute is not going to get us there. And I think that that's become less of a commonly held view in the let's get the newest NVIDIA chips and get 20,000 of them in one building and then push really hard. Meta in particular, XAI in particular, have been investing a lot in building up their compute power as a way to create more intelligent systems. So it's funny to hear him here say that that's not going to get us all the way to the

SPEAKER_298: mountaintop. What a helpful thing. That made me feel less crazy because I feel like sometimes there

SPEAKER_167: is that assumption that we don't need to like code things anymore. We just need to put enough chips in a room and then they will figure it out for us. And to me, that's always, I'm not a programmer folks. So like maybe that's how it'll work, but that always felt very counterintuitive to me. Like every other piece of software ever, we, a human had to like figure out how it was going to work. And now we're suddenly coming with this idea that, well, if you just put enough compute power together, it'll take care of all of that for us. And I've never really bought that. So it was, it was nice to hear a technologist agree with me in that way. Like, oh, I'm not crazy. This is a big assumption for us to make. It'd be fun to hear what he has to think

SPEAKER_305: about this now, but here's Alexander Wang. You believe in general AI that will hit

SPEAKER_306: that at some point? AI that is just generally smart can do anything a human can? I believe in some sense.

SPEAKER_182: I believe in the sense that like for most technological things that humans can conceive of that aren't physically impossible. If humans survive, they'll happen at some point. Like I think humans are like infinitely creative, infinitely ingenious, et cetera. I think it's very overblown. The timelines that people are talking about generally I happening. There's a lot of things that are wrong about like the common arguments. One of which is people say that if Moore's law keeps going, then we'll have all this exponential compute. It's only a matter of time before we produce these general AI Moore's laws. It's going to be dead. And then quantum computing is very far away, despite recent press releases, et cetera. I think that leg of the argument is not actually that strong. And then I also think it's not even clear that if you have infinite compute, you'll be able to produce

SPEAKER_171: general AI. I think that's very unclear. So infinite compute helps narrow AI because you're doing a number of scenarios and playing out every scenario in Go, the game, many more permutations than poker, many more permutations than chess, which is a finite data set. So yeah, more compute power on those things certainly gets you quicker ability, or even just throwing people into a random video game, like open AI is sure, definitely, but generally I taking somebody who've

SPEAKER_191: mastered chess and then saying master go and then master impressionist painting, it's different.

SPEAKER_192: It's very different. One of the arguments goes that once you have enough compute, you can create

SPEAKER_182: artificial life by basically simulating evolution. It's one of the more vogue arguments, you have so

SPEAKER_171: much compute power that you can say, start with this tiny piece of bacteria, whatever, then grow it and grow and grow an entire evolutionary system to the point at which there is a human-like species and then

SPEAKER_191: grow that human-like species in whatever number of scenarios with the big brain into whatever comes

SPEAKER_185: after us. Even if you just grow the human-like species that's as intelligent as us, then that's kind of good. That would be general AI as well. Yeah, exactly.

SPEAKER_191: Because generally I would normally, most people would define general AI as not even being smarter than us, but being as smart. Yeah. But somebody would have to code that and program that and build the systems to do that. It's not

SPEAKER_182: just going to magically happen, right? It's very unclear if that's even possible. That's an argument. I honestly think that's the most plausible argument. It's very much so science fiction in the sense that we're not close to being able to even validate the hypothesis. So I don't believe in general AI anytime soon. Lon, I just love how many people we've talked to

SPEAKER_09: over the last decade, two decades here on the show. So our archives are deep. We shall mine them. But that's all for Twist today. Lon, you're the best. We'll talk to everyone soon. Goodbye.

SPEAKER_311: Bye-bye.