Inside OpenRouter’s Tech Stack and Use of Effect
#6: Inside OpenRouter’s Tech Stack and Use of Effect
Louis Vichy, co-founder of OpenRouter, joins Johannes Schickling and Michael Arnaldi to talk about OpenRouter’s TypeScript stack, internal tooling powered by Effect, and the engineering challenges of scaling an AI platform processing trillions of tokens weekly.
Louis Vichy, co-founder of OpenRouter, joins Johannes Schickling and Michael Arnaldi to talk about OpenRouter’s TypeScript stack, internal tooling powered by Effect, and the engineering challenges of scaling an AI platform processing trillions of tokens weekly.
Effect is an ecosystem of tools for building robust, production-grade applications in TypeScript.
Song: Dosi & Aisake - Cruising [NCS Release]
Music provided by NoCopyrightSounds
Free Download/Stream: http://ncs.io/Cruising
Watch: http://ncs.lnk.to/CruisingAT/youtube
- (00:00) - Guest Intro & OpenRouter
- (07:45) - Why Everything Runs in TypeScript
- (15:43) - Scaling & Routing in OpenRouter’s Infrastructure
- (18:03) - Michael’s Story: Why Effect Was Born
- (23:11) - Effect vs. Result Types and Error Handling
- (26:04) - Culture, Hiring & Engineering Consistency
- (30:48) - The Case for Gradual Effect Adoption
- (32:30) - Generators, Pipe, and Functional Design
- (47:41) - Observability & Concurrency
- (58:35) - Agentic Systems, Orchestration & Observability
- (01:16:41) - Using Effect in OpenRouter’s Internal Tooling
Transcript
00:00you might think
00:00is just internal tooling,
00:01but we have a lot of
00:02internal tooling, just so you know.
00:03which is why Effect is
00:04actually being used a lot.
00:06So, instead of running a Python eval
00:08suite on the market,
00:10there's a bunch, we write our own, but it's
00:12using Effect.
00:12So, we actually have an eval suite
00:15and also a bulk test suite,
00:16that's all using Effect with Effect retry
00:19and so on and so forth.
00:20it's a very
00:21sophisticated system.
00:21I'm just saying is that if we use,
00:23you might think internal
00:24tooling is like some toy,
00:25but no, we're actually using
00:26it almost on a daily basis,
00:27it's actually production load.
00:29testing out all
00:29the endpoint, right?
00:30Which is a crucial part of our operation.
00:33well, I mean, almost every respond
00:34transformation is using the pipe now.
00:36And then also I'm
00:38introducing more and more generator.
00:40So, down the line, we finally will,
00:42it will be making it
00:43easier for us to transitioning
00:44over to Effect as needed.
00:49Welcome to Cause & Effect a podcast
00:52about the TypeScript
00:53library and ecosystem called Effect,
00:55helping engineers to build
00:57production-ready software.
00:58I'm your host, Johannes Schickling,
01:00and I've been building with
01:01Effect for over four years.
01:03With this podcast, I want
01:04to help others understand
01:06the powers and benefits of using Effect.
01:09In this episode, I'm
01:10talking to Louis Vichy,
01:12co-founder of OpenRouter,
01:14the unified interface for LLMs.
01:17In this conversation, we dive deep
01:19into the Cloudflare workers architecture,
01:21their routing layers, and how they're
01:23replacing their own result type
01:25implementation with Effect
01:27for better retries,
01:28observability, and long-running
01:30agentic workflows.
01:32Let's get into it.
01:34Hey, welcome everyone.
01:35It's so nice to have you here.
01:37Louis, you're from OpenRouter.
01:39Would you mind introducing yourself?
01:40We also have the Effect
01:42co-creator Michael here.
01:44Awesome. How are you all?
01:46I'm Luis. I'm a co-founder of OpenRouter.
01:48Me and my co-founder, we met
01:49through GitHub essentially.
01:51He was building out a V0 of
01:52OpenRouter using a framework I was
01:54building for my previous
01:55startup and that's how we met.
01:57Then we just love
01:58collaborating with each other and then
02:00around May is when
02:00OpenRouter first launched May 2023.
02:02So early 2023.
02:04Essentially, the idea of OpenRouter is
02:07at the bare bone,
02:08it is actually a
02:09switchboard for AI models.
02:10That's it. We reduce the switching cost
02:13between like Anthropic
02:14and OpenAI and Gemini,
02:16and Moonshot, Cohere, and
02:17Meta to essentially zero.
02:20We're also centralizing
02:21all the payment for you.
02:23Instead of like 60 different providers,
02:25you pay just us and
02:26then you get some credit,
02:27and then you can call
02:28any AI model you want.
02:31Essentially, that is the
02:33basic part of OpenRouter,
02:34but OpenRouter is
02:35actually much more than that.
02:36You have seen Elon tweet about us on X.
02:39the secondary product of
02:40OpenRouter is the data that
02:42we are allowing everyone else to access.
02:45The data about which model
02:46is being spent on the most,
02:48or which model is
02:49getting the most utility.
02:51We measure them based on tokens.
02:53A token is about roughly full characters.
02:57Last week, we did about
02:58five trillion tokens a week.
03:00Exactly a year ago though,
03:01we were doing about 150
03:03billion tokens a week.
03:05That's roughly about
03:06more than 20x growth.
03:09By the way, this feature was released
03:12literally the first month
03:14of the product, I think.
03:15Basically, my co-founder
03:16and I were thinking about,
03:17"Okay, we have this data.
03:18Should we just let everyone see it?"
03:20Because at the time,
03:23there were so many benchmarks.
03:24There were so many
03:25Llama or Meta Llama benchmark,
03:27Llama one, right?
03:28They're claiming some
03:29of these Llama fighting
03:30will be better than GPT 3.5.
03:32And we were like, "Is
03:33that actually the case?"
03:35Because when we were trying them locally,
03:37they're dumb as hell.
03:38So we were like, "Okay,
03:39let's make a benchmark
03:40that's based on real
03:42utilization, real usage, real money,
03:44being bet on the
03:45model, like a horse race.
03:47And see which one get
03:48the most of this money."
03:50And that's essentially
03:51is the secondary product,
03:53which is the ranking
03:53page where everyone can see
03:55which model is being spent the most on.
03:57And we do cap the amount
03:58of free usage on a provider.
04:00So all of that is actual usage.
04:02Like people spending money.
04:04That is awesome.
04:05So you mentioned you met
04:06your co-founder in 2023,
04:08May 2023.
04:09Now more than two years later,
04:12where are you at in terms of a company?
04:14Is it just the two of you?
04:16Probably not.
04:17So maybe you can shed some light
04:19on like OpenRouter as a company.
04:21So we were two people
04:22writing on the thing.
04:24He was checking Datadog.
04:25I was just writing
04:27code for up until January.
04:30And then we have our
04:31first engineering hire
04:32who also joined us
04:33through an open source project.
04:36We released a couple of open source
04:37project at the time,
04:38early 2024.
04:40And then Sam joined us
04:42through he sent us a PR
04:43on another project.
04:45and then we still have him to this day.
04:47last year we have four people.
04:48Now we have about 22.
04:49So team grow about 5x,
04:51but the utilization
04:52has grew more than 20x.
04:53So I think we have a decent wiggle room.
04:55We make enough of coffee, right?
04:57So got to buy more coffee.
04:58essentially the
04:59growth curve is exactly the same.
05:01The first year we went
05:02from zero to 7 billion.
05:047 billion token, you know, in a week.
05:06And that's when I showed my dad.
05:08And then a year later, September 2024,
05:11we went to about 150 billion.
05:13And that's when I
05:14show, you know, my friend
05:16is my south friend.
05:18And now, right, I'm showing you guys
05:20and a lot of people we're interviewing,
05:22we have five trillion.
05:24So it's been 20x and
05:25then more than 20x.
05:27Hopefully the next year we'll be higher,
05:29but we'll see, right?
05:30Well, AI usage is only gonna increase.
05:34And as far as I know,
05:37from friends and so on and so forth,
05:39OpenRouter is probably the best way
05:42to access those different models.
05:44We had internal
05:45conversations and one key aspect
05:49that I truly like about what you do
05:51is that you're not
05:52just a proxy to models.
05:54You actually
05:54consolidate the model responses,
05:57which is key here.
05:59So a lot of people actually
06:00mistake us as a wrapper, right?
06:02Building a wrapper is very easy,
06:04but building a dev tool is hard.
06:07Building a dev tool on top
06:09of the API is extremely hard,
06:11right, because you have
06:11to ensure that the Dev X,
06:13the user experience of the
06:15developer is actually good
06:17so that they can stick to us.
06:19And this is the key kind of mistake
06:20that people didn't think about
06:22until they need us, right?
06:23We have seen this a lot, by the way,
06:24is that user
06:25essentially didn't think too much
06:26of OpenRouter, they
06:27try it once, they're like,
06:28"Okay, I'm gonna use two or three model,
06:29"it doesn't matter, so
06:30I'm gonna do my own."
06:32A lot of people have
06:32built their own OpenRouters.
06:34And then the moment they have to say,
06:36"Okay, now I have to
06:37add Anthropic, you know,
06:39Claude 4.5 came out,
06:42they have to scramble
06:42their entire weekends, right?
06:44Either weekends or
06:45they're Tuesday and Wednesday
06:47to add a model.
06:48With OpenRouter,
06:49it's right there, right?
06:51You add the same
06:52code, you allow your user
06:53to switch up model name, you're done.
06:56And that essentially has been a case
06:58of how OpenRouter has been growing,
06:59is that every time there's a new model,
07:01we do partner with this
07:02model to launch them early,
07:04and then the moment the model is launched
07:06is outright on OpenRouter
07:08and people who miss the
07:09train, right, come back to us,
07:11and need people coming in
07:12because they also want to try the model.
07:14That's kind of
07:14essentially the growth engine
07:16we have had thus far, which
07:18is leveraging our partnership
07:20with all this new model coming in,
07:22Can you maybe categorize
07:24sort of like the products
07:26you had to build out over the years,
07:28and also maybe
07:29orthogonally the challenges
07:31that maybe you hit right away
07:33or that just appeared
07:34over time, I suppose,
07:36like working at that scale,
07:38just like pulling on that scale thread,
07:41there's probably like so
07:42many interesting challenges
07:43that are coming towards us.
07:45So the one thing that
07:45me and my co-founder really like
07:47is to have type safety, to be honest.
07:49So, I kid you not, like the
07:52initial version of OpenRouter
07:53all the models, all the endpoints,
07:55were TypeScript object,
07:57TypeScript literal
07:58type object in the code.
08:01And then the way that we do reference,
08:03we would do strict type satisfy.
08:05As a way to do literally
08:07like foreign key reference
08:08in the code.
08:09So our code were a huge
08:11database of these endpoints
08:13it's not crazy.
08:14The moment we add a new field, right,
08:16an alert, right, because
08:17of TypeScript and satisfy,
08:19an alert across all our code base saying,
08:21hey, you have to add this
08:22field in every, you know,
08:23this place.
08:24And then at the time,
08:25right, when we add new model,
08:26we add new endpoint, we
08:27just tell Devin to do it.
08:28It's automated because of type,
08:31because of type safety, right?
08:32The moment we just want to add something,
08:34we tell the model, it changed something,
08:37and then the rest is
08:37just TypeScript, you know, at type error.
08:40So it was not that crazy.
08:42it was actually usable.
08:44even satisfying a bunch of, you know,
08:45like literal number constraint as well,
08:47which is very fun.
08:49I mean, you know, fun
08:50doesn't scale, right?
08:52So we'll talk a lot more
08:54about the data in a moment,
08:55but is this entire system
08:56still written all in TypeScript
08:58or have you moved some parts
09:00that are maybe like critical
09:03to be more efficient?
09:04Are those written in Go and Rust
09:06or is everything still in TypeScript?
09:08Nope, everything.
09:10Everything's still in TypeScript.
09:12We very much just TypeScript all in.
09:12I mean, the main
09:13reason why I think is fine
09:15for OpenRouter to be strictly
09:17TypeScript is because,
09:18I mean, that won't lie, right?
09:20When we actually hit the bottleneck,
09:21we sure will switch to Go and so on,
09:23but right now we're running on top of,
09:26you know, Cloudflare.
09:28And essentially, the
09:30Cloudflare worker is much, you know,
09:32better to use TypeScript with.
09:35The tight strictness of TypeScript
09:37and also the flexible, the
09:39dynamic nature of it, right?
09:40Allow us to move much faster.
09:41The velocity of the engineers is much
09:43faster, I would argue.
09:44Technically Go can do the same.
09:46It's just that we start with TypeScript
09:47and we use TypeScript almost everywhere
09:48on both front end and back end, right?
09:50So I think it's best for a
09:53team to have the same language.
09:55I mean, time and time again, I mean, time and time again,
09:57the same story sort
09:58of like repeats, right?
10:00Yep.
10:01Like if you're looking at Facebook
10:03and Mark having written
10:05the first Facebook prototype
10:06in PHP, guess what?
10:08Still mostly in PHP.
10:10Yeah. The same for like--
10:12Instead of switching
10:13from PHP, they ended up
10:14making a new kind of
10:16Built the hack.
10:17Yeah, exactly.
10:18Or it's like, I mean, that's
10:19how TypeScript came to be.
10:20It's like, I would say
10:21the rationale of just,
10:23the whole idea, right,
10:24is that let's make sure
10:25we just speak the same language.
10:26So, in our code base,
10:28everything is TypeScript.
10:29Even the scripting is TypeScript.
10:31No bash.
10:31We don't use bash.
10:32We use ZX and TS up to write script
10:36to run like database migration or
10:38something like that.
10:39Or like even instant response scripts or
10:41something like that.
10:42So that every time that we communicate
10:43is in the same language.
10:45And so that someone
10:47else can pick it up, right?
10:48And we never really have dead code
10:50because TypeScript is
10:51very easy to kind of pull
10:53into a shared package
10:54and then you can reuse it.
10:55Honestly, I'm kind of surprised
10:57that this is not more common.
10:59In the Ruby on Rails world, for example,
11:01it's like so normal.
11:03Like obviously you
11:04build all of your scripts
11:05as Ruby scripts and then you run them.
11:07it's so surprising to
11:09me that in a JavaScript repo
11:12where like arguably
11:13it's as nice, if not nicer,
11:15to write scripts in JavaScript.
11:17This is like in the name.
11:18Like there's still
11:19often like more bash script
11:21than like custom JavaScript files.
11:24It takes about three years
11:25for someone to understand
11:26that you don't need your,
11:28database migration script to be as fast.
11:30you also don't need
11:31like a bespoke framework
11:33that you're allowed to have like a script
11:35that you can just like, even
11:36if it does only a single line,
11:38console log, hello
11:39world, like that's fine.
11:41You don't need like a
11:42bespoke CLI thingy to run it.
11:44Well, also I would also argue that it
11:45take about five years
11:46for someone to ease into all the
11:49TypeScript, you know,
11:50like not TypeScript, but most
11:52of Node.js ecosystem tooling.
11:54Right?
11:54It took a while.
11:55Because if someone has
11:56been in a Golang for a while,
11:58they probably have, they
11:59probably pollute them, you know,
12:00their home directory with
12:02all the Go binary already.
12:04I mean, I did the mistake three times.
12:05So I'm like, holy moly.
12:06I have so many of that.
12:07the zero point five version of Go,
12:09Was that you add Go in
12:10your user bin
12:13You clone it down and then
12:14every single main module,
12:15you just, you keep on cloning it.
12:17There's no, there's no Go model.
12:19Your machine becomes a Go machine.
12:21Yes. Your machine is essentially a Go machine.
12:23And then with
12:24Python, for example, right?
12:25You get used to either
12:26using UV or PIP, right?
12:28You get used to running almost everything.
12:30And so when people
12:31are, you know, kind of,
12:32used to it, right?
12:33It's really hard for them to switch gear.
12:34And this is why they
12:35having a list
12:36ensuring that when the
12:37team is still growing,
12:38we ensure that the
12:39consistency is there, right?
12:41Then there's no
12:42fragmentation of knowledge, right?
12:43And people can learn very quickly too.
12:45It's not that crazy.
12:46So tell me a bit more about
12:47how you've picked Cloudflare
12:49back then
12:50did you have a lot of
12:51experience with Cloudflare workers
12:53from previous projects and how, like,
12:55I suppose there's also a
12:57non-trivial amount of money
12:58being spent.
12:59So I'm curious how you
13:00think about this as well.
13:01It's actually very cheap.
13:02I'm not, I'm not kidding you.
13:03Like the infrastructure
13:04cost is break even completely.
13:06Just by the fee.
13:07So actually before
13:08Cloudflare, we were all on Vercel.
13:10Everything was on Vercel until they,
13:13they decided to charge
13:14for egress cost, right?
13:15but then also the,
13:17the network connection layer
13:18was not first class, right?
13:19So everything was completely abstracted.
13:21And we also have, even
13:23though we were on Vercel,
13:24we also have a Cloudflare
13:25proxy on top as well, right?
13:27For all the, you know, like
13:28for the DDoS protection and so on.
13:30So there's a three loop, three hoop.
13:32And so, I'm curious to
13:33hear a little bit more
13:34about the
13:35software aspect of like,
13:37what does a typical line of
13:39code in your TypeScript app
13:40actually look like?
13:41So currently you're
13:42running on Cloudflare workers.
13:44So in Cloudflare workers, for
13:45those who have never written
13:47a Cloudflare worker, it's essential,
13:49it's very web centric.
13:52So you have like an export default there.
13:54And in there, that's
13:56basically you export an object
13:57that returns a fetch function.
14:00And in that fetch function,
14:01you get a request coming in,
14:03you do your thing, and
14:04then you send back a response.
14:06It's like very simple, works really well.
14:08But like how you do your thing,
14:11it does like, that's the
14:12actual app that you're running.
14:13So I'm curious like how that looks like.
14:15The way I'm thinking about Cloudflare
14:17or worker in general is
14:18just the entry point, right?
14:19And if you ever kind
14:21of use any framework,
14:22entry point are just
14:24supposed to be entry point.
14:25You just try to attract your core logic
14:27as much as possible so that
14:28you can pull it into a package
14:30or a shared package so that
14:31regardless of entry point,
14:32but when we moved from
14:34Vercel to Cloudflare,
14:34it's like literally
14:35just moving entry point.
14:36And so there's no drag,
14:38so the core logic that we
14:40have is plain TypeScript
14:41function, right, with strict typing.
14:43A lot of time I would love
14:44to infer the type, right?
14:45Because I hate typing thing manually.
14:48I like to infer the
14:49type as much as possible.
14:51This is why one of my
14:52favorite brand teasers
14:53I give people is inferring type from
14:55third party library.
14:57You have to see if they have struggled
14:58through this thing before.
14:59the Cloudflare code
15:00is we just use the Cloudflare
15:01Hono library to wrap all the API
15:04under the same path and then does it.
15:06at that point, just the entry point
15:08with authentication and middleware
15:10and the rest of the core
15:11logic are plain function.
15:12Got it.
15:13So how is that like that
15:15inner chunk of business logic
15:17that's like actually
15:19what makes OpenRouter work,
15:21like where you're
15:22actually route the requests.
15:24So can you describe a
15:26little bit the architecture
15:27and like how trivial is
15:29what's happening inside there
15:31or how non-trivial, probably the latter.
15:34So maybe you can describe roughly
15:35like what is the life cycle of a request
15:38as it walks through like various stages
15:40of what makes OpenRouter, OpenRouter.
15:43let's make the tech request
15:44we have Llama for
15:45Maverick, let's say maverick.
15:46Maverick has about what like
15:49more than 10 different providers
15:51or serving the model.
15:53So when a provider is serving the model,
15:54we call that an endpoint.
15:55That's a concept that we have internally.
15:57A provider is, this company, right?
16:00Is the provider containing metadata
16:02but the provider
16:02meaning like company names,
16:04what they do or their location, right?
16:07Like the data privacy
16:08policy and stuff like that.
16:09The model is like the actual model on,
16:12that's the actual weight to the model
16:14with like context
16:15length combined those two,
16:16we call that an endpoint.
16:17let's say when we call
16:19the Llama 4 model Maverick,
16:20there's like more than 10 endpoints.
16:22The first thing is authentication, right?
16:24But authentication is
16:25literally just a function.
16:26We authenticate first.
16:27And then after you authenticate, we
16:30return a result, right?
16:31Either results, we
16:33really subscribe to the result
16:34monad paradigm.
16:36So almost every function
16:37we have returning results,
16:39we check the result and then we return
16:41a respond as needed.
16:43we would then take the request
16:44and we pass it through
16:46each of these endpoint.
16:48And while passing through the endpoint,
16:50so there are two stack that we applying
16:51for each of the requests.
16:52One stack is the
16:54routing, the routing stack.
16:55The routing stack
16:56allow us to like prioritize
16:58and order the endpoint
17:00according to what the user wants
17:01or according to some
17:02heuristic like latency,
17:04throughput or price.
17:05So the routings are a
17:07stack of decision step.
17:09And secondly, we have a plugin layers.
17:11The plugin layer allow us to like
17:13basically transform the request
17:15and also transform respond
17:16as the stream coming back up
17:18have a FIFO stack style.
17:20And that's essentially is
17:21the life cycle of requests.
17:22And then we just keep
17:23on hitting each endpoint
17:24until one that's
17:25actually, one is like saying,
17:26A, we can use a 200, we
17:28can now return some data.
17:30That's like a full cycle.
17:31And the code for all
17:32of this is very modular.
17:33Every step is a function.
17:34And then we just have an orchestration
17:36that's run through each of those
17:37would be very trivial to rewrite into
17:39Effect down the line.
17:40But we write this code
17:41before there was Effect.
17:42I think it was
17:42Effect 1.x or something.
17:44So, and I have to say
17:45just a brief comment
17:46because what you
17:47described is sounding very similar
17:52to what I had to do in my past,
17:55which is some sort of
17:56consolidation engine
17:57between different APIs
18:00that expose a single format.
18:03The reason Effect is born,
18:05like more than seven years ago now,
18:08it's because I was CTO of
18:10a Fintech company London.
18:12And we were doing consolidation
18:14over peer to peer landing providers.
18:16And we would buy and
18:18securitize those products
18:20to then sell them to
18:22institutional investors.
18:24And we had a large piece of software,
18:27which was basically we
18:29wanted to have a single format
18:30for loans, as effectively
18:34all of the loans are the same.
18:36There's a borrower, there's a lender,
18:38there's an interest rate,
18:39which is charged annually,
18:41monthly, daily, or
18:43whatever, and so on and so forth.
18:45But then you had those providers
18:46that would all have
18:49outstandingly different APIs,
18:51sometimes to achieve exactly the same.
18:54And we had a very similar
18:55approach to what you described.
18:57We also initially
18:58started with a result monad
19:01in Scala back in the day,
19:03and then testing became
19:05harder and harder and harder,
19:07to the point where we ended up
19:09doing tagless finally in
19:11Scala, which is
19:12it's absurd in complexity.
19:14It does achieve the goal
19:16of having everything
19:17modular and testable,
19:19but it is highly complex.
19:22And that's when I decided
19:23let's migrate everything
19:24to TypeScript because
19:26well, our domain is IO bound.
19:29There's no CPU compute
19:32used at any point in time
19:34except for data mapping, but even there,
19:38like you said before
19:39once you run your
19:40JavaScript in enough servers
19:43you're not really using
19:45multi-threading to
19:46handle a single request.
19:47You just have to respond
19:49to thousands of requests
19:51at the same time, but two requests can go
19:54to even two different
19:55instances and so on and so forth.
19:57Plus JavaScript is still very, very fast.
19:59So I was like, let's move
20:00everything to TypeScript.
20:02And I was punched in the face
20:04because none of the
20:04infrastructure existed
20:06that we were using in
20:07Scala for TypeScript.
20:09And that's why I built Effect in the first place.
20:12So it was precisely to be in the
20:15consolidation engine,
20:16a testable consolidation engine
20:18that had embedded open
20:19telemetry
20:20And this is what later became the open
20:22source project Effect.
20:24So I think it would be extremely good fit
20:26for actually what you're trying to do
20:28and funny enough exactly
20:29the same that I was doing.
20:31Which brings us to today
20:33where my understanding is
20:34that OpenRouter, the core
20:37product, is not yet using Effect.
20:40I think you're using Effect
20:42already for some internal tools
20:43and we can talk
20:44about that separately,
20:45but maybe addressing the
20:46elephant in the room first.
20:48You're here on the Effect podcast,
20:50where the core product of OpenRouter is
20:52not yet using Effect.
20:54Yet you've dropped the
20:55M-bomb before, the Monad bomb,
20:58which typically is a
20:59word that we're not trying
21:00to use too much because
21:02it's this interesting dynamic
21:04of what is the role of
21:06functional programming
21:07in what we're trying
21:08to achieve with Effect.
21:10My perspective on it is
21:12that we should follow the path
21:13that React has taken,
21:15where React is in essence
21:17a functional programming framework,
21:20but it does so in disguise,
21:22where it doesn't overwhelm everyone
21:26by being too dogmatic
21:27about functional programming.
21:29It's just an elegant system
21:31that tricks you into becoming
21:34a happy functional
21:35programmer without knowing.
21:37I would like to argue that
21:38because I was there back in 2015, right?
21:41When Angular and Ionic was
21:43kind of a king at the time
21:44because it's pushed by Google, right?
21:46And everyone was writing a
21:48directive and writing, you know,
21:50some, basically it's very
21:51much functional programming,
21:52but it was a bloat.
21:53It was bloating so
21:55much that the key sell,
21:57at least at the time I was still young,
21:58I was still inexperienced,
22:00but when I listened to
22:01people talking about,
22:02hey
22:02if you adopt functional programming,
22:04your program will be very, very lean.
22:06And I think there was
22:07that push of just whole ass
22:09in functional
22:10programming, don't be ashamed of it.
22:11And literally say, fuck
22:13object-oriented programming.
22:14I think that was a push.
22:16I would argue you should be,
22:18you should not be ashamed
22:18of functional programming.
22:19And even if they push it,
22:21I would say, I mean, sure,
22:22then just go slow, you know,
22:24then you have a slow code base.
22:27I wouldn't say we're ashamed of it.
22:29I think we're just
22:30trying to like not make it
22:32like an obstacle.
22:33So I think we've so far
22:36traversed the line pretty well
22:38by like inviting
22:40everyone who is FP positive,
22:44but we've also not tried
22:45to make it define the area
22:47to the barrier to entry through
22:50functional programming.
22:51so there was that guy, I forgot his name,
22:54one of the React engineers.
22:55Dan Abramov?
22:57Yeah, Dan Abramov, yes, that guy.
22:59I think he has a good
23:00sense of community there.
23:02I'm curious what he would say about this.
23:03Yeah, great point.
23:05We should invite him to
23:06the show and see whether.
23:08But I would say that like the result,
23:11I mean, the reason why we
23:12reached for the result monad,
23:14I mean, sure, most of my
23:16previous project were try catch,
23:18or like, you
23:18don't even try catch by the way.
23:20I just have a single error
23:22handler at the top of my apps.
23:24but then
23:25whatever it try, right?
23:27Whatever error is being
23:28thrown in underneath, right?
23:29Is an error type that
23:30I have defined, right?
23:32The problem is, down the line, right?
23:34You soon learn that it's
23:35very hard to debug those stuff,
23:37right?
23:37You soon learn that the stack tray
23:39can be very, very
23:40long and need to function
23:41is like any anything complex.
23:43It'll be very hard to go
23:45through the stack trays
23:46because you catching
23:47it on the top, right?
23:48So now you always want to
23:49somehow manage your error
23:50or localize your error
23:52at the call site,
23:53and then when you start doing that,
23:55you have nested try catch and
23:56it's a pain in the ass to read,
23:57right? Because every nested layer is,
24:00I review my code on
24:01my phone a lot.\
24:02And yes,
24:02almost every try catch
24:04kind of nesting layer is like,
24:05I just have to scroll on one side.
24:07I just want to scroll on
24:09one, you know, vertically.
24:11The result monad
24:11essentially solve this problem,
24:12By explicitly forcing you
24:15to handle the error case
24:16as a value.
24:17I think is a value add and it is weird
24:19I mean, it is surely a different syntax.
24:21I think people have to,
24:22they just have to do it.
24:24It's the same as when
24:26React first came out, right?
24:27With useState.
24:28If you say useState is an
24:30equivalent to the class state,
24:33that's completely wrong.
24:34Like class state is,
24:36you cannot just check it in and out.
24:38With useState of React,
24:39you can just take it in
24:40and out fully functional.
24:41And then now your component is so light.
24:44And that was, I would
24:46argue the main selling point
24:47at the time.
24:47because of the pain, right?
24:48The pain of, oh my God,
24:50my class component is
24:51so slow or so bloated
24:53that with this thing
24:54now, it will be so faster.
24:56And it has to adopt
24:57functional programming.
24:58So I'm curious to hear a little bit more
25:00of like when you grew
25:02the engineering team
25:03and like you as the founders,
25:05you had the privilege
25:06of like later foundations
25:08or like set the taste as like,
25:09hey, this is how we're doing things.
25:11I think that's probably
25:12still holds to some degree,
25:14sure, like new people joining,
25:16bring in like their opinions.
25:18But I'm curious like which
25:20sort of mini culture clashes
25:22you've experienced by hiring someone new
25:25and they look at this and it's like,
25:27what is that?
25:28How did you explain that?
25:29And how did you sort of
25:31like culturally assimilate them
25:33to those ideas?
25:35Well, I think we just have
25:37to hire very good engineers
25:38and good engineers with
25:39experience usually can swallow.
25:42Yep.
25:43Oh, I mean, swallow not in a bad way,
25:45but kind of like they have
25:46seen things before, right?
25:47It's not something crazy, right?
25:49I would argue the crazy part
25:50is we have like bash script there,
25:52bash script with
25:53different syntax of bash,
25:54different flavor bash even, right?
25:56And then we have
25:57TypeScript, we have coffee
25:58and then we have like,
26:00maybe we use zest here and we did there.
26:02That is the crazy part.
26:04I would say that as we
26:05grow the team, right?
26:06We have tried our best to
26:08if they tech debt, take
26:09tech debt is widespread.
26:10If they have to tech debt,
26:11it widespread is unified.
26:13So when we refactor, we refactor all of
26:15them at the same time.
26:16That's one key part of it.
26:18If there is a certain framework we use,
26:20we stick to it.
26:21So like, for example,
26:21right now we're using
26:23a lot of our front-end,
26:24our like React, right?
26:26We know how to spend,
26:27we are going to spin up a stencil,
26:30you know, web component anytime soon.
26:32The next one is probably will we react?
26:34We'll stick to react for a while
26:36until we decide, okay,
26:37I think we, you know,
26:38let's move everything, uproot everything
26:40from react to stencil.
26:41Sure, we'll have the
26:42discussion, but not right now.
26:44And then same with TypeScript.
26:45The fact that we picked
26:46TypeScript and forced everyone to,
26:49hey, get used to it,
26:50write script in TypeScript,
26:51do everything in TypeScript
26:52as much as possible, right?
26:54That really forced us to
26:55just keep on learning.
26:56Even though they were
26:57like from a Python shop
26:58of GoShop or something else, when they
27:00come in here, right?
27:01I mean, sure, there's a resistant layer,
27:03a resistant period where they're like,
27:05how can I introduce a Go, you know,
27:07the Go binary a little bit?
27:09But then over time, right?
27:11they start to get like used
27:13to the TypeScript paradigm
27:14because it's everywhere
27:15in the code base, right?
27:15You can have the AI agent look it up.
27:18Also like Linter, we're using
27:19biome right now for Linting,
27:21plus Grid.
27:22So we've been starting
27:23adding a bunch of this Grid,
27:24custom rules, which is very nice, right?
27:27Because you can do like a
27:28crazy regex on the code.
27:30I'm using the same.
27:31This is like, allows you
27:32really nicely to define
27:34some like, hey, this is
27:35our way of doing things
27:36from like all the 10 different,
27:38TypeScript engineers
27:38and they have 10 different
27:40preferences of like braces there,
27:43no braces there.
27:44And then you can basically say,
27:45this is the OpenRouter
27:46way and run a command
27:48and it gets linted and formatted.
27:51Yeah, it is very awesome.
27:52Like for example, we have a
27:53rule that's strictly saying,
27:54hey, if you have more
27:55than three position argument,
27:56just make an object.
27:58Plus I feel like experienced engineers
28:01usually focus on the problem
28:05and the way we solve the problem,
28:08rather than picking on the how,
28:11in the sense of like
28:13the thing you described,
28:15that you use a result type for,
28:17not gonna use result monad,
28:19because monad is a term I don't wanna use
28:23and not because I don't trust monad,
28:27but because a monad would be an
28:29abstraction on top of it,
28:30which is the thing we're
28:31trying to stay away from.
28:33Result types are great
28:35and they solve the problems you have.
28:39They solve the nesting of try-catches
28:41and so on and so forth. So an experienced engineer that comes in
28:44and sees the solution
28:45might disagree on the syntax,
28:47But the problem it's resolving,
28:49it's clear they had it before,
28:52they got tired of it
28:53from other code bases
28:55and they just digest
28:56the current solution.
28:58I wouldn't be opposed to
28:59contribute to a code base
29:00where there's a custom result type
29:03and effect is not yet used.
29:05I would say,
29:05so that apply for like
29:07a more senior engineer.
29:08For the
29:09slightly less senior engineer though,
29:11I think the main thing is curiosity.
29:13If they curious about why this was done,
29:15they'll learn sure
29:16learning faster on it too.
29:18Because if you imagine you
29:19can hire a very junior person,
29:20but also very hard headed on some stuff
29:23they've done before,
29:24then they might not budge at all.
29:26So I think for the less senior,
29:28curiosity is where we strive for the most
29:30and the way that we test
29:32was curiosity is very simple.
29:34They asked good question, really is.
29:36So I think solving challenging problem
29:38and then dig deep into it,
29:42show that they have clear ownership
29:44and clear digging, right?
29:46The more they can dig, the better it is,
29:47I think they show curiosity in a sense.
29:49So taking a slight step back of like,
29:52now you probably have a healthy mix
29:54of relatively experienced engineers
29:56and maybe some slightly more junior,
29:59but very curious engineers.
30:01And so you have right now,
30:03a very sizable TypeScript code base.
30:06Not sure, probably in the
30:08six digits lines of code,
30:10TypeScript repo, maybe even larger.
30:13And you clearly have
30:16Effect on your radar.
30:17You've mentioned that some internal tools
30:19are written in Effect,
30:21but the main product is
30:23not yet built with Effect.
30:25And maybe you can just walk us through
30:27and describe a little bit of like the
30:29internal conversations
30:31where you're considering,
30:32hey, should we use Effect
30:34for more parts of the core of the system?
30:37Which sort of like
30:38resistance questions you're getting,
30:40how do people who are
30:41already sold on Effect,
30:43how they make a case for effect
30:45and like just sharing your observations.
30:48Well, the main observation
30:48I have with the overall team
30:50is almost every time
30:51someone bringing a new technology
30:52is a shiny object syndrome.
30:55So very tough to sell, right?
30:57in terms of handling errors,
30:58we already have our own result monad
31:00or result type that we already have,
31:03which serve the purpose just fine.
31:05and then we already have
31:06this paradigm of writing,
31:06make sure you write functions.
31:08So when engineer, when
31:09engineer look at Effect, right?
31:10And the way it
31:10orchestrating, you know, function, right?
31:13A lot of time, the main question is,
31:15why can't we use co-function then?
31:16Why do we have to go
31:17through Effect piping
31:19or Effect matching when
31:20a function work just fine
31:22and an if and else work just fine too?
31:24And if needed, we can
31:25just make a new function,
31:26do a switch cases with
31:27complete check, right?
31:29And plop it into the current,
31:31functional flow, right?
31:33And I mean, my response a lot to that is,
31:35okay, what if you need to add a logger?
31:37What if you need to add
31:37a sidecar logger, right?
31:39What if you need to add
31:40observability or trace or span
31:41on top of, you know,
31:42this function call, right?
31:43Well, the answer to a
31:45lot of time would be,
31:46well, I just add a new line, right?
31:47Right before it.
31:49Well, except that doesn't work for spans.
31:51Precisely, right?
31:52A lot of time, it doesn't work at all.
31:53I think the main idea is
31:55people don't really need
31:56it at the moment
31:57And so there's no huge push.
31:59There's not, for example,
32:00if we don't add Effect today,
32:01we will still be around.
32:02We will not be dying.
32:03So that is why I would
32:04say the main pushback,
32:05because my team is
32:06also very rational, right?
32:07You know
32:07they're also very busy, right?
32:09it's not like,
32:09it's spending time kind
32:11of learning the namespace,
32:13the Effect namespace, right?
32:14Is already a drag.
32:15Learning pipe is
32:16already a drag, by the way.
32:18So this is why the
32:19first thing that I'm doing
32:20is just introducing
32:21small little usage of Effect.
32:23We're using pipe almost everywhere now
32:25when we composing a bunch of
32:27function in step by step, right?
32:28And that helped a lot.
32:29But the pipe function, by the way,
32:30one of the main questions I
32:31have for you guys actually
32:33is a polymorphism of the pipe function.
32:36A lot of engineers actually thought
32:37that the pipe function is too flexible.
32:39It's allowed you to
32:40transform the type, right?
32:41So it should have been called transform.
32:43Transform pipe maybe, not pipe,
32:45But to some engineers,
32:47when we think about pipe
32:48They assume it's
32:49literally just a pipe through
32:51where the type,
32:52the input type should be
32:53the same as the output type.
32:54And it just
32:54transformation step in between.
32:56I'm curious what you guys
32:56think about this, by the way.
32:57What do you think about polymorphism?
32:59Because essentially, the
33:00this is the case where the
33:01pipe type is not very scalable.
33:02It only has 20 letters
33:04You can only pipe 20 times.
33:06I think there's reasons
33:08why you don't wanna pipe
33:09more than a number of elements.
33:12And that's more along
33:14the lines of limitations
33:16in type inference
33:17capabilities of TypeScript
33:19and speed of type checking.
33:21So we could easily add
33:24200 arguments to pipe.
33:27The actual implementation supports any
33:30number of arguments.
33:31The issue is TypeScript cannot deal
33:34with the full polymorphism.
33:36So you need to add specific overloads.
33:38And before AI was very
33:40tedious to add those overloads.
33:43Now with Effect 4, I got Claude
33:46to add the various pipe signatures.
33:49I can easily tell Claude add another 200
33:52so Louis can be happy.
33:54that would be a solution.
33:55But generally speaking about the
33:58polymorphisms arguments,
34:00pipe is generally applied to functions.
34:03So you're piping the output of one
34:05function to the input
34:07of another function.
34:08In that case, there's no
34:10restriction about in and out.
34:12You have data that flows
34:14across a number of functions.
34:16If it's understood in this
34:17way, then it's not so confusing.
34:20The confusion that I also had,
34:24and at some point I was evaluating this,
34:27is should the Effect
34:29pipe be tied to Effect?
34:31But then you have
34:32cases where you might want
34:34to convert an Effect to a stream.
34:37You might want to repeat an
34:39Effect
34:39So you're having like slight
34:41differences between the types.
34:43So, I think putting an
34:44arbitrary restriction
34:46doesn't really help
34:48in the understanding.
34:49Now it is a good practice in your pipes
34:53to not change the type
34:54from the beginning to the end.
34:56Otherwise while you
34:57read, you lose context.
35:00Like if you change the
35:01type, most likely the best idea
35:03would be to slice those
35:05into two different pipes
35:07and have variable
35:08names that clearly state
35:10what you're doing.
35:11So this, I think it's
35:13sort of best practices
35:14and so on and so forth.
35:16And I think with
35:18touching on another point
35:19that you mentioned, it's
35:21the familiarity concept
35:23and people's brain is wired into
35:26imperative programming.
35:28By imperative
35:29programming, I mean the style.
35:31Not the fact that
35:32operations are actually executed
35:34in an imperative sense.
35:36I simply mean developers are used to
35:38write normal functions.
35:40Are used to write if statements.
35:41They're used to write while.
35:42They're used to write for.
35:44They're used to write all of this.
35:46With promises at the beginning,
35:49we didn't get async await
35:51And it was very tedious to write programs
35:54because you had to think
35:55in terms of this data flow,
35:57this sequential nested data flow.
36:00We escaped callback hell
36:02to enter a venable hell
36:06because then you had to access the
36:08property from before,
36:10then async await and
36:11everything flattened out beautifully.
36:14That you don't actually
36:16get with a result type.
36:18With a result type,
36:20you almost end up writing
36:22what they call Golang in TypeScript,
36:25which is you call a
36:26function then you have,
36:28if you don't wanna end up in
36:29pipes
36:30what you end up doing
36:31is you call a function
36:32then you have if there was an error,
36:35return the error, otherwise keep going.
36:37If there was an error, well, in
36:40JavaScript and in TypeScript
36:42we have generators, which we rely heavily
36:44in Effect to make code look
36:48in the way that developers
36:50are used to work with.
36:52The mental model should be the same.
36:55So my personal stance is that
36:58pipe should not be the
37:01default way we do things.
37:02We should write code in a way that
37:05everybody can understand.
37:06And if you use a
37:07generator, like switching mentally
37:10from yield to await
37:12it's not rocket science.
37:14After you've read 10 generators,
37:16you're gonna understand that you have a
37:19direct mental mapping
37:20that yield equals await, that's it.
37:23And you apply your knowledge.
37:24You can use it in a for loop,
37:26you can use it in a while loop,
37:27you can use it in an if condition.
37:29You can do all that you want.
37:32And if you want to access the specific
37:35result of something,
37:36you can say, okay, yield result of that.
37:40And then you have
37:40your if else that checks
37:42for the exact
37:43pattern that you would have
37:44with a result type.
37:46So I think like 90% of code
37:49should go in that direction.
37:52The rest of the code is
37:54higher order behavior.
37:56Like I have a
37:56computation, I wanna retry it.
37:59I don't wanna write a for loop.
38:01I don't wanna deal with exponential
38:03back off and everything.
38:05Like you are integrating different APIs.
38:07I think at some point
38:08you have some retries.
38:10At some point you have
38:12to decide to back off
38:13from the retries
38:14because maybe open AI is down
38:16and you all wanna keep
38:17hitting the endpoint.
38:18But those elements are
38:20really not imperative.
38:22They're not business
38:23logic, they're not core.
38:25They're just behaviors
38:26that you add to the function.
38:28And for that, I like to
38:29do .pipe(Effect.retry).
38:31That's it.
38:32An LLM can understand it,
38:34but a user doing
38:36review can understand it.
38:37And even a person
38:38that doesn't know Effect
38:40can read the function.
38:42Okay, I understand the business logic
38:44because it's all if else A equal B
38:47and if these then that
38:49while these conditions
38:51through keep going.
38:53And then they read pipe effect retry
38:57while the error is network error.
39:00Do not retry if the
39:01response is malformed.
39:03Cause if you call an API
39:04and they change the response,
39:06your decode logic will
39:07keep failing anytime you retry.
39:09So you might retry 500,000 times,
39:12not getting anywhere.
39:14So like to me, I
39:16really wanna make it simple
39:17for developers to read the code
39:20and use pipe and use generators as tools
39:24to really express business logic and
39:26higher order behavior.
39:28And I think like you are
39:29in the very good direction
39:31with the code base that you have.
39:33And adding concepts one by one
39:37is the right way to do it.
39:39And from our private
39:40discussions in the past,
39:42I also see for example, you
39:44use a lot of pattern matching.
39:46Why the code becomes readable,
39:48much more readable than
39:49a sequence of if statements.
39:51I would say that I actually consider
39:54building everything using generator
39:56but the code base started two years ago
39:58and generator did not
39:59work very well with async.
40:01Async generator was on
40:02a thing two years ago.
40:03And you should not use async generators.
40:06Yeah, you would not using it.
40:08Now I would love to hear this by the way.
40:10Why?
40:10Why shouldn't you not
40:12use async generators?
40:13You should not have two
40:15different representations
40:16for sync code and async code.
40:18That's madness.
40:19That's actually also the
40:20reason why I actually look up
40:21by Effect by the way.
40:22Oh, actually I look up what is it?
40:25Function JS.
40:27Because we had the result
40:28and then we have async result.
40:31Yes.
40:31And you cannot escape from that.
40:34So I think just to zoom out a little bit
40:37and just to kind of like
40:39observe the reflection here.
40:41I think what makes your current code base
40:45and your current programming styles
40:47a little bit distinct from others
40:49is that you went into this
40:52with a very strong opinions.
40:54Like, hey, we're not doing
40:55try catch in this house.
40:57And so you went like
40:59this proper error handling.
41:01And I think with that,
41:03you're like way ahead
41:04of where most TypeScript developers are.
41:07most Effect adoption
41:09I think happens to like
41:10wherever the pain is most acute.
41:12Maybe going to the doctor,
41:14like you say like, oh my
41:15gosh, this hurts so much.
41:16But like while you're at the doctor,
41:18you also realize like
41:19you have those other things
41:20that need treatment, but like one thing
41:21gets you to the doctor.
41:23And so the error handling thing,
41:25like that pain like you've already
41:27relieved yourself of that.
41:29Yes, you have
41:29appreciation for these other things.
41:33Most people come to
41:34Effect for error handling.
41:36Also a lot of for other things,
41:38but I think now
41:39with the patterns
41:40that you've like ended up with,
41:42like just by talking through it.
41:44Like I think we've like already uncovered
41:46a couple of things where you say like,
41:48well, yeah, actually that's not ideal,
41:50but like, you know, it's like we have
41:52good error handling.
41:53But what Mike calls sort
41:54of like the Golang style
41:56of writing TypeScript.
41:57I have done that myself as well.
42:00And like this works
42:02until it no longer works
42:03or like until it just gets very annoying.
42:06And I think the, what will
42:08probably be the pain point,
42:10like group of pain points
42:11that gets you to the doctor
42:13might be like better
42:14retrying, might be observability,
42:17might be like what
42:18you call it like logger,
42:19maybe metrics, et cetera.
42:21And I think this will
42:22make it actually easier
42:23for you to adopt Effect
42:25because you don't need
42:25to rewire your brain.
42:27So a lot of time we have been
42:29incrementally adopting
42:30some Effect, right?
42:31Cause you know, lodash is a bloat
42:34And I already added, I
42:36already use my authority
42:37to add Effect to the code base, right?
42:39As I'm saying, we added Effect.
42:41You're still
42:42retaining your CTO authority.
42:44Yeah, by the way, title wise, I'm not,
42:47I don't claim CTO.
42:48I'm just a co-founder
42:49engineer title wise.
42:51Yeah.
42:52Perhaps there's a
42:52better CTO out there for us.
42:54We will see.
42:55But I don't care too much about title,
42:56but I care more about like
42:58the dictator title for sure.
43:00I'm a dictator for like BDFL for sure.
43:02Yeah.
43:03I can push the man by the way.
43:04So like, for example,
43:06You don't call a CTO, but
43:08you get called a dictator.
43:10Yeah, I'm a DFL.
43:12And then, I mean, I was
43:13saying like last week,
43:14last week for the OpenAI dev day,
43:16they were launching
43:16the eval product, right?
43:17And then right the Monday before,
43:20they were paying a slack and saying,
43:22hey, how come the
43:22responding API don't have cost?
43:24I just came right on and just, you know,
43:26like push a thing on
43:27man, two man in 30 minutes.
43:29Shut up, right?
43:30Get it over with, you know, like,
43:31and that's the kind of thing I think is,
43:33as a co-founder you
43:34kind of have to do, right?
43:35Is you gotta hold it by, you know,
43:36you gotta hold it by
43:37your throat and shove it.
43:39Yeah, you have to know
43:40when to skip the steps
43:41and you have to know when
43:42to impose a long-term vision
43:45that is not yet apparent.
43:47And sometimes you're wrong, like,
43:49I do the same with Effect.
43:50Sometimes I'm wrong.
43:52I think I'm wrong and now, by the way,
43:53there's one thing that
43:53I might have been wrong,
43:54which is like, I'm too ho-ass
43:57into this idea of not using try-catch,
44:00but then when it
44:00comes to like the logging
44:02or the tracing server,
44:03you kind of have to use try-catch
44:04so you don't allocate a new variable.
44:06Because that way it's very easy to make a
44:09copy of the variable
44:09or when you do JSON parsing,
44:11you make a copy of the variable
44:13of a very deep nested
44:14object of every value,
44:16out of memory very easily.
44:18Exactly.
44:18And I think this is the thing where like
44:21when people are sort of like in the,
44:23the denial phase of Effect
44:26and sort of like they, they
44:27like justify to themselves,
44:28like, we don't need Effect,
44:30like we can do this thing like that.
44:32Okay, yes, you can do
44:33that thing like that,
44:34like you do for error handling,
44:36but like what if you need the,
44:37if you wanna do the other thing,
44:39if you wanna add that as well.
44:41And like then, like all of those things,
44:44they kind of like as you introduce them
44:45on top of each other,
44:47they don't seem to like
44:48linearly add complexity,
44:51but that's, I guess
44:52like one of the unfortunate
44:54emergent situations about TypeScript
44:56is like the more you
44:57add those requirements,
44:59like observability,
45:00better error handling,
45:02dependency
45:03management, like context management,
45:05that doesn't make your code
45:07like linearly more complex,
45:08but exponentially more complex,
45:10also visually more like visually,
45:13exponentially more unwieldy.
45:15And I think this is
45:16where Effect comes in.
45:17We've been adopting effect
45:18in a bunch of like our retry
45:20and like backup handling logic as well,
45:22because the API elegant, right?
45:24That's the whole idea.
45:25So why would you write your own
45:27when you can just pop this thing in,
45:28throw a fetch and it's done, right?
45:31And to review,
45:32I don't think anyone
45:33should adopt Effect in one day,
45:36like in any reasonably large code base,
45:39it would be absurd to
45:41think one would rewrite
45:43the whole code base.
45:44we call effect a good virus,
45:47in the sense that it tends
45:48to spread when you started,
45:51because now you have this retry,
45:53but then you call a lot of APIs.
45:55Obviously you are an API
45:57consolidation provider.
45:59So testing is essential.
46:01You have to do integration testing,
46:03you have to do end-to-end testing,
46:05you might wanna do some unit testing.
46:07And then you are in a
46:08madness of Jest mocks
46:11or Mocha mocks or whatever framework
46:14you're gonna use Vitest mocks.
46:16But mocks are really a very
46:19complex thing to deal with,
46:20because all your code is eager.
46:23If you call it in the wrong place,
46:25you have one piece
46:26that is using one module,
46:27another piece that is
46:28using another module.
46:30And so you might wanna elect,
46:33use dependency injection.
46:35I have seen how Epic Fabric
46:37and their dependency
46:38injection would help with that a lot.
46:40Yeah.
46:40Very good to use it, but you know,
46:42I mean the team has
46:43been using Vitest mock,
46:45do mock and stuff for a long time.
46:47By the way, we switched from Jest to Vite.
46:49And that was a feat.
46:51I know, I mean it's faster,
46:52it's like the test runs
46:5310 times faster, right?
46:54We switched from prettier to biome.
46:57I'm saying
46:58we are not foreign to adopting new thing.
47:01But it truly is all because,
47:03it's pushed come to shove, right?
47:05For example, if the test
47:06take about five minutes to run,
47:07oh my God, okay, let's
47:09try to shave this down
47:11and stuff like that.
47:12Yeah, and to be frank,
47:13I think like for your specific use case,
47:17I think one of the
47:18highest leverage points
47:20could actually be the observability.
47:22Because you already
47:23referenced to debugging troubles
47:25in the past of those,
47:28once you have a hundred
47:29of different functions,
47:30the stack traces becomes unreadable.
47:33Yes, they are.
47:34They're much less unreadable
47:36if you have spans at the right places
47:38and you have a full dashboard
47:39that tells you if something goes wrong.
47:41The one thing with
47:41observability and OpenRouter
47:43is that because we run
47:44the thing on the edge
47:46and the worker runtime,
47:47we don't truly have a span server
47:49to continuously collecting the span.
47:52So right now we're actually hooking
47:54into the Node.js diagnostic channel,
47:56which is not very good, I would say.
47:59At least on the worker runtime
48:00because the memory is very limited.
48:02So we have to be very
48:03careful with what we trace.
48:05Well, I wish at some point,
48:08we're gonna get to agentic behavior,
48:10long-running processes
48:11at some point in time
48:12in this discussion.
48:14Yeah, so are you familiar with the ECS or
48:18the entity component
48:19system in game design?
48:21Yeah, I think the reason the actor model
48:25in general is very popular in game design
48:29is because in games you usually have a
48:32lot of parallel entities
48:34and they do share messages with each
48:36other. It's a very
48:38interactive world. Basically, again, it's
48:41a world with entities that both some
48:43entities can be movable, some entities
48:45can be movable, some speak, some don't.
48:47But even if you have something like a
48:49room with a door, you have a character
48:52that hits the door, the door is open and
48:54All of those are sort
48:56of unstructured. It's not very
48:58deterministic. The player can take
49:01different ways and so on and so forth.
49:02And the actor model is perfect for that
49:04case. And if you think about it, the
49:06reason the actor model was even invented,
49:09as far as my knowledge goes, we go back
49:12to Erlang and we go back to the 70s when
49:14Ericsson had to model telecom networks
49:19where, again, you have very
49:22undeterministic and unrelated entities
49:24that communicate with each other.
49:27I can phone Louis, New York, I can phone
49:30Johannes in Germany, they can phone each
49:33other. You have race conditions when two
49:36people try to call the same
49:38It's very similar to what you
49:40would get in a game.
49:43And in my opinion, it's very similar to
49:45what an agentic system
49:47looks like to a very high degree.
49:49Have you thought about
49:50agentic system that's steerable?
49:53So when I was doing game design,
49:54we do a lot of steer algorithm.
49:56And steer algorithm is
49:57all about managing a crown
49:59Let's say you want to model
50:00a particle system, right?
50:02You have the particle
50:03system, it's actually,
50:04and each of them could
50:05be a bird, by the way.
50:05The particle system can be represented
50:08by a bunch of bird
50:09actors within this cap.
50:12It's kind of a point
50:13cloud of a bunch of birds.
50:14And now let's say you
50:15swarm this bird, right?
50:17You swarm this bird through
50:18an obstacle in the middle.
50:20Now they all have to
50:21independently dodge that ball
50:23and fly around that thing, right?
50:25Because the whole idea is when
50:26you steer this flock of bird,
50:27you just want to pick a pointer
50:29and you move that
50:30pointer to a certain location
50:32on the 3D map and then the flock of birds
50:34will just keep on following.
50:36They keep on
50:37following that point in space.
50:38They both try to steer
50:40away from each other,
50:41but they all kind of act in tandem to
50:44move to your pointer
50:46and then they dodge popsicle on the way.
50:49How do you think about steering agent
50:51or in that same vein?
50:55Have you thought about that?
50:56To be frank, I have not.
50:57I can apply some of my
50:59knowledge from mathematics
51:00in the past,
51:02especially studying how birds
51:04actually end up doing that.
51:06And it's all about local communication
51:09between one bird and the other.
51:12There's no overall concept.
51:14Basically the steering is almost like
51:16one tries to steer a little bit,
51:18the other follows the one
51:20close and so on and so forth.
51:22And I think even in that case,
51:23it's all about the connectivity
51:25between sorrounding entities
51:28the neighbor flock
51:30Each of the entity
51:30they should have knowledge of
51:32the entity right close to it.
51:33So you put a small little boating block.
51:36on each of them
51:37so more like small clusters of
51:38entities that sort of have
51:40deep communication between themselves,
51:42but almost not communicate
51:44in the far distance one.
51:47And also to quickly throw in here
51:49to answer your initial question,
51:51I've asked you to
51:51chat GPT and asked how,
51:54chat GPT how ECS
51:55compares to the actor model.
51:57And it says that they
51:59overlap conceptually.
52:00And the TLDR is that
52:02ECS is about what you have
52:05in terms of data and what operates on it,
52:08like which systems optimized for
52:11throughput and composition.
52:12And I think composition here
52:13is really interesting, right?
52:15In a game you compose like a big scene
52:18of like smaller things.
52:20And the actor is about who does what
52:23optimized for isolation concurrency.
52:25So I think we can basically
52:27build an overall system here
52:29with both and they're
52:30just like have sort of like
52:32different aspects, like
52:34what they're concerned about.
52:36Like composition is interesting.
52:39And the other one about concurrency,
52:40like ideally you have both,
52:43but both needs like this
52:45different kind of concerns.
52:47So I just wanted to weave that in
52:49as I think it fits really nicely
52:51the sort of like ball flock of birds
52:54kind of scene we've been describing.
52:57The kind of difference is more
52:59at the level of the API they expose
53:03rather than at the level of the runtime.
53:07Because they're still both actors.
53:10It's just that the API
53:11focus is more on one aspect
53:14or the other.
53:16For example, even in
53:17actor systems like Akka
53:20you have different ways
53:21of creating actors.
53:23And even in a fat cluster,
53:24which is our actor solution,
53:26you do have different
53:27ways of creating actors.
53:29You can focus on the behavior,
53:31which would be more along
53:32lines of data in data out.
53:36The kind of messages that the actor
53:38can respond to, can send to.
53:40Or you basically get a fiber with a cue
53:44that you pull on and
53:46everything you do is custom.
53:49So you have more
53:50flexibility, less structured API.
53:53I think like the idea of a
53:56distributed actor system
53:57is very, very essential to
54:02distributed systems in general.
54:04And my point is really that
54:07agentic behavior in
54:08applications is a distributed system.
54:11And it's just so happens that nowadays
54:13everybody builds AI applications.
54:15So they hit
54:17the distributed system problems day one.
54:19rather than facing them
54:21further down the path.
54:23And it's also why companies like Temporal
54:25is getting renewed traction.
54:27Because before you'd only use Temporal,
54:30if you actually have a lot of customers
54:31and what used to be an edge case,
54:34like I always make the example of an
54:37onboarding procedure
54:38where you put in
54:39username, you put in password,
54:40you put in email, the
54:42system has to record those
54:43in a database and it
54:44has to send an email.
54:45Like those operations
54:46happen within 100 milliseconds
54:48of each other.
54:49The probability of your server failing
54:51between the two operations near zero
54:54if you have queued two users that are
54:58registering in a day.
55:00But if you have 200 users that
55:02registering the same seconds,
55:03it's almost guaranteed that in a month
55:05you will get multiple
55:06instances of the issue
55:08of the server failing in
55:09between the two operations.
55:10So that's when you usually
55:12would reach for solutions
55:14like Temporal or like Effect
55:16Cluster.
55:17The difference with AI is that the
55:19average response time
55:20is much longer than used to be.
55:22Now we call an Open AI,
55:25API to OpenRouter and it goes through
55:28two different layers of proxying,
55:30but it has to stay
55:31alive for like 30 seconds,
55:32the same connection.
55:33The model can reason
55:34for more than 10 minutes.
55:35A lot of time, deep reasoning
55:37can go for a long, long time.
55:38And a lot of time during
55:40these deep reasoning steps,
55:41the model might not
55:42produce anything at all.
55:43So, OpenRouter, we actually,
55:45a lot of time we set
55:46to standing this,
55:48the comment, like it's
55:49actually in the SSE spec
55:50is that you have to send
55:51back comment to keep alive.
55:52The way they see the connection alive,
55:54otherwise either
55:54the client would not know
55:56if we're still alive or
55:57The DNS server, the DNS layer would
55:59actually cut us off.
56:00And even any proxy that is in between,
56:03because even when you call open AI,
56:04you're not just calling
56:05straight up to the server,
56:06you have layers and layers
56:08of content delivery networks
56:10And each would hit HTTP timeouts
56:13So like either you do web socket stuff
56:15where the ping is in the protocol itself,
56:18or you have to go on top and do a ping.
56:21But the point is here, the average length
56:25of an interruption with a service
56:27went from being
56:28milliseconds to being minutes,
56:31if not days when the
56:34other end-to-end reasoning
56:35could be that we have still a pattern
56:37where there's humans
56:38in the loop of agents.
56:40So sometimes an agent has to
56:42stop and call another agent,
56:44which is called a human that has access
56:46to human intelligence
56:47that is less artificial,
56:49but they work exactly in the same way.
56:50So humans are entities that share
56:53knowledge via messages.
56:54Currently we have three
56:55entities in this call,
56:57one entity is in New York,
56:58one entity is in Italy, the
57:00other entity is in Germany.
57:01Yeah, absolutely right. And we all communicate
57:03with, we're sharing messages.
57:04You know what's funny though?
57:05I just saw the
57:07premiere of the Tron movie.
57:08And in the Tron movie,
57:10so Tron has a bunch of program,
57:12which is all humanized.
57:14The program in Tron
57:14communicate with the human outside
57:16through MCP.
57:17It's a fun detail in that movie.
57:19I was like, "Ooh, cheeky."
57:21I wanna play back the
57:22same question to you.
57:24So you asked Mike how he thinks about
57:28how multiple AI systems could be steered
57:32through this sort of
57:34flock of birds, ball analogy.
57:36How are you thinking about this?
57:38Like just in terms of my own experiences
57:41about a steerable AI system versus a non
57:44steerable AI system,
57:46actually two coding
57:47agents I use on databases,
57:48Codex and Claude Code.
57:50Well, as of October 13th, 2025,
57:55maybe a new thing drops
57:56tomorrow and I'll switch.
57:58But as of today, I am using Claude Code,
58:02which can be steered and
58:03I make a lot of use of it.
58:04And therefore I'd use it
58:06differently than Codex,
58:08where it's much more like binary,
58:10where I feel like if it's off track,
58:13I like interrupt it and
58:15say, "No, no, no, no, no,
58:16please do this."
58:17And I guess it is sort of
58:19like similarly steering it,
58:21but it feels much more like
58:24stop and go and more forceful.
58:27So I'm curious like
58:29for you to extrapolate
58:30where this is going.
58:31And we're talking
58:32about multiple AI systems.
58:33So how are you thinking about this?
58:35Yeah, one of my main pain points,
58:36we're using Claude code or even like,
58:38you know, Cursor or so on
58:40Is that you set a goal or
58:41you say, "Hey, fix this thing."
58:42The problem is, right?
58:43That's the goal already.
58:44The flock could just find the goal.
58:46I shouldn't have to like say,
58:47"Oh, there's an
58:49obstacle, there's an obstacle."
58:50The obstacle is in
58:50the environment itself.
58:51go back to my college day
58:52when I was listening to this system,
58:54an autonomous steering system,
58:56self-driving is the same idea, right?
58:58You need sensor.
58:58You essentially need a
59:00way to emitting a bunch of
59:01either event, which if we're using ROS,
59:04it will be very simple,
59:05ROS for barring over system.
59:07It emits an event system,
59:08when you're just sending into the system
59:10a bunch of different light sensor
59:12to tell you about the environment.
59:14So first I think
59:15every actor in the system
59:16kind of have to emitting
59:17some kind of sensory system
59:19that tell the whole system,
59:20"Hey, here's the
59:21state of the environment."
59:22And secondly, I think
59:24that through those sensor,
59:25you pick up some kind of weight.
59:28allowing the system to
59:30like course correct.
59:31Automatically, instead of
59:32having the end user have to like,
59:34"Okay, we hit a roadblock now,
59:36the type is not checking here.
59:38What do you remove,
59:39let the satisfy?"
59:40So humans still have to right
59:42now steer somewhat manually.
59:44I would love for the end goal system is
59:46I put a pointer in
59:47the space in the world,
59:48meaning a goal, right?
59:49To say, "Hey, fix this unit test."
59:51The swarm of bot
59:52should just be able to say,
59:53"Okay, I'm going to
59:55use the browser agent,
59:56which is like top tier browser agent.
59:58Let's read the documentation
59:59for the test, for example."
01:00:01And then, "Okay, I know that you guys,
01:00:03I'm using the test with the set timeout,
01:00:05the set timeout set too low."
01:00:07That's one thing that
01:00:08it might see in the test,
01:00:10or in the output.
01:00:11And then from the dock of
01:00:13the Vitest agent, right?
01:00:15It will send back to my agent,
01:00:16and my agent will then be like,
01:00:18"Come on, okay, the set timeout too low.
01:00:20So let's change that."
01:00:21Just keep on doing that
01:00:23loop over and over again
01:00:24with our other agent,
01:00:25or it figure out like what sensor it
01:00:28needs from the world,
01:00:30from the system to
01:00:31steer its action as well.
01:00:34And I think so far,
01:00:35I mean, Claude Code is
01:00:37occasionally doing recently well,
01:00:38I would say.
01:00:39It has a browser inside,
01:00:40and like the chain of thought
01:00:42that it has to like
01:00:43solving a certain problem,
01:00:44seems pretty decent so far.
01:00:46here's like what I'm
01:00:47seeing of the whole system,
01:00:48like here's the data of the system
01:00:50that can help me steer myself.
01:00:53And I think what's
01:00:54also interesting is like
01:00:55that the entire ceremony is
01:00:58like all sequential, right?
01:01:01And so that therefore it makes it slower.
01:01:04And so I think where
01:01:06I'd like to see this go,
01:01:07and I'm sure it will
01:01:08go in this direction,
01:01:09is that it's more of like an orchestra
01:01:11performing a symphony.
01:01:13And right now it's just
01:01:14like solo, solo, solo, solo.
01:01:16And while you're
01:01:18encountering a problem,
01:01:19like that might be a signal,
01:01:20I was like, you know what, like,
01:01:22let me go off like a little
01:01:24bird just goes to the side
01:01:26and like studies the docs more deeply
01:01:28about like a system.
01:01:30So it's like, hey, I
01:01:30found something interesting,
01:01:31like we're on the wrong track.
01:01:33And like you could imagine multiple
01:01:35of those like things happening.
01:01:38And right now it's like
01:01:39everything's just like happening
01:01:40sequentially.
01:01:42And yeah, I think this is very exciting
01:01:44to see how this is unfolding.
01:01:47And I like your guys steering analogy
01:01:50and to look at it through like the ECS
01:01:52and an extra system lens.
01:01:54I have one observation
01:01:56that I recently learned myself.
01:01:59to take a step back,
01:02:01I've been very late to the
01:02:02AI coding game in general.
01:02:05Johannes has pushed me for a while.
01:02:08I developed the ability to ignore
01:02:11Johannes for enough time,
01:02:12but at some point sort of
01:02:14penetrates the protection
01:02:16when you're more
01:02:17vulnerable and you're like,
01:02:18okay, then I have to try.
01:02:20And since Claude Sonnet I
01:02:23think 4 or 3.5 came out,
01:02:26I've been heavily invested in AI coding
01:02:29and I really wanna make it work.
01:02:32And what I've realized is
01:02:33that I think we're putting tools
01:02:37either on the wrong place
01:02:39or we're not putting
01:02:41tools in enough places
01:02:43in the sense that
01:02:44currently we rely on the models
01:02:47to maintain some sort of loop.
01:02:49But the models are
01:02:50terrible to do looping.
01:02:52They can't even properly assign tasks
01:02:55They are exceptionally good
01:02:57when called with a specific
01:02:58prompt and a specific task,
01:03:00then they're done.
01:03:02The task might not be
01:03:03done, but the LLM is done.
01:03:04So what I found extremely
01:03:07helpful in approaching tasks,
01:03:09this is not really like AI coding
01:03:11in the sense of the AI
01:03:13assisting you while you code.
01:03:14It's rather a task like, okay,
01:03:17I give you the Effect repository,
01:03:19create JS docs for every
01:03:21single function in the repository.
01:03:24Go, use 1000 parallel
01:03:26agents to achieve the task.
01:03:28I don't care.
01:03:28Ready to spend 10, 20K
01:03:30to do it, just do it.
01:03:32we are approaching
01:03:33the point where I think
01:03:35there are ways to make it work
01:03:37and even with normal like Claude plans
01:03:40with $200 per month,
01:03:42I'm completely fine
01:03:43running multiple agents
01:03:44and so on and so forth.
01:03:45But what I've realized works best
01:03:48sort of goes in the
01:03:49direction of steering.
01:03:50Instead of letting, for
01:03:52example, the model calls tests,
01:03:54calls TypeScript to get type
01:03:57errors and so on and so forth,
01:03:59instead of having the model do that
01:04:01and assigning the model the
01:04:03responsibility to do that,
01:04:05I have those in a deterministic loop.
01:04:07And if I have a type
01:04:08error, I call the model again,
01:04:10I have a system which is LLM based
01:04:13that scores a specific example.
01:04:16And if that is not happy, I
01:04:17call the first model again,
01:04:19saying, look, my Oracle is
01:04:22not happy for those reasons.
01:04:23I call it a deterministic outside loop
01:04:26that ends up providing those events
01:04:29and this feedback
01:04:30inside the single agents
01:04:32that are steering them
01:04:34towards the right goal.
01:04:36And that works exceptionally well.
01:04:38Like well to the point
01:04:39where you can leave it going
01:04:41for a day, come back, you
01:04:43have 95% of the stuff done.
01:04:46And now that multiple
01:04:48times, because I wanna improve,
01:04:50I'm testing these on Effect 4,
01:04:52because I don't wanna
01:04:53break everything in Effect 3.
01:04:54And to the point where the
01:04:56quality is going up and up
01:04:58and the automation is
01:04:59also going up and up.
01:05:00And I'm almost there
01:05:01where it's almost better
01:05:03than me writing example manually,
01:05:05if I let it go long enough.
01:05:07I have never seen that
01:05:09with a normal coding agent
01:05:11where you don't have this external loop
01:05:13that steers it in the right direction.
01:05:16So I think I didn't
01:05:18understand the answer at first,
01:05:20but actually I'm doing stiffing myself
01:05:22now that I understand it better.
01:05:25So I think this sort of setup
01:05:27and system you've described
01:05:29is becoming more and more mainstream.
01:05:32And I think there's
01:05:33multiple ways to achieve it.
01:05:35Either it's already the de facto behavior
01:05:38of like some coding agents,
01:05:40or also this is where
01:05:42those like coding agent SDKs
01:05:44come into play.
01:05:45Like I think Claude Code was
01:05:47probably one of the earliest
01:05:48ones that provided that.
01:05:49Now, Codex has that AMP,
01:05:51I think also just release theirs,
01:05:53where you can like build
01:05:55your own coding agent
01:05:57and like express the sort of like system
01:05:59and workflow that you want.
01:06:00Converting a big repository
01:06:03from having not no JS
01:06:04doc to having JS doc,
01:06:06which can be massively run in parallel.
01:06:10That's a great candidate,
01:06:11but doing that is very
01:06:13different from building
01:06:15whatever like a single new module
01:06:18that you maybe have no
01:06:20idea what the exact goal is.
01:06:22And it's like, it's
01:06:22much more exploratory.
01:06:24Yeah, there's much more human in the loop
01:06:25in the other direction.
01:06:28Exactly.
01:06:28So yeah, Louis, I'm curious
01:06:30like how much you're already
01:06:32using those sort of like patterns as
01:06:34you're building systems
01:06:36and maybe your thinking is more refined
01:06:39in advance on this.
01:06:40Not that refined yet, by the way.
01:06:41We use a lot of AI model, AI agent.
01:06:44So we use the Devon a
01:06:45lot, you know workload.
01:06:47And I think right now
01:06:48the best way to steer Devon
01:06:50is to have a lot of knowledge.
01:06:52The knowledge truly helps steering it.
01:06:55And Devon is selectively like it's select
01:06:57the right knowledge per task as well.
01:06:59So that actually easily have a lot.
01:07:00I mean, this apply to
01:07:02also, you know what,
01:07:03I think Cursor and CLI
01:07:05and all the other coding agent as well.
01:07:07The more you can add into the memory,
01:07:09the better they adapt to your code base.
01:07:12in terms of having an orchestration layer
01:07:14to like either use
01:07:16multiple of them at the same time
01:07:17or like picking the right
01:07:18one for the right job though.
01:07:20We still haven't like
01:07:21think too much about it yet.
01:07:22Yeah, right now it's all
01:07:24just my workflow.
01:07:26Maybe a related but
01:07:27different question about this.
01:07:29Given how Mike has described his
01:07:32coming to terms with this new future
01:07:35and he at his own pace
01:07:37started like discovering
01:07:38and like exploring, experimenting,
01:07:40see what works well for him.
01:07:42And I think that
01:07:43fundamentally changed Michael's workflow.
01:07:46I had the same process
01:07:47and I think everyone does.
01:07:49But I think what is
01:07:51maybe interesting to explore
01:07:54is how you as OpenRouter as a company
01:07:58try to create like a higher
01:08:01level process around that.
01:08:02So for example
01:08:03maybe a stupid way to go
01:08:04about is say, hey, it's
01:08:05the first of the month.
01:08:06And we're again spending
01:08:08three days of like going
01:08:11from scratch,
01:08:12exploring the best practices,
01:08:13what it means this month
01:08:15to do agentic coding.
01:08:17Or maybe you have
01:08:17like a team that is known
01:08:20for being super cutting edge
01:08:22in every first of the month.
01:08:24They do like a presentation of sharing
01:08:26their best practices.
01:08:27Or you basically say,
01:08:28we're not trying to orchestrate anything.
01:08:31We just let emergence happen.
01:08:33So I'm curious how much
01:08:35hands on or hands off you are
01:08:37of like fostering, like
01:08:39embracing the productivity
01:08:41and efficiency benefits
01:08:42you get from agentic coding.
01:08:43I would definitely
01:08:44encouraging using agents
01:08:46in almost every aspect of the code base
01:08:48and also in the business as well.
01:08:50I think the encouragement
01:08:53is like the key part, right?
01:08:54asking our team
01:08:55to use it as much as possible to see,
01:08:58essentially the idea is to
01:08:59for them to learn the best way
01:09:00to use it, right, to leverage these tool.
01:09:01But the one thing I found though,
01:09:03oh, okay, with everything,
01:09:04right, there's a drawback.
01:09:05The drawback is reviewing the code.
01:09:06The key sin is when someone, you know,
01:09:09they rely too much on
01:09:10the bot and they say,
01:09:11hey, do this and then they
01:09:12just send it off for someone
01:09:13else to review without
01:09:14them reviewing themselves.
01:09:16That's like a huge sin
01:09:18that one day I will eradicate
01:09:21it somehow
01:09:22we have a way for
01:09:23you to eradicate that scene.
01:09:24Oh, awesome, I love it.
01:09:26It's called Effect
01:09:29Well, I'm telling you, so occasionally
01:09:31the bot is very smart.
01:09:32The bot, the agent is
01:09:34smart actually, I don't know,
01:09:37like a lot of time it
01:09:39will actually change
01:09:40either the TS config or it
01:09:44will actually add an ignore
01:09:45to the test setup or something like that.
01:09:48Or sometimes it mark the whole module.
01:09:50So the test looked like it
01:09:51actually testing something
01:09:52but it looked, you
01:09:53read deeper into the test,
01:09:54it literally marked the
01:09:55module you're supposed to test.
01:09:56Holy shit, it did.
01:09:58So even, let's say, even
01:09:59if we use Effect, right?
01:10:01If the bot mark Effect, what can we do?
01:10:04Just to take this
01:10:05slightly more dystopian,
01:10:08like maybe if you want to turn this
01:10:10into a Black Mirror
01:10:11episode, you could imagine
01:10:12sort of like the AI
01:10:14modeling like a mock human
01:10:16where like the
01:10:17instructions always you need
01:10:18to get permission from the human.
01:10:20So it models, it's mock
01:10:21human that always says,
01:10:23yes, go for it.
01:10:24I have
01:10:25experienced those problems.
01:10:26Of course, everybody doing
01:10:28AI coding has sometimes the,
01:10:31yeah, I made the test fast.
01:10:32You check and you added a deed dot skip.
01:10:35What are you doing?
01:10:36But I feel like most of the time
01:10:39when the AI agent is doing that,
01:10:42it's because it
01:10:44really has no understanding
01:10:46of the structure of the code.
01:10:48And in my experience that often comes
01:10:52from the agent
01:10:53working toward a low level.
01:10:55So it almost focuses too much on the
01:10:58implementation details
01:10:59versus what it has to do.
01:11:02And that's why for higher level,
01:11:05for example, application development
01:11:07as opposed to library development,
01:11:10like the AI is doing very poorly
01:11:13to implement the internals of Effect.
01:11:15It's doing exceptionally well to
01:11:17implement the tests,
01:11:18the examples
01:11:19on the effect repository.
01:11:21The reason is that
01:11:24Effect almost produces code
01:11:26that is closer to the human
01:11:30versus being closer to the machine.
01:11:32And LLM's being trained
01:11:35on tons of human language.
01:11:37They can sort of
01:11:38reason better in human terms
01:11:42versus that in code terms.
01:11:44Like if you ask
01:11:44Chat GPT a human type question,
01:11:48very rarely it ditches the question,
01:11:51there's not TS ignore at the human level
01:11:53in a sense because the
01:11:56language is higher level,
01:11:57it's richer and expressing concepts
01:12:01in a richer language is much easier
01:12:04than boiling down to the very
01:12:06details of the implementation.
01:12:08And that's where I see the key shift,
01:12:11the key difference when I build
01:12:13application level code
01:12:14and it uses Effect,
01:12:16almost never I have a problem
01:12:18like the AI ditched the task.
01:12:21It's perfect to create schemas.
01:12:22You have like, okay, I had these API
01:12:25returned the structure like this one,
01:12:27please create a service for me
01:12:29that interacts with this API.
01:12:32And it's fits down the full service
01:12:35which almost always is,
01:12:37I don't like the term one-shotted
01:12:39because we're really not
01:12:40here to one-shot things,
01:12:42but close to
01:12:44one-shoting the correct solution.
01:12:47And I've seen that happen over a lot of
01:12:49different code bases.
01:12:51And it's where I think like
01:12:53Effect can play a big role,
01:12:54especially linking to
01:12:56one of your concerns
01:12:57slash questions that you raised
01:13:00before we even started recording.
01:13:02And I wish you will
01:13:04use a different solution
01:13:06that we are building
01:13:07and where you will not
01:13:09be memory constraints.
01:13:11You will have spans
01:13:12collected out of the box
01:13:13and so on and so forth.
01:13:15That'll be very interesting.
01:13:16And then, well, I mean,
01:13:17and you guys can
01:13:18handle like caching, right?
01:13:19So caching is another layer that's like,
01:13:21caching at the DNS
01:13:22layer would be interesting.
01:13:24also no egress cost
01:13:25if you can figure that out.
01:13:27That part is hard.
01:13:28Yeah, no, no, that is figured.
01:13:30We don't have any egress cost
01:13:32because we're building on bare metal.
01:13:33So we don't pay for bandwidth.
01:13:36There's no reason why to
01:13:37charge people for bandwidth.
01:13:39There's no reason why
01:13:40to overcharge people
01:13:41for CPU prices or go
01:13:445X what AWS gives you.
01:13:46The only hard part, to be honest,
01:13:48that we had to deal with is networking.
01:13:52But networking in the
01:13:53sense of private networking.
01:13:55We got it figured,
01:13:57kind of EPC layer stuff.
01:13:59I had to write tons of
01:14:01eBPF code in the kernel
01:14:03to be able to figure that out properly.
01:14:06But I think we're getting there.
01:14:07And I hope that we're
01:14:10gonna be able to provide you
01:14:12with the right
01:14:13primitives to deploy a platform
01:14:16such as OpenRouter, which
01:14:17is really what we're trying
01:14:18to solve with Effectful.
01:14:21Because with Effect, I think we solve
01:14:23the production grade coding.
01:14:25But at some point, code ends,
01:14:28and you have to
01:14:30deploy your code somewhere.
01:14:32And especially with AI,
01:14:34I feel we are hitting.
01:14:36The real troubles were
01:14:38if you were deploying
01:14:38distributed systems in the past.
01:14:40But you would only do
01:14:41that if you were a scale up
01:14:44handling millions and
01:14:45millions of customers
01:14:46Otherwise, a small
01:14:48server, just a database,
01:14:50would get you out of the troubles.
01:14:52And the edge cases would not materialize.
01:14:54Because the average request time was,
01:14:56like what, 100, 200 milliseconds.
01:14:59Now with AI and flows like
01:15:01the ones you have in OpenRouter,
01:15:03my guess is that
01:15:04sometimes you have API calls
01:15:06that takes minutes.
01:15:07with minutes of
01:15:09time, be sure that something
01:15:11can happen much more
01:15:12frequently than it can happen
01:15:14in 100 milliseconds.
01:15:15And so long-running processes
01:15:18and less server-less pieces,
01:15:22I think will be much more important.
01:15:25And let me ask you a question.
01:15:28How important is latency in a request
01:15:31that takes five minutes to answer?
01:15:32Not a lot, right?
01:15:33Not a lot.
01:15:35Maybe the handshake so that the client
01:15:37is not easily, some
01:15:39of them are very eager
01:15:40to kill the request, but otherwise, yeah.
01:15:43People don't really care.
01:15:44Sure, but you might be,
01:15:46like the important part
01:15:47what might be to back off from a request
01:15:50much faster to not spend money
01:15:52versus actually letting the process run.
01:15:55So I think for these kind of use cases,
01:15:58and I've been discussing
01:15:59this with some other people,
01:16:01it's much better to have
01:16:03reliable infrastructure,
01:16:04maybe deployed
01:16:05slightly outside of the edge,
01:16:07so it's not close to your home,
01:16:09but it's cheap, doesn't
01:16:11have network costs whatsoever,
01:16:14has very cheap compute
01:16:15costs, very cheap storage costs,
01:16:18and you can actually run a server
01:16:21not being constrained by memory
01:16:23because I feel like
01:16:24being able to collect traces
01:16:26is much more important for the business
01:16:29than saving, what, 20 megabytes of RAM?
01:16:33like RAM is cheap, or 20
01:16:35milliseconds of request time?
01:16:37or maybe like
01:16:38100 milliseconds, yeah,
01:16:39which is doesn't really do much.
01:16:41By the way, the thing
01:16:42that you might think
01:16:42is just internal tooling,
01:16:43but we have a lot of
01:16:44internal tooling, just so you know.
01:16:46So, which is why Effect is
01:16:48actually being used a lot.
01:16:49because we are
01:16:50TypeScript everything, right?
01:16:51So, instead of running a Python eval
01:16:53suite on the market,
01:16:55there's a bunch, we write our own, but it's
01:16:57using Effect, right?
01:16:58So, we actually have an eval suite
01:17:00and also a bulk test suite,
01:17:02that's all using Effect with Effect retry
01:17:05and so on and so forth.
01:17:06Yeah, so it's a very
01:17:07sophisticated system.
01:17:08I'm just saying is that if we use,
01:17:10you might think internal
01:17:11tooling is like some toy,
01:17:12but no, we're actually using
01:17:13it almost on a daily basis,
01:17:14it's actually production load.
01:17:16Reason why we don't use it in the core,
01:17:17because while the core is working,
01:17:19don't touch it.
01:17:20But we do have a lot of internal tooling
01:17:22to do testing out all
01:17:23the endpoint, right?
01:17:24Which is a crucial part of our operation.
01:17:27We have, well, I
01:17:28mean, almost every respond
01:17:29to transformation is using the pipe now.
01:17:31And then also I'm
01:17:32introducing more and more generator.
01:17:35So, down the line, we finally will,
01:17:38it will be making it
01:17:38easier for us to transitioning
01:17:40over to Effect as needed.
01:17:41by the way, to the point,
01:17:42I'm actually still using async generator.
01:17:43And the main reason why
01:17:44I'm using async generator,
01:17:45because inside this generator,
01:17:47we have some async
01:17:48process, which unavoidable.
01:17:50No, no, my prior
01:17:52comment on you should not use
01:17:53async generators is
01:17:55because you should not have
01:17:56two representations of code.
01:17:58But if you're using
01:17:59promises with result types,
01:18:01100%, please use async generators
01:18:04so that your developers
01:18:06don't have to unwrap
01:18:07every error manually at every point.
01:18:10But it's still a much
01:18:11better user experience.
01:18:13I just meant that if you
01:18:15have one single representation
01:18:17of code, you don't need
01:18:19to use different types
01:18:21of generators
01:18:22Plus, adding like one
01:18:24of the key troubles I had
01:18:26in the past was, okay, I
01:18:28have a very synchronous
01:18:30code transformation, like these takes
01:18:31an API, does this
01:18:33this, this, this, that, okay.
01:18:34Now I wanna emit a Kafka event.
01:18:36Not now, not at the
01:18:37beginning, but a month later,
01:18:39I would write an event in Kafka.
01:18:41Look at what's the writing to Kafka
01:18:44is an asynchronous operation.
01:18:45So I stitch it at the
01:18:46end, and I have to bubble up.
01:18:48Everything becomes async.
01:18:50That's a problem.
01:18:52Now it's still a much
01:18:53better problem to have
01:18:55versus writing callbacks.
01:18:57Because callbacks don't
01:18:58have function coloring.
01:19:00Callbacks are fine for sync and async.
01:19:02They are a unified
01:19:03representation of computations.
01:19:06It's continuation passing
01:19:08style that we ended up calling
01:19:10callback passing style in JavaScript
01:19:13because we just like to
01:19:14invent new terminology
01:19:16so that when somebody looks at history,
01:19:18they don't see the
01:19:19theoretical papers from the 70s.
01:19:21Otherwise they understand
01:19:22we're just stealing ideas
01:19:24from the 70s and making
01:19:26them production ready.
01:19:28But there was a unified
01:19:30representation of code,
01:19:31which leads to, boom,
01:19:34boom, boom, boom, boom, of callbacks.
01:19:36Now with promises and async
01:19:37await, you no longer have that.
01:19:39But you do have the two different
01:19:40representations of code.
01:19:42And that's much better
01:19:44than reading the code
01:19:45like you described before.
01:19:46You want to review on the phone.
01:19:48There's no way you're going
01:19:49to review 100 nested callbacks
01:19:52with your phone screen.
01:19:54By the way, no way
01:19:55you're going to read that
01:19:56in a normal computer screen.
01:19:58You might need Johanness' set up
01:20:01with exceptionally large
01:20:03screens and so on and so forth.
01:20:04But even there, I
01:20:06think you should optimize
01:20:09for a different problem.
01:20:10So I think the final solution, which
01:20:13is I believe what we have with Effect
01:20:15is a unified representation that allows
01:20:18for sequential representation.
01:20:20But still, the intermediate point
01:20:23of having two different ways of doing
01:20:25things that all look
01:20:26good is perfectly viable.
01:20:29And especially in your
01:20:30code base where you're
01:20:31using a custom result
01:20:32type, all the time go for that.
01:20:35It's amazing.
01:20:36Yeah, this is the thing
01:20:37that I was looking for,
01:20:38which is the way to represent in both
01:20:40async and synchronous
01:20:41process
01:20:42in the same container, essentially.
01:20:44that's something I'm excited
01:20:45about most, about Effect, right?
01:20:46It blurred the line between async and sync.
01:20:48And you can now
01:20:49orchestrate them in the same flow.
01:20:51So it seems like you're well on your way
01:20:54to introduce Effect more and more
01:20:56throughout the entire system.
01:20:58And it's very interesting
01:20:59to hear how you've basically
01:21:01introduced mini versions and
01:21:03little pieces of Effect
01:21:04already to make it
01:21:06more and more swallowable
01:21:08by the rest of the engineering culture.
01:21:11So I feel there's plenty of stuff
01:21:14here that we need to unpack
01:21:17probably in another future
01:21:19episode to hear how your
01:21:21Effect journey is continuing.
01:21:23Before we're wrapping up, though,
01:21:25we talked before about that as part of
01:21:28your onboarding culture
01:21:30to ask a new employee
01:21:32what's the latest rabbit hole
01:21:34they went down on.
01:21:35So what's been your latest rabbit hole
01:21:37that you barely emerged again from?
01:21:41The most recent one-- I recently moved to
01:21:43New York, by the way,
01:21:44two weeks ago.
01:21:45Sorry, two weeks ago was two weeks ago.
01:21:47So that was a month ago.
01:21:49Oh, my god.
01:21:49And that led me to hold rabbit hole
01:21:51buying stuff for my
01:21:52apartment, essentially.
01:21:53Well, there are two rabbit holes.
01:21:54One is on buying a bed,
01:21:56and one is on apartment
01:21:58finding in New York.
01:21:59Luckily, I didn't--
01:22:00I was on a time crunch to find an
01:22:04apartment in New York.
01:22:06So that
01:22:07hole was a bit shallow.
01:22:09But the one about finding a bed, though,
01:22:12a good bed for your
01:22:13back, is it was very deep.
01:22:15And thankfully-- so I'm
01:22:17working off the Marlin.
01:22:19It is a very bougie cooking
01:22:20space down in Flat Island,
01:22:22Manhattan.
01:22:22And right next to this place, there are
01:22:26three mattress stores.
01:22:28Like right next to me is the Sattva,
01:22:30which is one of these
01:22:31very high-quality hotel--
01:22:32mattress.
01:22:34Down the block, there
01:22:35is a mattress firm store,
01:22:36which contains a bunch of
01:22:37variety of mattress store.
01:22:39And then on the other block is Avocado,
01:22:43which is this another
01:22:45very bougie, very expensive
01:22:47mattress.
01:22:48And I got to go in and try
01:22:50all of them, spend about a day.
01:22:52And it is fascinating.
01:22:55The final choice I made
01:22:56was to get a purple mattress.
01:22:58This is not a purple
01:22:59mattress app, though.
01:22:59I just tried all of them.
01:23:00So the best-- I'm
01:23:01going to say the best thing
01:23:02that you can do when buying a mattress,
01:23:03you just try them out
01:23:04yourself, tug it in, go on all side.
01:23:08You've got to try all side
01:23:09of your sleeping position.
01:23:10Yeah, I roll it all over
01:23:13and try to tug it in to see
01:23:14if your spine can get enough support.
01:23:17And of all of them so
01:23:19far, for me personally,
01:23:20the purple mattress has been the one
01:23:22that I can actually support my spine.
01:23:27The second option was the
01:23:28Sattva, the Sattva Classic,
01:23:30which is somewhat soft, but
01:23:33also decently firm in the bottom.
01:23:35And yeah, the purple
01:23:38mattress is kind of interesting.
01:23:39It's not one of those
01:23:43put traditional bad.
01:23:45He's going to send
01:23:46over the affiliate links,
01:23:47and we're going to put
01:23:48them in the description.
01:23:49It's going to be a podcast first.
01:23:50There was no affiliate, but
01:23:51thanks to a very good friend
01:23:52of mine. He was my co-founder of
01:23:54my previous company, too.
01:23:55He went mattress
01:23:56shopping together with me
01:23:57And then we did the good
01:23:59cop, bad cop negotiation
01:24:00at the mattress firm.
01:24:01And we get $1,000
01:24:03discount on that mattress,
01:24:04which is kind of crazy.
01:24:06$1,000?
01:24:06$1,000.
01:24:08$1,000. A $1,000 discount on a mattress?
01:24:10Yes, on a mattress.
01:24:11That kind of suggests
01:24:12that the mattress was
01:24:13more than $1,000.
01:24:14Yeah, I guess.
01:24:16Yes, the mattress was very--
01:24:17I mean, I talked to a friend of mine,
01:24:19and he told me, you
01:24:21spend a lot on your back.
01:24:23You're going to need it.
01:24:24So yeah, I think it was good spending.
01:24:27Yeah, very comparable bet, for sure.
01:24:29Unrolling the bed,
01:24:30though, was insane, though.
01:24:31It's very hard.
01:24:32So recommend, if you're buying a bed,
01:24:35just have two people.
01:24:37Just have a friend.
01:24:38Fair enough.
01:24:38Otherwise, you'll ruin your back while
01:24:40setting up the mattress.
01:24:43That's no good.
01:24:45Awesome.
01:24:46Hey, Louis, this was an absolute treat
01:24:48for us having you on.
01:24:48for us having you on. I'm looking forward to
01:24:51having you on in the future again
01:24:53and hearing more about how
01:24:55OpenRouter is adopting Effect
01:24:58and all the crazy
01:24:59growth you're going through.
01:25:01Also, a big thank you
01:25:02for Mike for spoiling us
01:25:05with his presence
01:25:06throughout this episode, as well.
01:25:08Very, very interesting conversations.
01:25:10I've learned a lot.
01:25:11So thanks a lot to both of you.
01:25:15Awesome.
01:25:16Thanks, Mike, for
01:25:19putting me in on the podcast.
01:25:23It was a big pleasure.
01:25:25And I hope to see you soon
01:25:28when we come to New York.
01:25:30Absolutely.
01:25:31I'll see you then.
01:25:31All righty.
01:25:32Have fun.
01:25:32Take care.
01:25:33Thank you for listening to the
01:25:35Cause & Effect Podcast.
01:25:36If you've enjoyed this episode, please
01:25:38subscribe, leave a review
01:25:40and share it with your friends.
01:25:42If you haven't done so already, you can
01:25:43join our Discord community.
01:25:45And if you have any questions, feedback
01:25:47or suggestions about this episode or
01:25:49about Effect in general,
01:25:51don't hesitate to get in touch.
01:25:53See you in the next episode.