Scaling AI for Customer Support at Markprompt with Effect
#2: Scaling AI for Customer Support at Markprompt with Effect
Join us as we talk with Michael Fester from Markprompt about scaling AI-powered customer support with Effect, building reliable and high-performance infrastructure, and enhancing developer productivity in a fast-evolving AI landscape.
Join us as we talk with Michael Fester from Markprompt about scaling AI-powered customer support with Effect, building reliable and high-performance infrastructure, and enhancing developer productivity in a fast-evolving AI landscape.
Effect is an ecosystem of tools to build production-grade software in TypeScript.
Song: Dosi & Aisake - Cruising [NCS Release]
Music provided by NoCopyrightSounds
Free Download/Stream: http://ncs.io/Cruising
Watch: http://ncs.lnk.to/CruisingAT/youtube
- (00:00) - Welcome & guest introduction
- (01:54) - Michael’s journey: from academia to AI & Customer Support
- (03:49) - What is Markprompt? Overview & use cases
- (07:45) - Markprompt’s system architecture
- (10:22) - Challenges of running AI-powered support systems
- (13:20) - Improving reliability with Effect
- (16:41) - Technical architecture breakdown
- (19:51) - The public API server setup
- (23:50) - Ingestion engine
- (26:29) - Onboarding engineers to Effect
- (30:51) - Migrating the codebase to Effect
- (35:19) - Effect in production: the power of schema
- (39:02) - Migrating to Effect: challenges & key takeaways
- (41:45) - Effect brings out the best in us engineers
- (45:34) - The Future of AI infrastructure
- (50:18) - Closing remarks & thanks
Transcript
00:00As an engineer, you want to build
00:01features, you want to
00:01build like real robust systems.
00:04You want to spend ideally
00:05all your time doing that.
00:06You don't want to spend your time
00:08fighting the language,
00:09building something that, you
00:10know, the language was not meant for.
00:12To give you an example,
00:13like concurrency in TypeScript.
00:14You want a way to build a concurrent
00:17system where you're
00:18thinking about what is
00:19the system actually doing rather than
00:20like setting up all the
00:22boilerplate in order to
00:23have that system in place.
00:25So I would say Effect really
00:27gives you the right premise.
00:28So now that we've completed this
00:30migration, kind of feel
00:31like this is the end game feels
00:33like more stable and at least if we
00:35continue building our
00:36code base on TypeScript with
00:38Effect, it doesn't feel like we're going
00:39to spend many more like
00:41cycles, like refactoring
00:43the code.
00:47Welcome to Cause & Effect, a podcast
00:49about the TypeScript
00:50library and ecosystem called
00:51Effect, helping engineers to build
00:54production-ready software.
00:56I'm your host, Johannes Schickling, and I've
00:58been building with
00:58Effect for over four years.
01:00With this podcast, I want to help others
01:02understand the powers and
01:04benefits of using Effect.
01:06In this episode, I'm talking to Michael
01:08Fester, co-founder of
01:10Markprompt, an AI-based customer
01:12support platform.
01:13In this conversation, we explore how
01:16Markprompt uses Effect
01:17across its entire architecture,
01:20including API and RPC endpoints,
01:22long-running workflows, and their
01:24customer-facing product.
01:25Let's get into it.
01:28Hey, welcome Michael.
01:29So nice to have you on the
01:30Cause & Effect Podcast.
01:31How are you doing?
01:32I'm good.
01:33Thank you.
01:33I'm really happy to be here.
01:35It's been, I guess it's been a while that
01:36we're trying to
01:37organize this, but I'm very
01:39happy that we're doing it now.
01:41Yeah.
01:41I mean, you're super, super busy.
01:44You're based in San
01:45Francisco building an AI company.
01:48So I'm really, really happy that you took
01:50the time to share
01:51your story about Effect.
01:53Maybe you can share a little bit about
01:56yourself, your
01:56background and what are you doing?
01:59Yeah, sure.
02:00So I have a background.
02:02I started in academia.
02:03Actually, I was doing
02:04research in mathematics.
02:05Then I started getting into AI and I
02:07built my first AI
02:08company in Europe, specialized
02:10on device voice recognition.
02:12And we got acquired by Sonos.
02:14And since then I moved here to San
02:17Francisco for my next
02:19venture, which is we're building
02:22AI infrastructure for customer support.
02:25So this is all like posts,
02:27chat GPT, GPT 4 coming out.
02:30And we saw this, this incredible
02:31opportunity to start
02:32working in this space.
02:35That is awesome.
02:36And I highly encourage everyone who is
02:40like interested in AI,
02:41particularly is looking
02:43for a better customer support solution to
02:46check out Markprompt.
02:48Can you share about a few companies who
02:50are already using Markprompt?
02:51Yeah, sure.
02:52So we've been working very closely with
02:55some companies very
02:56early on in the process.
02:57Actually, that's really interesting.
02:58Vercel started using us as one of our
03:00first customers,
03:01actually ElevenLabs, Replicate.
03:03Things are just moving so fast.
03:05So our approach here is not so much of a,
03:08you know, we have a
03:09vision of how things will
03:10end up, right?
03:11It's very much like a learning process,
03:13working very, very
03:14closely with customers, hand in
03:16hand getting their feedback,
03:17understanding together, how
03:19this thing is evolving, how
03:21we can improve, you know, the behavior of
03:23the LLMs or how we can
03:24extend the use cases.
03:26So this has been a very different way, I
03:28would say, of building
03:29a company from the past,
03:30because it's so new to
03:32everyone in the space.
03:33And we're just like scratching the
03:35surface of the
03:35opportunities of building these systems
03:38for you know, down the line.
03:40That's awesome.
03:41So before we go into the technical parts,
03:44maybe you can share
03:45through the lens of one
03:47of your typical customers, what
03:49Markpompt does and which
03:51problems it solves for some
03:53of those customers.
03:54Sure.
03:55So there's multiple facets of this
03:57running a support
03:58operation is it's not a one size
04:01fits all.
04:02Every company has their own ways of, you
04:04know, the nature of
04:05customer issues, the way that
04:07they engage with customers,
04:09is it like direct contact?
04:11Is it, you know, is it mostly self serve?
04:14Technical challenges are very different
04:16from one company to the other.
04:18So we offer really an
04:20end to end solution.
04:21So there's multiple
04:23components in our system.
04:24It all starts with understanding your
04:26system of record, which is
04:28things like your underlying
04:30customer knowledge, your, you know, your
04:33documentation, your CRM, the API is that
04:37you use for automating
04:39parts of your support.
04:41And we connect all that together.
04:43And then we build applications that are
04:45either customer facing
04:46or agent facing so and use
04:49internally for the company.
04:51So it can be, you know, everything from,
04:52you know, your contact form powered by AI
04:55to your chatbot to your
04:58Slack and Discord bots, right?
05:00This is customer
05:01facing email order replies.
05:04So you send an email and the AI agent
05:08will start, you know,
05:09trying to resolve the issue
05:11for you.
05:12And it might be possible
05:13that it resolves it fully.
05:16And if not, it's going to ask you for all
05:17the questions that the team then needs to
05:19use once you know, the ticket
05:21hits the inbox internally.
05:23And then there's the all the features
05:25which are internal
05:26facing for the support
05:28team to be able to streamline their work.
05:31Because the truth is that customer
05:32support before AI involved
05:36a lot of essentially like
05:38robotic work, just answering the same
05:41questions over and over again.
05:42This is not super engaging
05:44and exciting just to have
05:45to answer the same
05:46question over and over again.
05:47So if you can start like taking that
05:49burden off of your
05:50shoulders and help you, you know,
05:52work on more like high leverage strategic
05:56questions, then, you
05:58know, we can really help
05:59streamline the support operation.
06:00what happens when you've put
06:01an AI or both like customer
06:03facing and agent facing, then you can
06:06start, you know,
06:07extracting a lot of intelligence,
06:09a lot of knowledge, because you're really
06:10at the nerve center
06:11in a company, you are,
06:13you know, the interface between your
06:15customers and your company, right?
06:18So you have so much like knowledge that
06:20often just gets siloed
06:21within the support team,
06:23right?
06:23It's it's actually very rare that support
06:25team reports to product, right?
06:28And I think that's a
06:29that's really a problem.
06:30But if we can start rooting
06:32out all this insight and
06:33knowledge, we can start
06:35having a really, really good
06:36understanding on, you know,
06:37product shortcomings, where
06:39the engineering team might,
06:40you know, put some effort, right?
06:43Because this is where your customers have
06:44been experiencing a
06:46lot of friction, right?
06:47So we can start generating these
06:49reporting and all sorts of
06:51other insights that can help
06:53you as a company
06:54streamline your operations,
06:56things like spotting knowledge
06:57gaps, right, helping you write, you know,
06:59keeping your your
07:00documentation up to date,
07:02these are all things that
07:02we can start also working on.
07:05So this concept of
07:05agent is very versatile.
07:08The agents can solve issues, they can,
07:10you know, answer emails,
07:11but they can also start
07:12being tasked of gathering
07:14insights and producing new
07:15knowledge and keeping your
07:16system of record,
07:17always up to date.
07:18So there's really a lot of
07:19things that need to happen.
07:20You cannot approach the
07:22problem just from one
07:23angle, you really want to tie
07:24the whole thing together, because this is
07:26where you start creating
07:27these positive feedback
07:28loops, so called data flywheels, right,
07:31that enable you to
07:32start deploying the solution
07:33and trust it that it's
07:34operating really well.
07:36So yeah, there's,
07:37there's a lot of things to it.
07:38That sounds fascinating, and also quite
07:41complex in terms of the
07:42different surface areas, etc.
07:45So maybe we can go
07:46slightly more technical.
07:47And before we go into all of the Effect's
07:49specifics, can you more from like an
07:52engineering perspective
07:53provide an overview over your system?
07:56And which different high level components
07:59it's composed out of?
08:01Sure.
08:02So this is a, you know, apart from the
08:04machine learning things,
08:06the LLM things, this is a
08:08classical, you know, setup.
08:09there's some servers, both
08:11like short, like serverless instances for
08:13API endpoints, then
08:14there's some long running
08:16servers to do things like, you know, pre
08:18processing data, post
08:21processing, interactions.
08:23So yeah, there's a there's a bunch of
08:25servers here, everything
08:25is in node, everything is
08:27or TypeScript.
08:29The front end is a React application.
08:32So both we have a dashboard, which is a V
08:35duplication, there is
08:37some integrations
08:38inside of Salesforce,
08:39Zendesk, Slack, and so on.
08:41So it's a classical full
08:43stack, React TypeScript application.
08:47So you've built quite an expansive system
08:49here and building all
08:51of that in TypeScript
08:52I think TypeScript is sort of notorious
08:54for only scratching the
08:55surface and only covering
08:57the happy path.
08:58There's a lot of things that can go wrong
09:00in all of those systems, integrating with
09:03different systems.
09:04All of those things I think have led you
09:06to take a more serious
09:08look at Effect and ultimately
09:10adopting it.
09:11So can you share that journey of going
09:14from a non-Effect code
09:16base to converting step
09:18by step to Effect?
09:20Sure, so obviously there's the common
09:22complexity increase that
09:24happens when you're building
09:25a product and putting it into production.
09:27So as you add features and so on, then
09:29you hit sort of a ceiling
09:31and then you start assessing
09:33the situation and then you might start
09:35refactoring and so on.
09:36And then next phase, you hit some other
09:39limitations and then you
09:41reassess again the situation.
09:44So that's just the common thing of
09:45working on an
09:46increasingly growing code base.
09:49But then there's LLMs and these are
09:51non-deterministic systems and it poses
09:54some completely new kinds
09:56of challenges.
09:56How do you monitor the behavior of the
09:58system, for instance?
10:00How do you deal with readable streams
10:02instead of just sending over a payload?
10:05This has the potential to making your
10:07code base more complex
10:08if you need to stream some
10:10tokens down to the customer, but you
10:12actually also need to
10:13store these chunks on your own
10:16for post-processing.
10:17How do you deal with that?
10:18TypeScript doesn't give you
10:19some great answers to this.
10:22How do you deal with reliability?
10:24OpenAI and Anthropic and so on, they offer
10:27a service which is in high demand.
10:31We are in a period right
10:33now with scarce resources.
10:35So these APIs, they go down and they
10:37actually go down more
10:38than you would think.
10:40And so how do you build fallback
10:42mechanisms so that, say that
10:44OpenAI is down as your main
10:46provider, your customers cannot be down.
10:48If OpenAI is down,
10:49that's not their problem.
10:50That's our problem.
10:52So how do you build a
10:53fallback system to Anthropic?
10:55What happens a few weeks ago when OpenAI
10:58was down for, I think,
11:00five hours and then the
11:01entire industry shifts towards Anthropic
11:04that just has this
11:05spike that they're unable
11:07to deal with?
11:08So they are experiencing outage or just
11:11very degraded performance.
11:13There's outage and then there's actually
11:15performance, the
11:16speed of tokens and so on.
11:19All that, there's no good answer, no
11:22clear best practice in
11:25vanilla TypeScript to deal
11:27with these things.
11:28And then there's the fact that we are a
11:29startup working in a
11:31fast-paced environment where
11:33things change all the time.
11:36And we want to be efficient and we want
11:37to be able to not deal
11:39with all the complexity
11:41and all the problems and down times and
11:42reliability, but we need to
11:44be effective in shipping new
11:46features to constantly
11:47meet up with the demand.
11:49the zero to one of
11:50building a company here.
11:51So there's multiple factors that really
11:54made us think hard about
11:55what would be the foundation
11:58for us to be able to operate really fast
12:00with a small set of
12:02resources from a startup and
12:04be a highly reliable solution because we
12:07are catering to larger
12:08companies, enterprise
12:10companies.
12:10So reliability is just really key.
12:13Obviously, I can talk a lot about the
12:15whole telemetry part of
12:18things, making sure everything
12:20is properly monitored and that we can
12:21actually spot when things
12:24go wrong before our customers
12:25even see it and then we can take action.
12:28So there's a bunch of things here.
12:29And Effect just came at a very good
12:31moment where we still had
12:33a sufficiently small code
12:36base that we could actually start
12:38seriously thinking about
12:39completely converting it to
12:40Effect.
12:41So the timing was really good.
12:43So you've mentioned in regards to
12:45adopting Effect for the
12:46sake of improving reliability,
12:49one specific use case was that Markbrom
12:52is using LLMs such as
12:54the products provided by
12:56OpenAI.
12:56And if they go down, you need to fall
12:58back to another system.
13:00And in case that system also goes down,
13:02fall back to yet another one.
13:04All of that is helping to improve the
13:07reliability of Markprompt.
13:10Which other challenges did you face
13:12similar to that one in
13:14regards to reliability and
13:15how did you improve
13:16reliability through Effect?
13:19I mean, there's lots of examples where
13:21Effect has helped us
13:23consolidate our system.
13:25One example is one day we woke up and we
13:28realized that we were under
13:29attack by a malicious hacker
13:32who was interacting with a public facing
13:35deployment of Markprompt and
13:37sending, I think, 20 million
13:40requests per hour.
13:43And this was across the globe, so it was
13:46kind of hard to deal with.
13:47So Ethan Niser, he spent the summer with
13:50us and he's been working
13:53on a lot of Effect-related
13:54things and on his first day,
13:56this attack started happening.
13:58And so he built this great rate limiting
14:02system in the middleware
14:03that is capable of looking
14:04at multiple facets of a payload.
14:08And the interesting thing here is that
14:09you cannot just rely
14:11on IP and fingerprinting
14:12and so on.
14:13You actually also want to start looking
14:14at the LLM payload itself.
14:16What kinds of messages is there, pattern
14:18that we can spot here in
14:20the nature of the requests?
14:22And so it's a multi-tiered rate limiting
14:24system that is very easy
14:25to automatically trigger
14:27and also manually trigger.
14:28So this is an example also where building
14:30with composability has just allowed us to
14:34build a fairly sophisticated
14:35system here, which just works.
14:38We are now very
14:39confident that it just works.
14:40And by the way, one of the reasons that
14:42we're confident that it
14:43works is that it's very
14:44easy to test because the whole dependency
14:46injection story is
14:47just such a natural thing
14:48to do.
14:49You just operate a provider and you can
14:51very easily simulate a
14:53situation that is hard to
14:55reproduce with other means.
14:56So dependency injection, I would say, is
14:59a huge factor to the
15:00reliability because we
15:01can restart testing all sorts of
15:05different combinations of
15:06non-happy paths and building
15:08that with just change or service that you
15:12provide is just very,
15:13very easy with Effect.
15:14Telemetry is obviously a thing.
15:16So I think that the whole testing story
15:19is not just about one thing.
15:22You want to do unit tests.
15:23You want to do integration tests with
15:25dependency injection.
15:26You also really want to have a full
15:28understanding of what's going
15:29on when you are in production.
15:31And so being able to have good traces,
15:33good logging of every
15:35component of your system
15:36is non-trivial.
15:38And so we rely heavily on telemetry and
15:41Effect allows us to just
15:43add spans everywhere in
15:45a very, very simple way.
15:46And it just naturally fits
15:48into the rest of the system.
15:50And this really has allowed us to have a
15:51very, very good
15:52understanding of failure scenarios
15:55before our customers even see them.
15:57So these are just general patterns that
15:59are very natural to do
16:01in Effect that we don't
16:02even think about because they actually
16:05don't incur any sort of
16:07additional thinking on our
16:08part.
16:09It's just adding a line
16:11and then it just works.
16:12Yeah, this makes a lot of sense.
16:14I can totally see how reliability is
16:17almost sort of like a emerging factor,
16:20like a second order effect that comes
16:22when you have a composable system,
16:25and when you have observability, when you
16:28can look into what's going on,
16:30when you can eat your
16:32vegetables, then you're healthy.
16:34And so
16:35Building a reliable system is going the
16:37extra mile on all of those things.
16:39This makes a lot of sense.
16:41Given that you built
16:42quite an expansive system,
16:44maybe we can try to break it down a
16:46little bit more with a technical lens on.
16:49So I'm sure that you have
16:51a client side application.
16:53You've mentioned that it's like a single
16:54page app built with Vite,
16:57but more on the backend side.
16:59Can you describe to me what are the
17:02different subcomponents of your system?
17:04Yeah, so in a very simple way, the main
17:07subcomponents here,
17:09there's the application server,
17:11which is essentially an RPC
17:12server that the client
17:14application is speaking with.
17:15Then we have a public API.
17:18So Markprompt is built as
17:19Stripe for customer support,
17:21in the sense that every part of the
17:23system can be
17:24interacted with via our APIs
17:26so that you can actually build deeper
17:28product experiences
17:29inside of your core product.
17:32So we have a public facing API server.
17:34Then we have the ingestion engine, which
17:37takes in all your systems data.
17:40So this can be public facing like a
17:41website that we would
17:42scrape on a regular basis.
17:44It could be GitHub repository or
17:47historical tickets from
17:49Zendesk and Jira and Confluence
17:51and databases from
17:53Salesforce and whatnot.
17:55So we do a bunch of things here.
17:57We import data.
17:58We chunk it up.
17:58We monitor the whole structure of
18:02documents through the AST
18:03and then we build
18:04embeddings for multi-levels.
18:06And then we store them
18:07in a vector database.
18:08And then we have all the
18:09application related servers,
18:11so things like Discord or
18:13Slack or Zendesk and so on,
18:15which are more like simple servers here.
18:18Got it.
18:18Yeah, that makes a lot of sense to break
18:20it up into those different pieces.
18:22I'd actually love to learn a
18:24little bit more about each,
18:25maybe starting with
18:26the application server.
18:28So you mentioned that you're
18:29using Effect RPC in that regard.
18:31What did you use before
18:33you were using Effect?
18:34And what are the qualities of using
18:37Effect for an
18:39application for an RPC server here?
18:41Yeah, sure.
18:43I mean, this before was
18:44just like simple API handlers.
18:46So you would have two
18:48completely decoupled systems.
18:51And now pretty much all the
18:53endpoints are in full RPC.
18:56It starts with a schema, actually.
18:58We define the input schema, the output
19:01schema, and the error schema
19:03for each of these endpoints.
19:05And then it trickles down all the way to
19:07the client application.
19:08So we have this.
19:09Ethan actually wrote this and I hope he's
19:10going to publish it.
19:11Sometimes soon, essentially React query,
19:13but wrapped around with Effect
19:15so that we have the full schema from the
19:17backend all the way to
19:18the client side code.
19:19And we have nothing more to deal with.
19:21So the whole thing is completely
19:22streamlined in terms of the
19:24communication between the client
19:26and the server.
19:28That makes a lot of sense.
19:29And so before you
19:30didn't have any schema layer,
19:32anything to express the kind of data
19:35you're sending across?
19:37No, we didn't actually.
19:39Got it.
19:39Well, I suppose that through the process
19:42of just modeling the schema,
19:44I'm sure that there were a couple of code
19:46paths that you didn't
19:47even account for before.
19:49So the--
19:50Absolutely.
19:51The public API server where you're
19:55exposing an open API spec, I suppose,
19:59how similar or different is that to
20:01your application server?
20:02Can you describe the public
20:03API server setup a bit more?
20:06Yeah, sure.
20:06So this is more of a
20:08standard HTTP server.
20:10This is actually using Effect HTTP.
20:12So again, we have the full--
20:13Everything starts with the Effect runtime
20:17taking over directly
20:18at the request level.
20:19So again, we can model
20:20everything very nicely via schemas.
20:23And one really nice thing of having
20:26everything defined by schemas is
20:28that we can, for instance, automatically
20:30generate the open API spec.
20:32So this is something that we now have
20:34every single endpoint is
20:35automatically documented
20:37just via the schema.
20:39Got it.
20:39How much overlap does your
20:41internal application server,
20:43that RPC server with
20:45the public API server,
20:47does it have a lot of overlap?
20:49Do you reuse a lot of code internally?
20:51Or are those rather
20:52fulfilling different purposes?
20:54They're fulfilling
20:55completely different purposes, yes.
20:58So yeah, I mean, these things are
21:00completely separate.
21:02So to give you one very important
21:04differentiator is that on the RPC server,
21:07we provide permissions.
21:10So you want to make sure that a given
21:12user can perform an action
21:14that they are allowed to.
21:16And so we have a very nice system that
21:19provides the level of
21:21permissions of a user.
21:23This is completely transparent.
21:25We actually used RLS before,
21:28role level security to
21:29enforce these policies.
21:31But there was a real performance impact.
21:33It was just slowing down the crazy,
21:34especially when you're dealing with
21:36hundreds of thousands of data points.
21:39So this had a real impact.
21:41So we got rid of that.
21:42And actually, everything
21:43now is modeled in code.
21:44So the permission, which I guess was
21:46always how things have
21:48been done without RLS.
21:52And so this is, again, an example where
21:54Effect at the type level,
21:56you can immediately spot whether this is
21:58a user that is allowed to
22:00perform a single action,
22:02because that service
22:04requires these sets of permissions.
22:06And so this is very,
22:07very nice to model as well.
22:09That sounds very interesting.
22:10So you've built yourself an internal DSL
22:13to model those permission rules.
22:16Can you explain a little bit more how
22:18that's implemented and how
22:20that can be imagined from
22:22like a type level
22:23perspective or from an API perspective?
22:26Yeah.
22:27I mean, so every user comes
22:28with a set of permissions, right?
22:30And these permissions
22:30can be modeled via types.
22:33And services can declare what level of
22:38permission they required in
22:40order to perform an action.
22:42And so we immediately know if within an
22:45RPC endpoint, we're using a service,
22:47but that the provided
22:48permissions don't match.
22:52Well, then we can flag it.
22:53And so we can guarantee that there's no
22:55code being shipped where
22:56there's an inconsistency here.
22:59I love it.
22:59That sounds very elegant.
23:00I would love to see some code for that.
23:03Could you imagine that there could be a
23:05more commonly reusable package for that?
23:08Or is that only something that makes
23:10sense within your code base?
23:13Good question.
23:15Obviously, we have been just crunching
23:18through this code and
23:19getting into production.
23:20So we haven't thought too much about how
23:23this would extend to a
23:24more generic use case.
23:27But I guess this could be
23:29a very interesting topic.
23:31Right.
23:31Well, anyone in the audience is happily
23:34invited to try to
23:35replicate this by themselves.
23:37Maybe that's even something that we
23:39could, as part of the Effect org, also
23:42look into providing.
23:43To hear you describing
23:44this use case in this DSL,
23:46that makes a lot of
23:47sense for a lot of use cases.
23:49So moving then to the ingestion engine,
23:53can you describe a little bit more of how
23:56that looks like from a
23:57lifecycle perspective?
23:59So you've mentioned the ingestion engine
24:01is mostly meant to bring in data from
24:03various data sources,
24:05whether it's
24:05Salesforce, whether it's Notion.
24:07Walk me through how that works and how
24:10does that work in regards to Effect?
24:13Do you use Effect streams for that?
24:15Yeah.
24:15How does that work?
24:17So it's still pretty bare bones here.
24:19This is something that we have been
24:20discussing also with the
24:21Effect team and the Effect cluster.
24:23This is where we will probably be going.
24:26So now we have our own system, which is
24:28basically just some
24:31long running node servers
24:32that are able to go and fetch the data.
24:36So that could be make a large request to
24:39an SQL database or starting a scraper
24:42that visits a website and when it's done
24:45and purged all the pages,
24:48it is then indexed in a
24:49temporary cache on our side.
24:52And then it goes into the post processing
24:53pipeline, where again, we obviously we
24:56parallelize all this
24:58because every page
24:59needs some post processing.
25:01So we chunk it up and we try to get a
25:05sense of the structure
25:06of the document, the AST,
25:08and then we build these embeddings for
25:10each levels in the AST and
25:12then we store it in a database.
25:15And so the main challenge here is that
25:17these are long running
25:18tasks which can time out,
25:20they can fail and whatnot.
25:24And so now we rely on external systems to
25:26do the whole workflow orchestration,
25:28but this is something where we are going
25:30to be investing some
25:30effort as well because
25:32Effect is the perfect solution here
25:35because we have the full
25:37sets of tools for dealing with
25:40these kinds of situations.
25:42Yeah, I'm really looking forward for
25:45Effect Cluster and Effect
25:46Workflows to mature further.
25:49I think there's more and more use cases,
25:52particularly when it
25:53comes to AI applications
25:55where you want to express
25:56things as long running workflows.
25:59And I think this is where Effect is
26:02really like such a perfect foundation.
26:05So yeah, I think we're going to see some
26:08very interesting updates
26:09throughout the year here.
26:11So stay tuned for that.
26:13Maybe changing gears slightly.
26:15Initially, you've built all of the
26:17system, I think mostly by yourself.
26:19This is where you started
26:21adopting Effect by yourself.
26:23and then later you onboarded other
26:25engineers who didn't have
26:27prior Effect experience.
26:29Can you share a bit more about that
26:31story, how you onboarded
26:33other engineers who didn't
26:34have Effect experience to become
26:36successful working on your code base?
26:39I mean, this is such
26:39an interesting question.
26:40Because you would think that Effect is
26:44mainly for a TypeScript audience.
26:47Web developers who are proficient now in
26:50TypeScript and realize they
26:52need to go to the next level
26:54in terms of building complex systems.
26:56My take here is actually that having a
27:00code base which is already
27:01fully Effect will actually
27:04lower the barrier to entry for people who
27:07have less experience with TypeScript.
27:10So we've seen that with Elliot, our
27:13founding engineer who has
27:14joined a few months ago,
27:15with very little TypeScript experience,
27:17was able to very quickly take
27:20it up and then be productive.
27:22We've run some trials with some
27:23candidates in the last few weeks with,
27:26again, little experience in
27:27TypeScript.
27:28The thing is that we are hiring a lot of
27:30machine learning engineers.
27:32And so they have typically already a lot
27:35of experience with more
27:36of the machine learning
27:37sets of tools like Python
27:39and very little on TypeScript.
27:41But what we have seen is onboarding them
27:44on our code base, they
27:45were actually able to be
27:46very productive, very early on.
27:48And the great thing is that there's not
27:50much room for bad
27:51practice or the kinds of things
27:53that you need to learn when you get
27:55started with a new language.
27:56So what I think is that actually
27:58Effect will
27:59allow them to be proficient and
28:02build very reliable
28:03code within a few weeks.
28:05If it had not been an Effect and more
28:07like vanilla TypeScript,
28:08yes, sure, there's fewer
28:09things to learn.
28:10It's kind of easy and so on.
28:12But you would build up this understanding
28:15and intuition for the
28:16language over the course of
28:17years.
28:18Right.
28:18And then within years, you'll be able to
28:19build very complex and
28:21reliable systems because you
28:23have been proficient in the language.
28:26So Effect gives you these guardrails that
28:29with the whole thing is
28:31in Effect, sure, you need
28:32to understand the
28:33whole catalog of things.
28:35And this you just take up
28:36as you have new use cases.
28:37But if you have a really good foundation,
28:39like the full RPC
28:41server, you basically just
28:43you start writing schema, right?
28:45You write a schema and then
28:46you fit it into the system.
28:48And then once it's there, it's done.
28:51You don't need to change it.
28:52You haven't done anything bad
28:54or anything that will break.
28:56Right.
28:56So my take on this is that actually it
28:59might actually have the
29:00opposite effect of what
29:01we would expect, which is to be able to
29:03onboard non TypeScript
29:06engineers much, much faster.
29:09Yeah, that definitely sounds
29:11a little bit counterintuitive.
29:12And the first time I heard about this
29:15anecdote, and you shared
29:16this privately with me before,
29:18and that person you've just mentioned,
29:21Elliot, gave a brilliant talk at the last
29:23Effect Meetup in San Francisco where he
29:26shared his first hand
29:28experience learning Effect.
29:30And yeah, I think that is a
29:32little bit counterintuitive.
29:33But when you think about it more, it
29:35makes a lot of sense because most
29:37experienced TypeScript
29:39engineers over years, they had to first
29:42learn how to do anything
29:44with TypeScript at all and then
29:46later learn to ignore like those 80 paths
29:49out of 100 how you could do
29:52something with TypeScript,
29:53but that will lead you
29:54down a dangerous road.
29:56Whereas Effect constrains you a little
29:58bit more and puts you on a
30:01path to success in a way.
30:03That's also like it has been my
30:04experience for I think for me
30:06coming more from TypeScript,
30:09I had to unlearn a lot of things how I
30:12used to do things in TypeScript before.
30:14But this is where for me, a good
30:17foundation to pick up Effect
30:19was actually dabbling in other
30:21languages such as Swift and Rust, where I
30:24already primed my mind more on more saner
30:29engineering practices and learning the
30:31right habits and
30:33embracing the right abstractions.
30:35And so I can totally see how someone who
30:38doesn't have prior TypeScript experience
30:40maybe experience in other languages,
30:42which have more appropriate primitives
30:45can be up and running with Effect in no
30:48time. That is amazing to hear.
30:50So can you walk me through a little bit
30:53more through the process
30:54how you went tactically
30:56about migrating the code base as it was
31:00before Effect one by one
31:02step at a time to migrating
31:03it to Effect. Rome wasn't conquered in a
31:06day, so wasn't your code
31:08base. So how did you go about it?
31:10We had some great help. And this is where
31:12I think the Effect community
31:13is such a wonderful community
31:15of people who are nice and helpful. You
31:19kicked it off with us. So we've been
31:22this for years. And immediately I saw the
31:25importance of this, but
31:27never found the time to
31:30actually learn how to do it in practice.
31:32I think this is also
31:33maybe related to the lack of
31:34documentation and concrete examples back
31:36in the day. How do you
31:37actually convert an API
31:38endpoint to Effect? How do you just do
31:41it? So I think this little
31:43catalyzer was really important,
31:45which allowed us, our team, to then get a
31:49sense of where we could
31:50start fitting it in. It was still
31:52like very, very new. This was earlier
31:53last year. And then we had the pleasure
31:57of having Ethan with
31:59us over the summer. And he wrote a lot of
32:01very important code for both
32:03the backend side of things,
32:05schema, the database, the React query
32:08part, which put us in a
32:10really good foundation.
32:11We had most of it basically sketched out.
32:14And then in September, we decided to
32:18do a big, big push and just get
32:20everything migrated. So this was like a
32:23bit slow in the beginning.
32:24And then very, very quickly, within a few
32:27weeks, we had the
32:28whole thing migrated over.
32:32Good. That makes a lot of sense. And
32:34happy to have acted as
32:36a little catalyst here,
32:38but you've picked up everything so
32:40quickly. So I think this was rather a
32:43little bit of giving you
32:45more courage to go this way. And I think
32:48you figured out quickly
32:50how to work with everything,
32:53and you migrated everything in no time. I
32:57would say, honestly,
32:59the important thing is the
33:00catalyzer here. Once you have the starter
33:03blocks, then it's quite easy to
33:05generalize. It's more
33:07like where do you start? And the
33:09documentation obviously is much better
33:10now. So I think this is
33:11a great thing. But I think this little
33:14initial nudge is
33:17what made the difference here.
33:19If you think about your system as sort of
33:22like a tree of different subsystems,
33:27you can go about converting an existing
33:29system to Effect in different ways.
33:31Either you go first for
33:33the leaves, and then you go up one step
33:36at a time until your entire
33:37program, your entire system is
33:39refactored with Effect. But you can also
33:41isolate specific sub nodes
33:44and from the outside call the
33:47effect and inside of it call other
33:50things that are not yet Effect with
33:52Effect.runPromise or
33:54effect dot promise to run some effects,
33:57some non-effect code yet.
34:00Did you find any particular
34:02topology adoption strategy for you to be
34:06working well? So did you go first
34:08all for the leaves? Or did you go
34:11wherever you wanted to
34:12refactor something anyway? How did you
34:14go about the decision process of when to
34:17refactor what with Effect?
34:19Well, yeah, it was the leaves in
34:20the beginning. And then it was the top
34:24down afterwards, because
34:26we just realized that this
34:27was what we needed. And we just needed as
34:29quickly as possible. So
34:31then there was a combined push
34:34from the whole team to just get the
34:36migration done as quickly as
34:37possible. So the RPC was written
34:40really like top down, right? Because,
34:43well top down in the
34:44sense that one route would be
34:47migrated to RPC, Effect RPC, then another
34:49one and so on. But for
34:52sure, this is a nice thing with
34:53Effect is that you can do it on the
34:55leave to start with and you're not
34:57touching anything else
34:58of your code base. But yeah, you
35:00definitely feel the urge to
35:01get the whole thing end to end.
35:05The runtime should be initiated at the
35:08very beginning of the
35:10thing, right? So that you have
35:11the full happy path, because you get to
35:14love the happy path here, right?
35:16Nice. So if I remember
35:18correctly, I think it's now been pretty
35:20much a year since you've been using
35:22Effect in production.
35:24looking back, what were some
35:26of the things that were
35:27like unexpectedly good with
35:30Effect and also some things that were
35:32more challenging that you thought or
35:33unexpected in some ways?
35:36I didn't expect the Schema package
35:39to be, you know, I didn't have any
35:42expectations that this
35:43was really a core thing. In fact, because
35:45this is really wonderful, how you can
35:49control both like the
35:50decoding part, the encoding part, how you
35:53can pipe everything
35:54together. This is one of the sort of,
36:00it shifts away how you think about a
36:01problem. Because usually,
36:03now when I think of a problem,
36:05I start thinking about what is the
36:06schema, right? It is very
36:07different from what is the actual
36:09application code. You redesign your
36:11schema, you think, well, how should it
36:13transform the data here?
36:16And then you work your way up to getting
36:18it all the way in your
36:21code. So I think that schema was
36:23a very, very positive,
36:24unexpected thing that came out of this.
36:27To give you an example,
36:29I always wanted to, again, in the quest
36:32for extreme lining and being
36:34as efficient as possible as a
36:36startup, things like documentation, I
36:39care a lot about documentation. And the
36:41more we can put that
36:42on like autopilot, the better, right? So
36:44the ability to take
36:46your public API endpoints,
36:47and the other schemas generate an open
36:49API spec is exactly the kind of things
36:52that you want, right?
36:54So this was a very, very nice discovery.
36:57Right. That makes a lot of
36:59sense. And I think that goes
37:00hand in hand with what I've also found in
37:04my development practices over the years,
37:06is that I always started to think more
37:10about the types of a system.
37:13That's like typically what I
37:13start out with. I just like described the
37:16types. But then I've also
37:18found myself thinking about,
37:20actually, this data of this particular
37:22type doesn't just live inside of this
37:25boundary of the system,
37:26but it needs to move across network
37:29boundaries, across threat boundaries,
37:31etc. And now you're,
37:33you need to serialize, deserialize that
37:35variable. And this is where
37:38schemas come in. So I ended up
37:40doing exactly the same as you. Like I
37:43start thinking about like,
37:44okay, what data am I dealing
37:46with here? And I modeled it as schemas
37:49and derived the types from it.
37:51And well, we see this here as
37:53like a big revelation turns out in other
37:56programming languages, that's what you
37:58always do. When you have
37:59like in Rust, etc. When you have like a
38:02struct definition, you can
38:03derive schemas from it. But
38:06I think for TypeScript, this is really
38:08for most engineers, that's
38:10something novel. And I think
38:12this is a pattern that should be embraced
38:14much more since I think
38:16default is you just ignore
38:18the existence of a schema and you JSON
38:21stringify everything. And then you're
38:23surprised what happens
38:25to your classes or to your dates. So
38:27yeah, that schema is such
38:29an integral part of Effect,
38:32I think is no coincidence. And I'm very
38:34happy that it gets so
38:36much attention for detail.
38:38Yeah, I mean, I would say it's not only
38:41the fact that you have like
38:42decoding and encoding and so
38:44on, it's the fact that it's so much in
38:48harmony with the rest of
38:49the whole like Effect way of
38:51doing things, right? So it just feels so,
38:55you know, an integral
38:57part of effect itself.
38:59Yeah, I fully agree. So in regards to the
39:04more challenging aspects,
39:06what were some of the challenges that you
39:08found working in an Effect code base?
39:11Sure. I would say it can give you a sense
39:14of, you know
39:17maybe a
39:17false sense of security.
39:19Because you think that everything
39:20is dealt with at the
39:21type level and you get no
39:23flagged of compile time errors and so on.
39:26And this is partly
39:28true. But I think it's very,
39:30very important. It's very important to
39:33understand that this is not
39:34the case. And it gives you
39:36really good like foundation, but you're
39:39not exempt of or die or catch all that
39:44was put at the end of
39:46your pipes. And so all the other errors
39:48in between are, you know, just you don't
39:52see them. And we had
39:52an example where we had this
39:55bug in our code base, and it
39:56was quite hard to figure out
39:59where it went wrong. So I think this is
40:01something that, you know,
40:02Effect is a very, very powerful
40:03tool. And it brings out the best in us,
40:08you know, engineers, I think.
40:10But it also needs to be
40:12dealt with knowing that, you know,
40:15it's not like a safety guard
40:17or anything, you can mess up
40:19and you can do things that are very hard
40:21to then investigate because
40:23you've built up this, this
40:24sense of everything is just working
40:26smoothly. So this is one thing I think
40:28that this is as important
40:30to keep in mind when you're building, you
40:32know, on top of Effect. I
40:34think another thing that it gets,
40:35a little bit hard to track is,
40:38dependency
40:39injection is just wonderful,
40:42in Effect. And I think that's, that's, I
40:43would mention this probably
40:44as the second most important,
40:46discovery or delight of
40:48working with Effect. But,
40:50you know, where do you put your
40:53dependency? Where is it in your code? How
40:55do you like chase the
40:56dependency, you know, the service
40:58that you provided, that is now affecting
41:01a completely different
41:02part of your code base,
41:03that might be, you
41:05know, some some tooling
41:06to build inside of the
41:07IDE itself, to be able to make it easy to
41:10spot the connection, right.
41:12But now we still like in this
41:13very barebone situation where it's kind
41:15of hard to understand
41:17exactly, you know, the hierarchy of
41:20your dependencies, right, and where they
41:23are actually being used
41:24in your code. So it can be
41:26a little bit hard to navigate sometimes.
41:28Right. Yeah, I found that
41:29myself as well in the project
41:31I'm working on. And it is actually
41:34something that we're thinking
41:35about, that we could possibly
41:37enhance through a IDE integration, etc.
41:41So stay, stay tuned for
41:44for that. You said something
41:46interesting just before, where you said
41:48effect brings out the
41:50best in us engineers. Can you
41:53elaborate on that? What do you mean by that
41:55Well, as an engineer,
41:58you want to build features,
41:59you want to build like real robust
42:00systems, you want to spend
42:03ideally all your time doing
42:05that. You don't want to spend your time
42:08fighting the language, fighting, you
42:10know, building something
42:11that you know, the language was not meant
42:13for, to give you an
42:14example, like concurrency and
42:15TypeScript, right, these are things that
42:17you know, you want a way to,
42:21you know, build a concurrent
42:22system, where you're thinking about what
42:25is the system actually doing
42:26rather than like setting up
42:28all the boilerplate in order to have that
42:30system in place. So I would
42:34say Effect really gives you
42:36the right premise. So one thing I'm
42:40thinking about is, now that we've
42:42completed this migration,
42:44the full migration, it kind of feels like
42:46this is the end game. I know
42:48this might sound a bit like
42:50naive or unrealistic, but it kind of
42:54feels like more stable.
42:56And at least if we continue
42:58building our code base on TypeScript with
43:00Effect, it doesn't feel
43:02like we're going to spend
43:04many more like cycles, like refactoring
43:06the code, right, it feels
43:09like this sort of timeless,
43:12like code base now now now it's, it's,
43:14it's there. Effect is
43:16going to evolve, our code base is
43:17going to evolve. It feels like this is
43:19going to be like adjustments
43:20and not like a full rewrite.
43:22And this is what I mean by, you know, it
43:24feels like kind of like
43:25building a house with some really
43:27good like walls and floor and ceiling and
43:29so on. It's just there now.
43:32And this is what we are all
43:34I guess, striving for as engineers is to
43:36have these, you know, the
43:38right tools, right, the right
43:39foundations, instead of having to fight,
43:42you know, all these like small things
43:44that degrade over time,
43:46and then need to revisit them. And you
43:48realize how silly you were
43:51writing this code, you know,
43:52last year kind of thing, it feels like
43:54more harmonious and stable
43:57Right. I love that. And
43:59I think that is also something that has
44:02really brought me to Effect.
44:04And to flip it on its head,
44:06something that I experienced so many
44:10times and have observed so many other
44:13teams as sort of like a
44:15malfunction of engineering teams and of
44:18projects that engineers are
44:20afraid to make changes in a
44:22project. And I've seen this time and time
44:25and time again. And when
44:27you're afraid to make changes to a
44:29project, particularly if the code was
44:31written by someone else, maybe
44:32the person is no longer there.
44:35That's terrible. Like if you're, if
44:36you're afraid to change something, you
44:38might not change it and
44:39you might not improve it. Or if you
44:41change it, you don't know
44:42about the consequences, etc.
44:45And effect allows you to, gives you the
44:48confidence to change something and you
44:51know what's going to
44:52happen. It even makes it fun to do so.
44:54And I think that's
44:55sort of like a twin to the
45:00aspect of composability, almost like the
45:03other side of the same
45:04coin, where if you can compose
45:07things together, you can decompose things
45:10that makes, that's, those
45:11are just aspects of changing
45:13something and changing not in a big bang
45:17way, but just moving things
45:19around almost feels like a,
45:21like a dance. We're getting a little bit
45:23too, too like pie in the sky
45:26here, but that's really like
45:27my reality as well, like working with
45:29Effect and one of the things
45:31that I like so much about it.
45:33And another thing here may be looking a
45:35little bit more into the
45:36future, but we're increasingly
45:37living in the future where we get, where
45:40we do a lot of our code
45:42development in an AI-assisted way.
45:45Now it's maybe no longer just us, us as
45:48engineers doing performing
45:50changes on our code, but it's now
45:52AI systems performing changes on our
45:54code. And this is where the same
45:56underlying dynamic is even more
45:58severe. If there is a system, if there is
46:02some actor here who's
46:04changing our code, who is maybe
46:07even less competent or less knowledgeable
46:10about this, then maybe even
46:12more scary to make changes
46:14to a system. And if the material here,
46:18the house that we're
46:18building that you've mentioned,
46:20is where it's inviting to have changes be
46:24made, that sets an even better foundation
46:27for those AI-assisted,
46:30for this AI-assisted future. So that's
46:33something I love thinking about.
46:35Yeah, I agree. I think the guardrails
46:41here are so important,
46:42right? You want to make sure that,
46:44whatever the LLM generates has as little
46:49unintended side effects as possible,
46:52because it could hallucinate some code.
46:55So if you can make sure that
46:57the blast radius is as tight,
46:59small as possible, then I think you have
47:01a really good foundation. Obviously, I
47:03don't know where this
47:04is heading. Maybe we're going to be
47:05completely wrong and that we're not even
47:07going to be thinking
47:07about code in the same way in the future.
47:12But it feels like it's a
47:13really good way of ensuring
47:16that whatever an LLM generates stays
47:20within the intended radius.
47:25Exactly. And I think just to round this
47:28part out, Effect gives
47:30you a middle ground between
47:34a natural language like English that
47:37might be more intuitive to
47:40describe what a system should do,
47:41but maybe less concise, as opposed to a
47:45type system or code,
47:47which can be more concise,
47:49but maybe less intuitive. And I feel
47:51Effect is the best of
47:52both worlds, where you can,
47:54through the help of a type system, and
47:56here in this case,
47:58TypeScript provides a marvelous type
48:00system that allows you to give you the
48:02correctness benefits that you wouldn't
48:04get from English, yet
48:06it gives you that more intuitive way to
48:09express and compose things. I feel like
48:13Effect is converging
48:14towards something that's the best of both
48:16worlds here, that is
48:18actually a wonderful substrate
48:20for AIs to also compose
48:22systems out of.
48:24Wish that the latest versions of LLMs
48:26have even better
48:27understanding of the latest developments
48:28in Effect. But I think you mentioned that
48:31you have a nice hack, which is to
48:33clone the Effect repository in cursor and
48:36then have it look at
48:38that while you're coding.
48:40Yeah, so that's actually a shout out to
48:42our friend Tim
48:44Suchanek. He taught me about
48:48this little trick, and I've since applied
48:50it many times also for other projects.
48:52I think we're currently in this uncanny
48:54valley, where LLMs always
48:57lag behind on the latest and
48:59greatest of actual information. The
49:02Effect docs, for example, in
49:04particular, they have really
49:06advanced a lot, got a lot better over the
49:09last couple of months. And
49:10those LLMs that we're using
49:13on a daily basis have not fully caught up
49:15to that level yet, and
49:17are also sometimes trained
49:19on old code snippets, different syntax,
49:22etc. And so when you ask
49:24them, you might get an outdated
49:25response. So what Tim is doing, and since
49:29he's taught me about
49:31this as well, is basically
49:33check out the Effect code base on your
49:36computer locally, open
49:38Cursor in it, and then
49:40Cursor uses the full Effect repo with
49:43all the examples, tests,
49:44etc. as the context. And then
49:46inside of that, you use the Cursor chat
49:49to ask your question, and
49:51you're going to get a really
49:52useful result. So I've been using this a
49:55bunch myself, but we're
49:57also working on some exciting
49:59improvements to the Effect docs, etc.,
50:02where we're trying to
50:04combine the latest and greatest
50:07documentation with some of our own AI
50:11developments as well. So stay tuned for
50:13that. You won't need
50:14those little tricks for too much longer.
50:18So Michael,
50:19thank you so much for taking
50:21out so much of your busy day to share
50:24about your experience and journey with
50:26Effect. Really greatly
50:28appreciate that you share all of that
50:31with the entire audience
50:32here. Before rounding this out,
50:35is there anything else you would like to
50:37share with the folks
50:38listening? I'm sure that you're
50:42very busy building a lot of things, but
50:44you're also interested in
50:46bringing on more engineers.
50:49Anyone in particular who
50:51should reach out to you?
50:53Yeah, so we are constantly hiring and we
50:57are obviously, it's a full TypeScript
51:01Effect code base, but we are also very
51:04much looking for
51:05engineers with an ML background.
51:07And we would warmly welcome them even if
51:11they don't have
51:12specific TypeScript experience.
51:14So yeah, if you're interested in working
51:18on such a code base, which I
51:20think is really, really nice
51:22and working on some really interesting
51:24LLM related topics,
51:27then absolutely reach out.
51:30Yeah, I couldn't give you a warmer
51:33recommendation to work at
51:35Markprompt. Michael, Marie,
51:38Elliot, all the folks working at
51:40Markprompt are A plus humans. If I
51:43wouldn't be doing all of the
51:45things I'm doing, I'd strongly consider
51:47working with you all as
51:49well. So whoever gets a chance
51:50to work with you, I think you're making a
51:53very smart decision. So
51:55now moving back my focus to
51:57you, Michael, thank you so much again for
52:00taking time, sharing all of your
52:03experience and all of
52:05those anecdotes with us. I greatly
52:06appreciate it. Well, thank
52:08you, Johannes. Always a pleasure
52:09to speak with you. Awesome.
52:12Thank you so much. Thank you.