Scaling Voice AI at MasterClass with Effect & TypeScript
#3: Scaling Voice AI at MasterClass with Effect & TypeScript
In this episode Johannes Schickling had a conversation with David Golightly, Staff Engineer at MasterClass, to explore how his team built Cortex – a real-time voice AI orchestration layer that powers personalized conversations with celebrity instructors like Gordon Ramsay and Mark Cuban.
In this episode of Cause & Effect, Johannes Schickling had a conversation with David Golightly, Staff Engineer at MasterClass, to explore how his team built Cortex – a real-time voice AI orchestration layer that powers personalized conversations with celebrity instructors like Gordon Ramsay and Mark Cuban.
Song: Dosi & Aisake - Cruising [NCS Release]
Music provided by NoCopyrightSounds
Free Download/Stream: http://ncs.io/Cruising
Watch: http://ncs.lnk.to/CruisingAT/youtube
- (00:00) - Intro & David’s background
- (04:56) - Discovering Effect & early impressions
- (08:32) - Why RxJS wasn’t enough for Cortex
- (16:15) - MasterClass On Call
- (19:10) - Building the orchestration layer
- (25:30) - Incremental adoption of Effect at MasterClass
- (31:43) - Text-to-speech component
- (40:08) - Error handling, observability, open-telemetry
- (01:01:20) - Looking ahead: Effect 4.0 & the future
- (01:08:00) - Closing thoughts
Transcript
00:00
When I started looking
00:01
into Effect, I started seeing,
00:03
Well, this has a lot of promise and
00:05
potential if you use
00:07
it in the right way.
00:08
What I mean by that is you don't have to
00:10
necessarily go from 0%
00:12
to 100% overnight.
00:14
You don't have to completely
00:16
redesign the entire application
00:18
to take advantage of some of the benefits
00:20
that Effect has to offer.
00:22
That's how I started, was incrementally
00:24
adopting Effect in
00:26
parts of the code base.
00:27
And this was really
00:28
what enabled it to work.
00:33
Welcome to Cause & Effect,
00:35
a podcast about the TypeScript library
00:37
and ecosystem called Effect,
00:40
helping engineers to build
00:41
production-ready software.
00:43
I'm your host, Johannes Schickling, and
00:44
I've been building with
00:45
Effect for over four years.
00:47
With this podcast, I want to help others
00:49
understand the powers and
00:51
benefits of using Effect.
00:53
In this episode, I'm talking to David
00:55
Golightly, who's a Staff
00:56
Engineer at MasterClass.
00:58
In this conversation, we explore how
01:00
David has built Cortex,
01:02
MasterClass's AI voice chat system,
01:04
leveraging Effect streams heavily
01:06
to build a cutting-edge AI
01:08
experience. Let's get into it.
01:10
Hey David, so great to have you on the
01:12
podcast. How are you doing?
01:14
Doing great. Thanks for having me.
01:17
I'm really excited. The two of us had the
01:19
pleasure to now meet in person twice in
01:22
the course of the last half a year.
01:24
The first time we've met in person was at
01:26
the Effect TypeScript
01:28
Meetup in San Francisco.
01:30
And then a couple of weeks ago, we had
01:33
the pleasure to meet again in beautiful
01:35
Italy in Livorno, where you've also given
01:38
a talk about Cortex and what you're
01:41
building at MasterClass.
01:43
And it was such a pleasure to spend a
01:46
bunch of time together and talk about all
01:48
things TypeScript related beyond.
01:51
You also have some beautiful music gear
01:53
in the background. But today we're going
01:55
to talk about how you're
01:57
using Effect at MasterClass.
01:58
So would you mind introducing yourself
02:01
and sharing your background?
02:04
Yeah, sure. My name is David Golightly. I
02:07
am a Staff Engineer at MasterClass.
02:10
And I've been in the industry building
02:12
mostly web applications for
02:14
about almost 18 years now.
02:18
I've also dabbled in some mobile
02:20
applications, some back end, some other
02:23
embedded applications over the years.
02:25
But my main focus has been first
02:27
in the JavaScript world and then more
02:29
recently in the TypeScript
02:30
world for the last several years.
02:32
My actual background, though, I don't
02:34
have a Computer Science degree. My
02:36
education is in music.
02:38
And I grew up
02:40
surrounded by pianos.
02:41
My father is a piano technician and also
02:45
a piano professor who plays
02:47
classical, you know, everything you could
02:50
think of classical music.
02:52
He has played it. You know, he's played
02:54
with orchestras and he's, you know,
02:56
played solo concerts and chamber music.
02:58
And so forth. So just steeped in
03:02
classical music growing up in pianos and
03:04
also in repairing pianos, which I found
03:07
somehow translated
03:09
into programming sort of.
03:12
You're doing a lot of really repetitive
03:13
tasks. You have 88 keys. You have 230
03:16
some strings on a regular piano.
03:19
So there's a lot of repetitive tasks and
03:21
you quickly figure out ways to work more
03:24
efficiently when
03:25
you're doing piano repair.
03:27
So later on, after I got my degree in
03:29
music, I found that it wasn't well, well,
03:32
music is a passion of mine and it's
03:34
something that I still do.
03:35
It's not something I want to try to make
03:37
a career in. And so I quickly discovered
03:40
that, well,
03:41
programming is just right there.
03:43
And I actually enjoy doing it somewhat.
03:45
And so I devoted some energy into
03:47
learning how to do that professionally.
03:50
And, you know, all these
03:51
years later, here I am.
03:52
That is awesome. And such a funny
03:54
parallel since we never talked about
03:57
this, but my mother also
03:58
happens to be a professor in piano.
04:01
And so I likewise also grew up surrounded
04:05
by pianos and played the piano for quite
04:09
a while, but then also found myself more
04:12
attractive to technology.
04:14
But I think this is a fairly common
04:17
overlap of people who had their start in
04:22
music or a different kind of art and then
04:25
gravitated also towards programming.
04:28
And I think it's just a certain way of
04:30
like how someone's brain works that
04:33
really like there's a couple of people
04:36
come to mind to have professional music
04:38
background who are just like absolutely
04:40
brilliant in the engineering field.
04:43
So that's a very funny parallel. But
04:46
keeping it a bit more focused on like
04:48
Effect, what has brought you to Effect?
04:50
How did you first learn about it? And
04:53
what did you think about it when you
04:55
saw it the first time?
04:56
Yeah, so I was not necessarily going to
05:00
adopt Effect for the work I was doing,
05:02
say, maybe over a year ago,
05:05
which was more React focused.
05:08
It wasn't something that I was
05:08
necessarily considering.
05:10
You know, I think React is a
05:12
great framework and it totally
05:13
transformed how we do
05:16
front end development.
05:17
But it's not without its problems in
05:20
terms of state management and so forth.
05:21
But I was mostly happy with it for the
05:23
front end and we try to
05:24
keep things lightweight there.
05:26
But then I got asked to work on this new
05:28
project Cortex, which I'll
05:29
talk about a little bit more.
05:31
And that was going to be a server side
05:33
application that was also not a
05:35
conventional API server.
05:36
Instead, it was managing a lot of async
05:38
events, a lot of open WebSockets, more
05:42
than one of like several open WebSocket
05:44
WebSocket connections that
05:46
it needed to then coordinate.
05:48
So I was looking at a proof of
05:51
concept that somebody else at our company
05:53
had built that was really, you
05:55
know, what we call async hell.
05:57
That is just lots of callbacks, lots of
06:01
event listeners passing
06:03
around references to random things.
06:05
And we're starting to try to build into
06:07
it observability and error handling and,
06:11
you know, more advanced interruption type
06:13
features that involve coordinating the
06:15
state of multiple WebSockets.
06:17
And it just wasn't going to work.
06:18
It was not going to scale.
06:20
The implementation was just going to get
06:22
increasingly tangled.
06:24
And so that's when I started looking
06:26
around for, well, what is
06:27
the state of the art here?
06:28
I had used RxJS in the past.
06:30
And I think that, you know, for people
06:32
who've used RxJS, when they see Effect,
06:34
that's what their first thought is.
06:36
Oh, this is RxJS.
06:37
I've seen this before.
06:38
You know, this is this is familiar to me.
06:39
It's just another RxJS kind of knockoff.
06:42
I didn't really find it to be the case,
06:44
though, for several reasons.
06:46
But I think that what really caught my
06:48
eye with Effect was the
06:51
really active community.
06:53
Because when I'm looking to adopt a new
06:55
framework that is going to form the
06:58
backbone of how we build an application,
07:00
I don't want to have to
07:01
be inventing stuff out of whole cloth
07:03
I don't want to have to
07:04
resurrect things from a couple of
07:06
examples that don't
07:07
really apply to my application.
07:10
I don't want to be in
07:12
the dark and be on my own.
07:13
And I especially don't want to ask my
07:15
teammates to accept a framework where
07:18
they don't have the proper
07:20
documentation or the guidance.
07:22
They don't know where to find help.
07:23
And so really, I found that the community
07:25
seems to be Effect's biggest asset.
07:28
And just how helpful everybody is and how
07:30
enthusiastic everybody is
07:32
really ease the adoption process.
07:35
That is awesome to hear.
07:36
And that definitely also reflects my own
07:39
perspective and my own reality.
07:41
This is what drew me
07:42
to Effect very early on.
07:44
The community was much smaller at this
07:46
point and I tried to play a positive role
07:49
in like growing and
07:50
forming that community.
07:52
But it's also something I'm like super
07:54
excited about and super
07:56
proud of that we like together.
07:58
Yeah, broad Effect to
08:00
the point where it is today.
08:02
And it attracts more
08:03
like brilliant minds.
08:04
And it's also interesting that Effect
08:08
attracts a lot of experience engineers
08:10
like yourself, but also new engineers
08:13
that are drawn to
08:15
building something great.
08:17
And that community is such a compelling
08:19
aspect of that as well.
08:21
So maybe to linger on the RxJS part for a
08:25
little bit since yes, quite a few people
08:28
are kind of comparing Effect with RxJS.
08:33
For folks who are familiar with RxJS but
08:35
not yet with Effect, where would you say
08:38
there are parallels and the commonalities
08:41
and where would you say
08:43
things are much more different?
08:45
Well, in RxJS, the idea is
08:47
it's very stream oriented.
08:50
So you essentially are creating
08:52
everything is an observable.
08:56
It can emit events.
08:58
And this is really useful actually.
09:00
I think a lot of programming situations
09:02
can be modeled in this way.
09:04
You have especially the asynchronous
09:06
stuff like really anything side-effectey
09:08
is happening asynchronously.
09:11
You can't necessarily expect it or
09:13
predict it, but then you have to react to
09:15
it and do something in response to it.
09:18
You have user input,
09:21
user loads the web page.
09:22
You didn't know when or why
09:23
they were going to do that.
09:24
They click on a link or they decide to
09:27
watch a video or fill out
09:28
a form or what have you.
09:31
This is ultimately the timing and
09:33
coordination of these things
09:34
and these inputs are driven by humans.
09:36
And likewise, when you have systems where
09:38
you have connections coming from third
09:40
party services or you have other kinds of
09:43
asynchronous behaviors where you have to
09:45
connect users to each other.
09:49
A lot of the hardest parts of programming
09:51
happen when we have asynchronous behavior
09:55
that we have to model.
09:56
And this is why I think you look at
09:59
paradigms like HTTP or more specifically
10:03
like a model view controller that you see
10:05
in Rails that is still so successful.
10:07
And one of the things that that has
10:09
largely done is allow us to architect
10:11
applications around kind of one shot
10:13
actions where you get essentially a
10:17
function call via an HTTP request.
10:20
And your application now has to implement
10:24
the function call and respond with it.
10:27
But it's everything that happens between
10:29
the request and the response is largely
10:32
typically serialized.
10:34
It's sequential. It's
10:35
not in parallel, right?
10:36
Which is great. It makes things a lot
10:38
more easy to reason about.
10:40
You know, things happen in a sequence A,
10:42
B, C, D, E and then you're done.
10:44
But most of what we have to deal with,
10:47
especially in building web interfaces or
10:50
building applications like
10:51
Cortex, is not like that.
10:53
Because you have a lot of things that can
10:54
happen and you might be in the middle of
10:56
one thing and then something else happens
10:57
that you didn't expect.
10:58
And now you have to deal with that. And a
11:01
lot of this also involves kind of keeping
11:03
state updates going.
11:05
This is why honestly, this is why React
11:07
superseded jQuery is because they're
11:09
really good at this.
11:10
But getting back to your question about
11:12
RxJS, RxJS is kind of an even more next
11:16
level approach at this where you can
11:17
model everything as an observable.
11:19
You don't know when it's going to happen,
11:21
but you know that you might receive one
11:23
or more different kind of events.
11:25
And then how do you combine events from
11:28
one observable with events from another
11:30
observable when they need to interact to
11:32
produce some sort of output.
11:34
And RxJS is really built
11:35
around this sort of model.
11:38
The thing is though, RxJS is also trying
11:41
to be not just RxJS, but it's a reactive
11:45
framework that is cross-platform.
11:48
So you have the same APIs more or less
11:51
that are designed for JavaScript or
11:56
TypeScript that are then also ported over
11:58
to Java or ported over to Swift or ported
12:01
over to Kotlin or ported over to
12:03
whatever other language or framework.
12:06
And so I think there was an intentional
12:08
desire on the part of the designers there
12:11
to keep things language agnostic in their
12:14
designs, which in my opinion is kind of a
12:17
flaw because it means that it's
12:20
yes, you can transfer that knowledge
12:21
from one language to
12:23
another pretty easily.
12:24
If you know reactive programming, you can
12:26
do it in any language.
12:27
But it means that you're passing on a lot
12:30
of the strengths of each language that
12:32
you're in in order to make things as same
12:35
as possible between every
12:38
language that you're working in.
12:40
Effect in my mind is taking kind of a
12:42
very different approach, even though it's
12:44
based on the ZIO, which
12:45
is the Scala equivalent.
12:48
It was the inspiration for Effect.
12:50
My understanding is that now, Effect's
12:52
design is pretty detached from needing to
12:55
keep parity with ZIO.
12:57
Effect is basically about TypeScript and
13:00
getting really good at TypeScript, making
13:02
TypeScript the best it can be.
13:05
And there is no, as far as I know, I
13:08
can't speak for the core team, but as far
13:11
as I know, there's no attempt at saying,
13:13
"Well, what if we make Effect
13:15
for Java or Swift or Kotlin?"
13:17
It's not about that. It's just about what
13:21
is TypeScript good at and how can we be
13:23
really good at that.
13:24
So that's a really big point, I think,
13:26
because with RxJS in TypeScript, it never
13:30
felt like they really got how to make a
13:33
TypeSafe reactive stream
13:35
application really work.
13:37
the types
13:37
kind of sort of worked.
13:40
As far as I recall, there wasn't really a
13:42
great story around requirements
13:44
management or even really error handling
13:47
the way there is with Effect.
13:48
And so it lacked a lot of the
13:50
potential, it failed to take advantage
13:53
of a lot of the potential that TypeScript
13:55
has to offer in the way that Effect does.
13:58
Does that answer your question?
14:00
No, it does. And it
14:01
reflects also my perspective.
14:03
In fact, I've been using ReactiveCocoa
14:06
back in the days, and I believe that was
14:09
early on in Swift, and it was quite
14:12
pleasant to use, but it was very
14:14
deliberate for the purposes of streams.
14:19
And there was sort
14:20
of like a peak moment
14:22
where it got really into it.
14:23
But then I felt like a lot of impedance
14:26
mismatch where everything wanted to be
14:29
modeled as a stream, but
14:31
not everything is a stream.
14:33
Some things are just like a one-off
14:35
thing, and you just do an HTTP request
14:39
and you get it back.
14:41
Ensure that might fail, and eventually
14:43
you're just interested in one value, but
14:45
you kind of model your tree
14:47
tries, etc. as a stream, etc.
14:50
So this was my first exposure
14:52
really to this mindset, but it felt like
14:56
the gravity was too hard on
14:59
this very specific primitive.
15:02
And this is where I would rather like
15:05
create more distance between Effect and
15:07
RxJS, because Effect is not trying to
15:11
tell you, "Hey, everything is a stream."
15:13
No. Effect gives you a stream abstraction
15:16
that lets you use the stream abstraction
15:19
when you have a stream.
15:20
For example, when you use a WebSocket or
15:24
when you use something else, maybe you
15:26
want to read a file and use streams
15:29
through the bytes, etc.
15:30
This is where streams are a great use
15:32
case, but if you just want to do a
15:35
one-off thing that happens to be
15:37
asynchronous, then you don't need to be
15:39
forced that this is the
15:40
stream mindset, but this is where Effect
15:43
gives you different primitives.
15:44
And I think this is kind of the huge leap
15:48
beyond what RxJS gives you, and I think
15:51
this was always like the big caveat for
15:53
RxJS that not everyone buys into the
15:57
"everything is a stream" mindset.
16:00
And this is where Effect is much more
16:02
applicable to any sort of circumstance
16:05
where programming
16:06
languages is applicable to.
16:08
Very interesting to hear your perspective
16:11
on this. You've been
16:12
mentioning Cortex now a few times.
16:15
So before we dive into how it's built
16:17
specifically and how it leverages Effect,
16:20
can you motivate what Cortex is and what
16:24
role it plays within MasterClass?
16:27
Right, yes. So Masterclass has been
16:30
developing over the last year plus our
16:34
own in-house voice AI chat that has got
16:38
kind of a MasterClass twist to it,
16:39
and that you're not talking with
16:41
fictional, invented characters or
16:44
anonymous service bots, but we got
16:48
authorization from many of our
16:50
instructors who are working closely with
16:52
us to essentially clone
16:54
them into an AI persona.
16:57
This includes an LLM that is trained on
17:01
their ideas, their writings, their public
17:05
speaking, and it also
17:06
includes a voice clone of them.
17:08
And in some cases also, we have some
17:11
chefs like Gordon Ramsay is being
17:14
released soon, and we have a lot of his
17:16
recipes that are included, and he can
17:18
walk you through
17:18
making one of these recipes.
17:20
So it's a really interesting, interactive
17:23
way that I think enables MasterClass
17:26
instructors to make themselves really
17:29
available to a much broader audience than
17:32
they would be able to on
17:34
their own as individuals.
17:36
And so this has also presented an
17:38
enormous technical challenge because I
17:40
think when you are talking to an
17:43
anonymous AI, well, you might
17:45
be a little bit more forgiving.
17:47
You know that it's a bot that you're
17:48
speaking to on some level.
17:49
But with these real people who are very
17:54
invested in the integrity of their image
17:56
and their public persona, the bar is much
18:00
higher, I think, in terms
18:02
of really representing that.
18:03
And this is also like if you sign up and
18:05
you try to interact with Gordon Ramsay,
18:06
well, if you're a fan, how many times
18:08
have you seen him on TV?
18:10
You can watch the MasterClass courses,
18:12
you know, it's also
18:12
like more than an hour of content.
18:15
You know what he sounds like, you
18:16
know how he speaks, you know what he's
18:19
passionate about, what he doesn't like.
18:21
You know, this personality is something
18:24
that you've probably learned already. And
18:25
now we have to put that into an AI agent
18:28
or an AI bot that you can talk to.
18:31
And so I feel like the very similitude has
18:34
to really be there. And so that's been
18:36
really challenging just to kind of get
18:38
through the uncanny valley stage of that.
18:41
And so this is the background of what
18:43
we're building. If you want to try it
18:45
out, you can go to
18:46
oncall.masterclass.com, all one word
18:51
oncall, O-N-C-A-L-L, dot masterclass.com.
18:54
And you can sign up and try it out.
18:57
So this is the general product that we're
18:59
building. And we have a whole LLM team,
19:01
you know, a machine learning team that is
19:03
working on getting the voice models and
19:05
the LLM models really trained up to
19:08
represent each instructor.
19:10
And what I got brought on to do was
19:12
essentially to create an orchestration
19:14
layer because we have several different
19:17
components in the pipeline in order to
19:21
build this experience.
19:22
First, we have a speech-to-text component
19:26
where the end user's voice of microphone
19:30
stream is sent to a speech-to-text
19:33
service that then listens to the user's
19:36
speech and derives transcripts
19:38
from it of what they've said.
19:41
And then at certain intervals when we
19:43
believe that the user's done speaking and
19:45
is ready to hear what the AI has to say,
19:47
which is not a trivial problem, by the
19:49
way, we then send that transcript to the
19:52
LLM to generate a response.
19:55
And then the LLM generates text response,
19:57
and then we send that off to a text-to-speech
19:59
service to generate the audio for
20:02
that response in the instructor's voice
20:05
and then send that back to the user.
20:06
So we have several different
20:08
asynchronous services, the
20:10
speech-to-text and the text-to-speech
20:12
components of which are WebSockets.
20:15
And then the LLM is a HTTP streaming
20:19
connection essentially wrapped in an
20:22
async iterable in the JavaScript world.
20:25
So this is coordinating
20:28
between these different elements.
20:30
But as you can see, there's kind of a
20:31
step one, step two,
20:32
step three to this process.
20:34
What makes it kind of interesting is, as
20:36
I mentioned, we don't know if
20:37
the user is done talking yet.
20:39
The user can kind of start
20:40
talking again at any time.
20:42
What we want to do is shut down anything
20:44
the bot is doing at that point because
20:46
the user is supreme.
20:48
The user drives the conversation.
20:50
We don't want the bot talking over the
20:52
user when the user is
20:54
trying to say something.
20:55
And so it's really crucial that we are
20:58
very responsive to that and stop whatever
21:01
is in progress downstream, whether it's
21:04
at the LLM stage or
21:05
the text-to-speech stage.
21:06
And so this is
21:07
essentially what Cortex is doing.
21:08
Cortex is a Node.js application that is
21:12
connected to bi-WebSockets from our two
21:15
clients that we have, Web and iOS
21:18
clients, that connect
21:19
to Cortex over WebSocket.
21:20
And Cortex, in turn, provides WebSocket
21:24
interface that abstracts these multiple
21:27
services and receives the microphone
21:31
audio stream from the user
21:32
and emits back bot audio events.
21:35
So from the client's perspective, the
21:38
contract is simply send the user's
21:42
microphone audio to Cortex and receive
21:44
back the bot audio and then the clients
21:48
have to actually play
21:49
it through the speakers.
21:50
And that's kind of the, you know, the
21:52
whole the whole contract.
21:53
There's a couple of other details in
21:55
terms of, you know, if a tool call
21:58
happens or, you know, if state change
22:01
happens or something like that.
22:03
But that's the main gist of what that
22:06
WebSocket contract to
22:08
Cortex looks like under the hood.
22:10
However, Cortex has to orchestrate the
22:13
various services that I talked about,
22:16
execute tool calls and and do a whole
22:18
bunch of other, you know, state
22:20
management under the hood.
22:22
And so that's what we built with
22:22
Effect. It's 100 percent.
22:24
I mean, 100 percent. But like it's it's
22:27
pretty much the every piece of it is
22:29
built in Effect or
22:30
using Effect in some way.
22:32
That's awesome. So you've mentioned that
22:34
initially when you got into this project,
22:37
you got a first like proof of concept and
22:39
where it really like exposed you to all
22:41
of the complexity that's for solving this
22:45
actual product to
22:46
implement this actual product.
22:48
This complexity need to be tamed. Did you
22:51
work off that initial proof of concept or
22:54
did you rather roll
22:55
something from scratch?
22:57
And how did you go about that?
22:59
That's an interesting question because
23:00
I'm always trying to figure out how to.
23:04
Like re-architect a
23:06
piece of software, you know.
23:08
It's something you don't want
23:09
to have to do very often.
23:11
Also, I don't know, actually, if it's
23:13
me or just the stage of career I'm
23:16
at and the kind of problems that I get
23:18
brought in to help with.
23:19
But I often find myself looking at a
23:21
piece of software that is
23:22
in need of re-architecting.
23:24
I think it's largely because I was
23:26
specifically asked to
23:28
re-architect it in this situation.
23:30
But the proof of concept was interesting
23:33
because, you know, it was built by our
23:37
project lead on call, who
23:39
is now our VP of architecture, who's
23:42
just a brilliant person who has no
23:44
experience really with
23:45
JavaScript or TypeScript.
23:47
And asked chat GPT to help. I think it
23:50
was chat GPT to help with building it and
23:53
essentially just piece that all together
23:55
from chat GPT prompts.
23:58
There were a couple of things that came
24:00
out of that that were, for example, a
24:02
library that is like a private library
24:05
that isn't really maintained.
24:06
There wasn't really a
24:09
top level architecture.
24:11
you had individual pieces that
24:12
were kind of worked in isolation, but
24:13
then the glue between them became really
24:16
convoluted and hard to follow.
24:19
So so that that kind of became a problem.
24:21
But it worked. You know, ultimately, this
24:23
proof of concept kind
24:24
of worked for a demo.
24:27
It didn't handle interruptions very well.
24:30
It didn't have great latency.
24:32
I think in reality of like most
24:35
production grade systems are actually let
24:39
me take that back, probably not
24:40
production grade, but applications,
24:43
JavaScript applications that are running
24:44
in production are probably
24:47
That sort of state where is working well
24:51
enough on the happy path, but all of like
24:54
those higher level requirements that
24:57
really make a end user experience superb
25:00
that you can interrupt an AI model when
25:05
when it's talking to you or that's more
25:07
resilient to errors, etc.
25:09
This is where you need to go beyond that
25:11
convoluted everything
25:13
kind of pieced together.
25:15
And I think this is like a perfect point
25:17
to to start with, like
25:20
applying Effect to to go beyond that.
25:23
That's sort of like a messy state.
25:27
Yeah, this is why I wanted to
25:31
tame the asynchronous
25:32
behaviors in in Cortex.
25:35
And I was looking for a package that
25:37
would help me do that. And
25:38
this is when I found Effect.
25:40
And then at that point like I
25:42
said, I had seen RxJS. and I had
25:45
good experiences at
25:46
first building RxJS
25:47
But like kind of one of the common
25:49
complaints is that, well, now, like you
25:52
said, everything has to be a stream.
25:53
Now you have to model
25:54
everything in this way.
25:56
And it really completely changes how you
25:58
architect your software and you need a
26:01
lot of buy in from your team if you're
26:02
going to start doing this, because now
26:04
everybody has to learn this.
26:05
They can't just sit down and, you know,
26:07
start coding on whatever part, you know,
26:09
feature they're trying to do.
26:10
They have to first learn this framework
26:11
and that can be a big ask for folks.
26:15
And so that's something that I wanted to
26:16
be conscientious about.
26:18
However, when I started
26:19
looking into Effect, I started seeing,
26:22
well, you know, this has a lot of promise
26:24
and potential if you
26:25
use it in the right way.
26:27
And what I mean by that is like, you
26:29
know, you don't have to necessarily go
26:32
from zero percent to
26:33
100 percent overnight.
26:34
You don't have to completely redesign the
26:37
entire application to take advantage of
26:39
some of the benefits
26:41
that Effect has to offer.
26:42
That's kind of how I started, was
26:44
incrementally adopting
26:45
Effect in parts of the code base.
26:48
And this was really what
26:49
enabled it to work at all.
26:51
If I had to say, hold on, I need to take
26:54
this for a month and just completely
26:56
build it from scratch and nothing's going
26:58
to work and I can't deliver any new
26:59
features until I'm done.
27:03
That was not going to fly.
27:04
I was in a position
27:05
where I did not have to ask for
27:07
permission to go live in a cave for a
27:10
month or however long
27:11
it was going to take.
27:12
I always like avoid
27:13
being in that situation.
27:15
I don't think that's a good place for us
27:16
to be in as software engineers.
27:18
We need to be shipping like every day,
27:20
shipping features, shipping bug fixes,
27:22
shipping products constantly.
27:25
And like we never have the luxury to just
27:28
say, well, hold on, I'm just going to
27:30
like spend a month not shipping, you
27:32
know, to do this technical thing that I
27:34
can't really explain
27:35
why I have to do this.
27:36
Right. Nobody, nobody likes that.
27:38
So fortunately, Effect is not
27:41
a framework that is like that.
27:43
That's not what was
27:44
asked of us to adopt Effect.
27:47
Instead, it was you can start plugging in
27:50
Effect judiciously here and there.
27:52
And I started at the top level of the
27:54
application around really around the
27:56
WebSocket connection layer, around the
27:58
HTTP layer and the application runtime.
28:02
And that allowed us to kind of have a top
28:05
level of error handling that immediately
28:09
provided benefits for recovery and so on.
28:13
But we didn't stop there.
28:15
That was just the first step.
28:17
We started kind of rebuilding different
28:20
parts of the application.
28:21
Some of them were like isolated behind a
28:24
run promise or, you know, like a stream
28:26
that was, yeah, emitting
28:28
events into an event emitter.
28:30
But it allowed us to kind of rebuild
28:32
isolated parts of the application on
28:35
their own where like selectively where we
28:37
thought they had the most value to be
28:39
converted into Effect.
28:40
And often driven by product requirements.
28:44
So when I wanted to implement
28:45
interruptions, that was the reason to now
28:49
rebuild the speech-to-text component in
28:52
Effect, because it was going to be a lot
28:54
more work to do that.
28:56
If I was going to have to do it
28:58
the old way, it was like, really, it
29:00
wasn't like rebuild the whole thing
29:02
in Effect, the entire application.
29:05
Or just keep working the old way. It was
29:07
really for each isolated piece.
29:09
It was like, do I want to like add more
29:11
spaghetti to this application in this
29:13
component or can I just rebuild this
29:15
piece in Effect and make it much more
29:18
expressive and elegant while building the
29:20
feature that I'm being asked to build.
29:22
So it was much less of a refactoring
29:26
process and more of a incrementally
29:29
adopting it while also building and
29:31
delivering features.
29:32
I love that. And I think you've hit on a
29:34
couple of really nice points here.
29:36
One is that Effect, even though you get
29:40
a lot of benefits once you have
29:43
Effectified more and more of your
29:45
application, you're going to find that
29:46
you can delete a lot of code.
29:48
Everything just fits
29:49
nicely together in the same way.
29:51
If you incrementally adopt React, let's
29:54
say we're just in this transition of like
29:56
transitioning a larger code base from
29:58
jQuery, something else to React.
30:00
You don't need to do that in one like
30:03
long night, but you
30:04
can do that step by step.
30:06
And once you've reached a point where
30:08
everything is in React, then you can
30:10
delete a whole bunch of like glue code, a
30:13
whole bunch of like things that bridge
30:15
from the old way to the new way.
30:17
But it is also totally fine that you go
30:20
through this like transitionary phase.
30:22
But the other thing that you've mentioned
30:24
that I think is like a super great
30:26
insight, which is like, how do you
30:28
prioritize what to refactor with Effect
30:31
when or when to rethink
30:33
something and apply Effect?
30:35
And this is so elegant that you can like
30:38
go top down from like, hey, what is the
30:41
thing that we want to improve for users?
30:43
What is like the business outcome that we
30:46
want to affect here?
30:48
And then like looking at those, Effect
30:51
typically has
30:52
something in store for that.
30:54
If you want to improve performance, if
30:56
you want to improve reliability,
30:58
resilience, error handling, whatever it
31:01
might be, Effect typically
31:02
has something in store for that.
31:05
And then you can prioritize what to work
31:07
on depending on your end user needs.
31:10
And hearing here about the speech to text
31:13
aspect and applying Effect for that,
31:17
that sounds super interesting.
31:19
So I want to hear a little bit more like
31:20
zooming into from macro into a bit more
31:23
micro into this component.
31:25
Can you give a rough overview of like how
31:28
that thing was built before?
31:31
What you diagnosed the biggest things to
31:35
improve where and which primitives of
31:38
Effect did you use to
31:40
ultimately implement it?
31:42
Well, actually what I prefer to do is
31:44
zoom in on the text-to-speech component.
31:47
Oh, I'm sorry.
31:49
I remember that wrong.
31:50
But yes, no, it's okay.
31:51
You remembered it right.
31:53
It was you.
31:53
I said speech-to-text earlier, but I feel
31:55
like the text-to-speech component.
31:58
This is the component that I also talked
31:59
about in my Effect Days
32:01
talk and did a deep dive on.
32:03
I feel like the text-to-speech component
32:05
was kind of like a real unlock, really an
32:09
aha moment of like, wow, this is this is
32:12
what Effect can do in
32:14
terms of like saving
32:15
a lot of complexity from your code base.
32:18
As I mentioned, this is a
32:20
component where like it's a WebSocket and
32:22
we stream LLM thoughts to it.
32:26
So an LLM response, as you might be
32:28
familiar, it sends what are called chunks
32:32
in a chat completion response.
32:34
And the chunks are usually a token or two
32:36
tokens and they're not
32:37
complete words a lot of the time.
32:39
So then we're like accumulating the
32:42
chunks into basically what we call
32:44
coherent thoughts and the coherent
32:47
thoughts can then be sent to the text-to-
32:50
speech component to
32:51
generate the bot audio.
32:53
However, if there's an interruption, we
32:55
need to shut down the LLM and we also
32:57
need to shut down the
32:58
text-to-speech component so that we don't
33:01
continue to generate more thoughts based
33:03
on the previous thing that the user said
33:05
before they continued talking.
33:07
And now we want to start over and respond
33:09
to the most recent thing
33:11
that that user has said.
33:12
So the text-to-speech component now it's
33:16
a WebSocket connection.
33:17
When you send it a
33:19
coherent thought, that connection
33:22
will then respond asynchronously with one
33:26
or more different
33:28
events that you might get.
33:30
And we're basically just
33:31
streaming those up to the client.
33:32
But when there's
33:34
an interruption, we need to actually shut
33:36
down the WebSocket
33:36
connection, close the connection.
33:38
Abruptly, so we don't get any more
33:40
messages from it and then reopen it.
33:43
And then in that period of time, and it's
33:45
not usually very long, it can just be
33:47
like a hundred milliseconds or two
33:48
hundred milliseconds where we're waiting
33:50
for that WebSocket connection to open.
33:52
We hope that
33:52
we've created a connection, but we've not
33:54
yet received the open
33:55
event from the connection.
33:56
And it's in that time that we were often
33:59
getting the LLM was trying to send
34:02
messages to it, but it was erroring
34:04
because we were sending WebSocket
34:07
messages out to a
34:08
WebSocket that was not yet open.
34:10
So we had to now queue those messages to
34:13
wait for the open event from the
34:16
WebSocket connections and then flush the
34:18
queue when it was open.
34:20
So as you can imagine, this created some
34:22
code complexity in the pre-Effect
34:26
implementation and it was something that
34:29
Effect turned out to be actually very
34:31
good at because these are the kinds of
34:32
things that Effect has out of the box.
34:35
In Effect we were able to replace the
34:39
WebSocket handling code with the
34:41
WebSocket utility from
34:43
Effect from effect/platform APIs
34:47
And that has kind of a magical property
34:51
to it that you don't really ever have to
34:53
think about when the WebSocket is closing
34:55
and opening and you don't
34:57
have to wait for an open event.
34:59
What it essentially gives you is what is
35:02
called in Effect a channel.
35:03
And this became a primitive
35:05
that I became curious about.
35:06
It's something that I wish was a little
35:08
bit more first class in the effect world.
35:11
It's certainly used behind the scenes in
35:14
a lot of things like stream and other
35:17
APIs in the Effect world.
35:19
But this is what you get essentially when
35:21
you create a WebSocket connection using
35:23
the effect/platform API.
35:24
But then if you use the Effect stream
35:26
operator pipe through channel, you now
35:28
have a duplex stream, which is one where
35:32
you can start listening to other streams.
35:35
And then instead of doing like a run for
35:37
each or run collector, whatever you're
35:40
doing and you typically do it with a
35:41
stream, you now are piping the events or
35:45
the items in that stream out through the
35:48
channel through the WebSocket connection.
35:50
And then downstream from that pipe
35:52
through channel, you are getting incoming
35:54
events that are coming from that
35:56
WebSocket connection that you can then
35:58
emit further on down your application.
36:01
So this is great. But this is also what
36:04
it is also doing is this is abstracting
36:06
the WebSocket lifecycle
36:08
into the stream lifecycle.
36:09
So if you emit an error
36:12
upstream, it will close
36:14
the WebSocket for you.
36:15
And then if you have a stream.retry, it
36:18
will reopen the WebSocket for you
36:19
in the event of an error.
36:21
And because the streams are pull based,
36:23
you don't have to rely on queuing
36:26
explicitly. You aren't going
36:28
to miss any of your events.
36:30
When the stream is reopened, it will
36:32
start pulling those events
36:33
and behaving as expected.
36:35
So this really allowed us to
36:38
abstract all of the sort of tangled like
36:39
we had, I think, a promise that was a
36:42
reference to a promise that was kept
36:44
around and was
36:44
awaited in different places.
36:45
And it was it was a mess before. And now
36:49
we had a single stream that was just a
36:52
linear stream that where you had the
36:55
stuff going out, going in the top of the
36:57
stuff coming from the stream, coming out the bottom.
36:59
And it became very easy to reason about
37:02
what was happening. And you didn't really
37:03
even have to think about the stream
37:05
lifecycle at all. It was just you made an
37:07
error when you want it to
37:09
close and then just retry.
37:10
The WebSocket
37:11
connection is now handled by the stream
37:14
lifecycle. So you can use
37:15
the stream retry stream.
37:18
will be shut down when the
37:21
scope of the stream ends and the
37:24
WebSocket will automatically closed.
37:26
We also have a flush event that we send
37:30
out to the text-to-speech
37:32
service, which essentially says we're
37:34
done sending you a new speech for now.
37:38
So send us everything you got. And the
37:41
peculiarity of this particular service is
37:44
that they will then accept your flush
37:46
event and they will
37:47
promptly close the stream on you.
37:50
That's not really what we wanted, but I
37:52
don't know. They designed it this way.
37:53
And I don't really have, you know,
37:55
leverage to get them to redesign their
37:57
entire API. I just have to
37:58
work with what we have. Right.
37:59
This is a lot of application development.
38:01
You don't have the liberty to redesign
38:04
every API that you're working with. You
38:07
have to abstract it in some way. And so
38:09
this is what we're having to do here. But
38:10
the Effect stream
38:11
primitives make it really easy.
38:14
That sounds brilliant so far. And I think
38:16
what is so nice about this is
38:18
that it is A very clear and very
38:22
intuitive from a user perspective what
38:25
the system should do.
38:27
And as users, we're like all familiar
38:30
with when this goes wrong and how
38:33
frustrating it is. Like if I'm talking to
38:36
the AI, the AI goes off like in a wrong
38:39
direction that I don't want.
38:41
And I want to interrupt it.
38:42
And it doesn't act on this interruption.
38:45
You need to listen to it for another 20
38:48
seconds until I finally need to repeat
38:50
what I've just said.
38:52
And all of those things, they need to be
38:54
handled. And all of that is a lot of
38:56
complexity. And if you leave that
38:59
complexity unchecked, like you said, it
39:02
was a mess. And I think that nicely
39:04
reflects the majority of JavaScript
39:07
applications that are out there or
39:09
TypeScript applications.
39:10
And I think it's really like this
39:13
fine line to walk where you capture
39:17
all of like the existential complexity
39:20
that your use case requires and shaving
39:23
off like all the accidental complexity
39:26
and isolating this nicely.
39:28
And even to a degree where you say like,
39:30
okay, those services that you need to
39:32
call, they're not in the ideal shape that
39:35
you would like it to be. But then you can
39:37
just like wrap it and create your own
39:40
world that you're happy in and where you
39:43
can model your application in a nice way.
39:45
And yeah, I'm very happy to hear that
39:48
effect streams and the various primitives
39:51
that you've been using. You've been using
39:53
the WebSocket abstraction that Effect
39:56
gives you, and I
39:56
suppose also queues, etc.
39:59
That all of this has been so nicely
40:01
fitting together to model your use case.
40:05
So going a little bit beyond streams,
40:08
which other aspects of Effect have you
40:11
been using or which other kind of
40:13
superpowers have you been getting out of
40:15
Effect that have played a
40:17
meaningful role in the application?
40:19
Well, the error handling has been huge.
40:22
Honestly, we modeled all of our possible
40:25
errors. We have, I think, maybe up to 30
40:29
or so errors that the system can emit
40:31
that are domain specific tagged errors.
40:35
And those are decorated with a specific
40:40
error code and an error source, because
40:43
one of the things that will often happen
40:46
or that was happening originally, I
40:49
think, was, oh, we got an error. The
40:51
connection went down.
40:52
Well, we got an error and something weird
40:55
happened, and I don't know why, and now
40:57
I'm in a weird state. Oh, we got an
40:59
error, and the whole service just crashed
41:03
or something like this, right?
41:04
And even if you can just wrap everything
41:08
in a try catch, best case scenario, you
41:11
have some unrecoverable error, your
41:13
connection goes down, and you don't know
41:16
why. You just know, oops,
41:17
bye, and then all of a sudden.
41:19
And so it's frustrating for the user, and
41:23
it's also frustrating for the rest of the
41:24
team when they're trying to diagnose
41:26
what's going on, which we spent a lot of
41:28
time doing in our
41:29
development workflow internally.
41:31
And, you know, I'm a big fan of passing
41:35
the buck, so I don't like things to be my
41:38
fault. And I think
41:39
we're all in that domain.
41:41
I say this to joke, actually, I'm fine
41:44
with taking responsibility if it's my
41:46
fault, but I would rather things not go
41:48
wrong because of the
41:49
decisions that I made.
41:50
A lot of times early on, it was like,
41:53
oh, Cortex is down. Oh, Cortex emitted an
41:55
error that I don't understand.
41:57
And, you know, fair enough from a
41:58
client's perspective or from a test
42:00
engineer's perspective,
42:01
that's what it seems like.
42:03
But that doesn't really give you enough
42:05
information to troubleshoot because most
42:08
of the time it's not, you know, Cortex is
42:11
just a hub. Cortex is just passing events
42:14
from one service to another.
42:15
It's not Cortex really as the source of
42:18
the errors. Instead, what we see a lot of
42:22
the time is, oh, one of our
42:23
backend services went down.
42:25
Oh, a backend service emitted something
42:26
that we didn't expect. Oh, it's being
42:29
slow or something like this.
42:32
And now we're able to like create
42:34
targeted pinpoint errors whenever a
42:37
specific thing goes wrong
42:38
somewhere in the system.
42:39
And then those propagate up to our top
42:41
level error handling. And so if we have
42:43
something that's unrecoverable that
42:44
happens, we can now close
42:47
the connection like before.
42:48
But now we can send up detailed
42:51
information that allows our people to
42:54
diagnose what's the
42:55
problem. So it's not Cortex.
42:57
It's like, oh, our speech-to-text service
43:00
crashed or is out of memory or something.
43:03
And so now we aren't able to create calls
43:07
until we fix that
43:08
piece of infrastructure.
43:09
So that gives us a lot more information
43:11
that kind of actually saves a lot of time
43:14
debugging the system. It points you
43:16
directly to where the source of the
43:18
problem is instead of making
43:21
you go on a debugging hunt.
43:23
So that has been huge for us. The error
43:26
handling has been extremely valuable.
43:28
There are a lot of other errors that are
43:30
recoverable, but we want
43:31
it to log and report them.
43:33
So the whole error handling story in
43:36
Effect is fantastic, just surfacing when
43:40
things can go wrong and
43:42
forcing you to deal with it.
43:43
It has also meant, interestingly, I feel
43:46
like that, you know, like within Cortex,
43:50
not every function is an effect.
43:51
You know, not every single line of code
43:54
is a yield star. There's a fair amount of
43:57
just plain old, data manipulation
44:00
that is happening throughout the code.
44:03
It's data manipulation using
44:05
functions that are
44:06
synchronous and aren't going to throw.
44:11
Right. You could have very high
44:12
confidence that, you know, if you're
44:14
trying to get an index of a thing from a
44:15
string, you know, or
44:17
whatever, you're not going to throw.
44:19
you can do like a
44:20
lot of just kind of conventional
44:22
programming in areas that
44:23
are kind of safe and sandboxed.
44:25
And it doesn't mean that every single
44:27
calculation needs to be done in an
44:30
effect. It just gives you a safe place to
44:33
do that kind of work without having to
44:36
worry, you know, oh, if this throws or
44:38
oh, if this does something asynchronous
44:39
and I, you know, don't handle it, you
44:42
know, don't await it or whatever.
44:44
You know, usually those those kinds of
44:46
cases are they get a lot of attention
44:48
because we have to
44:50
think about them so much.
44:52
But that's not usually the most of the
44:55
work that we're doing. Ideally, right?
44:57
We're thinking about like, purely
44:59
functional transformations of data from
45:02
one state into another.
45:03
taking the input from
45:05
some kind of asynchronous effect and
45:07
sending it out to
45:08
some asynchronous effect.
45:09
But like the actual program, the business
45:11
logic is usually something that is like
45:13
pretty, you know, step by step, you know,
45:16
just just logic. Is usually when
45:18
we're not interfacing with an external,
45:21
you know, service or
45:22
some kind of side effect.
45:24
Then we can just write code like normal.
45:28
You know, we don't have to model
45:29
everything as a stream just to add up
45:32
some numbers or something.
45:33
Right. And I think that the super plain
45:36
way how you put it just write code like
45:38
normal. I think this is kind of the in a
45:42
nutshell, the difference
45:43
between Effect and RxJS
45:46
Where in RxJS you need to do everything
45:49
as a stream. And in Effect, you can write
45:52
code like normal. And another aspect of
45:55
writing code like normal
45:57
is trading errors as values.
46:00
This is we're all super used to just
46:04
passing around and
46:05
manipulating data. And somehow, we're
46:09
kind of brainwashed into
46:11
thinking that errors need to be like we
46:13
need something we need.
46:14
We're almost like paralyzed about like,
46:16
how should we deal with errors? But if
46:19
we're just trading errors as values as
46:22
well, errors as data and passing them
46:24
around and Effect just makes that easy by
46:26
giving us a separate channel
46:28
to deal with that error data.
46:30
And then, like you say, like you don't
46:33
want to you'd like to pass it along, then
46:35
it's just data that you pass along. So I
46:38
think that's just like code like it's
46:41
normal. I think that is like one of
46:43
Effect's superpowers.
46:45
And closely related to errors is like
46:48
having visibility into when errors happen
46:51
when something doesn't go as expected. So
46:55
and I think if I remember correctly, the
46:57
telemetry part the observability part has
47:00
also been a key aspect of
47:02
building Cortex and operating it.
47:04
So maybe can speak a little bit more to
47:07
how you do observability and telemetry
47:10
usually within MasterClass, particularly
47:14
within JavaScript applications and how
47:17
Effect has helped you to
47:20
maybe improve that even further.
47:22
Right. Yeah. So I sort of have to admit
47:25
that we don't have an excellent story for
47:27
the most part or before Cortex didn't
47:30
have an excellent story about
47:31
observability at MasterClass
47:33
in the JavaScript world.
47:34
We have a number of services.
47:36
We have we have a Raygun. We have New
47:39
Relic and we have Core Logics. We get our
47:42
logs to we send our logs to and we have.
47:44
So we have a bunch of observability
47:46
services for things like video. We have a
47:48
dedicated video monitoring service that
47:51
we integrate and
47:53
a few high value business.
47:55
Applications like that. We want to keep
47:57
an eye on, you know, error rates for
48:00
people visiting our home page or things
48:01
that are really
48:02
indicate business traffic, business
48:05
being impacted by
48:07
some technical problem.
48:08
However, usually that that amounts
48:11
to like something
48:12
that's easily reproducible.
48:13
And easily fixable usually and there's a
48:16
either some infrastructure that needs to
48:19
be restarted or code change that needs to
48:21
be rolled back or something like that.
48:23
Cortex really represents a new level of
48:26
complexity when it comes to understanding
48:28
what's going on internally. And I think
48:29
that a big reason for that is that it is
48:32
not a one shot HTTP server type of
48:35
application, but is instead
48:38
You know, a stream of streams and is
48:41
handling all of these asynchronous events
48:43
that are passing through it. It's not
48:45
directly doing much of any work. It's
48:47
just literally in between all these
48:49
services handing events
48:51
from one service to another.
48:52
So, as I mentioned before, when
48:54
things go wrong, they're mostly not going
48:55
wrong in Cortex. And likewise with
48:57
observability
48:59
where the system is spending time
49:01
doing work, it's mostly not spending time
49:04
inside of Cortex doing that work.
49:07
Cortex is instead, waiting for
49:10
events from other services. And so what
49:12
we're interested in measuring is not
49:14
really the conventional, I think,
49:16
telemetry story when it comes
49:17
to building tracing and spans.
49:20
I think this is a story that is also
49:22
baked into the default Effect telemetry
49:26
story, right? When you have a Effect dot
49:29
with span with a name, you know, and you
49:32
and you and you wrap
49:33
that around an Effect.
49:35
Well, that's great. That ensures that
49:37
that span or that Effect, the time that
49:40
it executes is going to be represented by
49:43
a span in your trace. And for most like
49:46
one shot type of actions that you might
49:48
perform, that works great,
49:51
which is most of the time.
49:52
If you're doing actual work within an
49:55
effect, within a one shot effect, then
49:57
that is the conventional way that you do
50:00
telemetry. We're not really doing that at
50:02
all in our telemetry
50:04
implementation in Cortex.
50:05
Instead, we're interested in measuring
50:07
time between events that are coming from
50:10
outside services. Cortex is the only
50:12
place where we can really gather this
50:13
information because it's the hub.
50:15
But Cortex isn't sitting around. We don't
50:19
have like an effect that is, you know,
50:21
sleeping until it gets a certain
50:23
notification from or an event from
50:25
another stream.
50:26
It wouldn't make any sense
50:27
to build the application that way.
50:28
And so it doesn't really make a lot of
50:30
sense to build our telemetry that way.
50:33
I suppose what we're doing with
50:35
Open telemetry is a little unconventional
50:36
in that a span doesn't represent Cortex
50:40
doing that work. Instead, it represents
50:42
usually really represents like the LLM is
50:44
working or the text-to-speech
50:46
service is working.
50:48
And we're just waiting for that. But it's
50:49
measured from Cortex, not from these
50:52
other services. But because
50:53
it's really all streams.
50:54
What we have to go on isn't an effect
50:57
that we're measuring. It is literally the
51:00
time between two events in those streams
51:03
that we're measuring. And so we really
51:05
had to roll a lot of our
51:07
own telemetry handling.
51:10
But, you know, we were going to have to
51:11
do this anyway,
51:12
ultimately, because when you have.
51:14
Let's say we're not using Effect. We're
51:16
using the original approach, the
51:19
non-Effect approach that is event
51:21
emitters everywhere, you know, WebSocket
51:23
event handlers and so forth.
51:25
You get a transcription from the speech
51:28
to text service and you want to start the
51:31
span of time that it takes that you're
51:34
measuring that it takes to.
51:36
Generate a response
51:37
to that transcription.
51:39
Well, you can have a central hub where
51:42
like maybe you're
51:42
collecting all of these events.
51:44
But you're starting that span in one
51:48
event handler and then you're entering
51:49
you're ending it in a different event
51:51
handler for a different service.
51:53
And so you need a place where you're
51:55
holding on to those references might be a
51:57
centralized location that is listening to
51:59
all of these events.
52:00
But then it becomes kind of tangled up
52:02
because you're having to.
52:04
Keep these references around and
52:06
keep them alive, from one
52:08
event handler to a
52:10
completely different event handler.
52:11
And this is an area where.
52:14
Yeah, we had to roll some of our own code
52:15
in Effect do it this way.
52:17
But I feel like Effect kind of made it
52:19
easier to do it this
52:20
way anyway.
52:21
Allowing us to have a kind of a
52:23
connection level, long lived reference to
52:27
these spans and then just manipulate
52:30
spans in what is essentially a stream dot
52:34
tap where we are listening to all of the
52:37
incoming events and then just
52:38
Starting and stopping them based on
52:40
which events are occurring.
52:42
It's not been perfect, honestly.
52:45
It has been a little error prone
52:46
sometimes and we've had to go in and kind
52:49
of tweak things when we
52:53
have unexpected behaviors.
52:55
It's an area that has provided immense
52:57
value for us, however.
52:59
It's given us a lot of insight into what
53:01
people are experiencing
53:02
if people are experiencing.
53:04
Really slow behaviors, slow responses.
53:08
If people are experiencing the bot is
53:10
talking over me or this sort of thing.
53:12
If we have errors somewhere in the
53:14
system, we can see exactly where and when
53:16
that happened and in what service and in
53:18
what part of that
53:19
services work it happened.
53:20
And we're able to trace, you know, what
53:24
was the sequence of of chunks
53:26
that were emitted by the LLM.
53:28
You know, how long did it take
53:29
for us to get that first
53:30
chunk out of it out of the LLM.
53:32
You know, comparing the user message in
53:35
the bot response and you know, if the
53:37
user interrupted the bot, how much of the
53:39
bot speech did the user here?
53:42
And so a lot of these questions that are
53:44
really of interest to the business and
53:46
also for us technically.
53:48
When it comes to how on call is being
53:51
used and how people are experiencing it
53:54
are really answered by the telemetry that
53:57
we've built using
53:58
Opentelemetry and Effect.
54:00
But,it's a very
54:02
custom system that I I don't know that it
54:05
has its optimal form yet.
54:08
And I also don't know that is necessarily
54:10
going to apply to
54:12
everybody in the same way.
54:14
I don't know. This is like I said, it's
54:15
it's it's a very custom system that is
54:18
built for our use case that will not
54:21
apply in most conventional applications.
54:24
But I think that's OK.
54:25
There's always special cases.
54:28
This makes sense. And, where
54:30
Opentelemetry or the previous
54:32
technologies that is based on Open
54:35
Sensors and Open Tracing, I believe those
54:40
were the two predecessor technologies
54:42
that merged into Opentelemetry.
54:44
Where they're historically coming from is
54:46
like from distributed tracing. And that
54:49
is typically in a
54:50
microservice kind of architecture.
54:52
We have one service request response
54:54
style calling into another
54:56
one, calling into another one.
54:58
So I think where those systems or this
55:00
technology shines historically, at least,
55:02
is on a request response pattern where
55:06
you just get that burned on charge that
55:08
that you know from like a
55:09
performance profiler in a single threaded
55:12
environment or a multi-threaded
55:13
environment, now we get it
55:14
over like a network boundary.
55:16
So this is where those shine. But going
55:19
beyond that for different modalities,
55:21
like for long running streams or I've
55:24
been also experimenting with using
55:26
OpenTelemetry in a front-end setting
55:28
where like a front-end session, you don't
55:31
have that request response, but you have
55:33
a front-end session.
55:34
For example, think about how you use
55:36
social media. You might be doomscrolling
55:39
for a very long time. So is your entire
55:42
session, is that the trace that you have
55:45
with possibly like thousands of spans?
55:49
Or where do you make cut basically? How
55:51
do you design your trace and your spans?
55:54
I think that is still something that
55:55
we're figuring out as an industry.
55:57
And it's been cool to hear about your
56:01
usage of it. And I think this also speaks
56:03
to the flexibility of Effect. Yes,
56:06
they're like default patterns make it
56:09
really easy and kind of trivial to
56:11
instrument your app out of the box.
56:13
But if you want to instrument it in a way
56:15
that's a little bit more specific to how
56:18
you would like to give your trace
56:21
meaning, that's possible as well.
56:23
Maybe taking a step back here for folks
56:26
who are curious about Opentelemetry and
56:29
like generally see the value in
56:31
observability, but maybe haven't taken
56:33
that leap yet themselves instrumenting
56:36
their app, which sort of guidance would
56:38
you offer to people what to focus on,
56:41
maybe what to initially leave out of the
56:44
picture just to get going?
56:46
Oh, that's a really good question. I feel
56:47
like the answers are going to be really
56:49
specific to your use case. And in the
56:53
case of an application like Cortex,
56:55
extremely custom. And we have spent a lot
56:58
of time refining and iterating on our
57:02
telemetry implementation. But most
57:04
applications probably
57:05
don't need that, honestly.
57:07
I think, especially in the
57:09
JavaScript world, there's both browser
57:11
and Node based auto instrumentations that
57:15
are available that do a lot out of the
57:17
box. So I feel like a lot of what I would
57:21
want to start with are the ends of the
57:23
application when your code is calling out
57:26
to another service or when you receive
57:28
events from the user.
57:30
Because that kind of determines the shape
57:32
of that user session or that interaction
57:34
that you might be
57:35
interested in measuring.
57:37
And then it will identify kind of
57:39
hotspots of like, oh, the user waited a
57:42
long time for this response or whatever,
57:44
what's up with that? And then you can
57:46
start to drill further.
57:48
And then the other thing that I think is
57:49
really super valuable is distributed
57:53
tracing where you are propagating a trace
57:56
across service boundaries. And sometimes
57:58
this is just, you know, you're
58:00
instrumenting in your browser application
58:02
or in your iOS application.
58:04
you're making calls out to your API service and
58:07
you want to see what's going on in the
58:09
API service as well during that time
58:11
period. You're propagating the trace from
58:14
your client to your server so that when
58:17
you see them all together,
58:20
you can kind of piece together.
58:22
Oh, the client called out to the server
58:24
and then the server made these database
58:25
calls and you can see that all in one
58:27
distributed trace. That's really super
58:29
valuable. So just focusing on the ends,
58:33
either incoming events or outgoing
58:36
service calls, and then making sure you
58:39
have your distributed trace propagation
58:41
set up correctly. Those would be the top
58:43
things I would recommend.
58:45
Right. I agree. And I think the benefits
58:48
of having a good observability story
58:50
for your application and for your system
58:52
is so manifold. Like it helps you with
58:56
correctness to kind of understand like,
58:59
oh, like something is not going well.
59:01
That you're not just like
59:03
completely in the dark and looking for
59:05
the needle in the haystack, but that you
59:07
actually have a great foundation to
59:10
figure out what went wrong.
59:12
That is the, I think the foundation where
59:14
people start leaning on observability
59:17
beyond just console log. But then also
59:21
like doesn't just help you with making
59:22
things correct or diagnosing when
59:26
something was not correct,
59:27
but also making it faster.
59:29
Like otherwise you might just know like,
59:31
okay, that API request has taken two
59:33
seconds, but why? And sometimes there's
59:36
like really counterintuitive situations.
59:38
It's very simple and very
59:40
easy to realize, oh,
59:42
this is why it's so slow.
59:44
And also speaking of AI, this will like
59:49
be a perfect foundation to give an AI
59:52
even more context what is going wrong and
59:55
let the AI iterate further and help you
59:57
make your app more
59:59
reliable and more performant.
01:00:00
One of the frontiers that we're going to
01:00:03
be exploring, I think now that we've
01:00:05
cracked the seal on observability is
01:00:08
integrating it with our existing data
01:00:10
analytics or our end user analytics that
01:00:14
we are already collecting.
01:00:15
In MasterClass we're a really good, robust data
01:00:17
team that is, you know, while respecting
01:00:21
like anonymity and user privacy is still
01:00:24
really invested in
01:00:25
understanding the user journey.
01:00:28
What are people doing? You know, why are
01:00:30
they there? What is enticing
01:00:31
them? What's driving them away? And these
01:00:33
sorts of questions that are really
01:00:35
foundational to understanding the best
01:00:39
way that we can
01:00:39
deliver value to our users.
01:00:41
Integrating this with a sort of Opentelemetry
01:00:43
will give us even more insights
01:00:46
of like, oh, did a user try to
01:00:49
load a page, but then they bounced
01:00:52
because it took too long to load and
01:00:54
things like this that will give us an
01:00:57
integration level between the sort of end user metrics
01:01:01
that we've been using and also the technical
01:01:04
implementations behind the scenes that
01:01:05
are underpinning that user experience.
01:01:09
I'm really looking forward to being able
01:01:11
to explore that further. And I feel like
01:01:14
it has tremendous
01:01:15
potential to offer value.
01:01:17
That sounds awesome. So speaking of a bit
01:01:20
more forward looking perspectives
01:01:22
I'm curious now that you've been part of
01:01:25
the Effect community, I think at this
01:01:27
point way beyond a year already and super
01:01:30
impressive what you've been able to build
01:01:33
with Effect in that short period of time.
01:01:35
What are the things that you're looking
01:01:37
forward most to when it
01:01:39
comes to the Effect ecosystem?
01:01:42
I'm really looking forward to seeing
01:01:43
Effect 4.0 come out. That looks awesome.
01:01:47
A lot of the improvements they've made to
01:01:49
the bundle size and to the implementation
01:01:53
complexity look really promising.
01:01:56
And, you know, I really admire how
01:01:59
responsive the Effect team has been to
01:02:03
the community, to the needs of the
01:02:05
community and really listening to
01:02:07
feedback, incorporating it, iterating.
01:02:10
That's really, I think, been vital to any
01:02:13
platform like this, getting any traction.
01:02:16
It's a very ambitious platform. It just
01:02:20
has to be said, the design of it to
01:02:23
encapsulate so much information about
01:02:26
what's going on at an atomic level in
01:02:28
your application, but then extending that
01:02:30
out into really
01:02:31
building whole frameworks.
01:02:34
The HTTP API, the Schema, the tracing
01:02:38
integration, the possibility that the
01:02:42
database abstractions, the networking
01:02:45
abstractions like WebSockets, file
01:02:47
system, node runtime,
01:02:50
there's React integrations.
01:02:52
Really, you name it, there's just tons
01:02:55
and tons of, and there's a whole bunch of
01:02:56
stuff coming in down the pipeline,
01:02:58
Cluster and AI integrations.
01:03:01
The paradigm is really extensible and it
01:03:05
has proven itself really robust and able
01:03:08
to handle all these different scenarios.
01:03:10
But now making all of those things all
01:03:13
work, you know, that's
01:03:15
an incredible challenge.
01:03:16
I hope something that they
01:03:19
succeed at. It's just so much.
01:03:22
But I think that with this large and
01:03:25
growing community, I think there will be
01:03:27
people using every aspect of that.
01:03:30
Probably not everybody is going to use
01:03:33
every piece of the Effect
01:03:34
ecosystem. I am not, certainly.
01:03:37
And most people will only use a small
01:03:39
sliver of it. And that's fine. That's all
01:03:41
you really need to do.
01:03:43
But when you have hundreds or thousands
01:03:45
of engineers all building different kinds
01:03:48
of applications with it, now you start to
01:03:50
get a lot more signal on
01:03:52
for this specific use case.
01:03:54
Here's what I'm trying to do.
01:03:56
Here's how I've approached it.
01:03:57
And that information, feeding that
01:03:59
information into the design of the
01:04:01
platform is going to be tremendously
01:04:03
valuable to making it more extensible.
01:04:07
But anyway, I'm really interested in the
01:04:09
Effect 4.0 developments that I think have
01:04:12
come out of this exact kind of feedback
01:04:15
integration and iteration.
01:04:17
And also, I'm really excited about things
01:04:20
like the, I think there was at the Effect
01:04:23
Days, there was some, there was a demo
01:04:24
that I think Mattia did.
01:04:26
Mattia, who did about the error reporting
01:04:30
and that was integrated into the editor
01:04:32
when there's like type errors in Effect.
01:04:35
Sometimes those can be cryptic and it's
01:04:38
nice to see that they're working on
01:04:40
making those a lot more
01:04:41
human readable and friendly.
01:04:45
It's nice to see the dev tools getting so
01:04:47
much love. It's nice to see the
01:04:50
documentation getting so much
01:04:51
improvement, so often very thankless.
01:04:53
Honestly, more than any specific
01:04:56
technical feature of the API, I just love
01:05:00
to see the developer
01:05:01
experience getting some attention.
01:05:03
And it makes things so much easier on
01:05:05
everybody to start building really cool,
01:05:08
ambitious things if you get really good,
01:05:13
clear, understandable errors at build
01:05:16
time or in your editor.
01:05:18
And if you have really good debugging
01:05:21
tools that let you understand what's
01:05:23
happening, if you have really good
01:05:24
documentation that has use cases and
01:05:26
examples of all kinds of different types
01:05:30
of patterns that you might
01:05:31
want to take advantage of.
01:05:33
This is the sort of, I think often unsexy
01:05:36
work of building a framework like this
01:05:38
that is so fundamental to people, like
01:05:41
end users being able
01:05:42
to get value out of it.
01:05:43
And it seems like the Effect team is
01:05:45
taking this seriously. And that to me,
01:05:48
more than almost anything, is what gives
01:05:50
me excitement and hope about
01:05:53
the future of this framework.
01:05:54
100%. I mean, I couldn't agree more.
01:05:57
They're all super ambitious efforts on
01:06:01
their own. And luckily, we have a small
01:06:05
but super talented and absolutely
01:06:08
brilliant core team of folks who are
01:06:11
working on the various pieces.
01:06:13
You've been mentioning the docs. We have
01:06:15
Giulio working more or less full time on
01:06:18
the docs as well as on the Effect Schema,
01:06:21
which we could probably also fill an
01:06:23
entire show on just talking about the
01:06:25
Effect Schema, which I
01:06:26
think you're also using.
01:06:27
Oh, yeah. We're using Effect Schema all
01:06:29
over the place. I didn't
01:06:30
even talk about that. Yeah.
01:06:31
Yeah. But then also the the dev tools.
01:06:35
I'm super excited that
01:06:37
Mattia will be joining.
01:06:39
Well, at the time of this recording, a
01:06:42
couple of days from now, but when the
01:06:43
time comes and the recording airs, I
01:06:46
think Mattia will have already started
01:06:48
working full time on the dev tools and
01:06:50
other pieces and improving
01:06:52
the developer experience.
01:06:54
And then at the core of all of it, Effect
01:06:56
4.0 and what it will enable. That's
01:06:59
really been the distillation
01:07:01
of all the feedback that we've gathered
01:07:03
over the last couple of years.
01:07:04
And I think really tackle some of the
01:07:07
most common points of feedback or
01:07:09
frustration head on. And I think it's
01:07:12
really like a huge improvement over
01:07:15
what's already really great and useful.
01:07:19
So I'm fully on board with with
01:07:22
everything you've you've shared. And
01:07:25
yeah, I think I can't thank you enough
01:07:28
for sharing all that what you've been
01:07:31
building with Cortex at MasterClass.
01:07:34
It's really, really impressive.
01:07:36
What you've been able to build in such a
01:07:38
short period of time speaks to
01:07:40
experience, but also speaks to just like
01:07:43
how capable the building blocks are. And
01:07:45
like when like experience and ambition
01:07:48
comes together with great materials.
01:07:50
I think this is how we're going to get
01:07:51
great experiences and great applications.
01:07:54
So thank you so much for doing that work
01:07:57
and sharing it.
01:07:58
And thank you so much for coming on the show today.
01:08:01
Well, thank you so much for having me.
01:08:03
And I just you know, I have to say that
01:08:05
so far, I think the on call product has
01:08:08
been a huge success and Cortex has been
01:08:12
very, very crucial part of that.
01:08:16
And I feel like it would not have been
01:08:18
possible without the Effect ecosystem and
01:08:21
also the support of the Effect core team
01:08:23
and the community to helping us get
01:08:27
there. I think that has played a fundamental
01:08:29
role. So thank you. I mean, to you,
01:08:32
Johannes, and thank you to the broader
01:08:34
Effect Team and the community for all of
01:08:37
the support and assistance and enthusiasm
01:08:39
and building this incredible framework.
01:08:40
It has really been a game changer.
01:08:43
That is awesome to
01:08:44
hear. Thank you so much.
01:08:46
Thank you.
01:08:48
Thank you for listening to the
01:08:49
Cause & Effect Podcast.
01:08:51
If you've enjoyed this episode, please
01:08:53
subscribe, leave a review
01:08:54
and share it with your friends.
01:08:56
If you haven't done so already, you can
01:08:58
join our Discord community.
01:09:00
And if you have any questions, feedback
01:09:02
or suggestions about this episode or
01:09:04
about Effect in general,
01:09:06
don't hesitate to get in touch.
01:09:08
See you in the next episode.