Event-Driven Systems in FinTech. How Spiko Leverages Effect
#5: Event-Driven Systems in FinTech. How Spiko Leverages Effect
This podcast episode features Samuel Briole, CTO of Spiko, a Paris-based FinTech startup building infrastructure for issuing regulated financial products on public blockchains, specifically risk-free products. Spiko utilizes the Effect extensively from day one, for both backend and frontend development, particularly for managing asynchronous operations, API integrations, and smart contracts.
This podcast episode features Samuel Briole, CTO of Spiko, a Paris-based FinTech startup building infrastructure for issuing regulated financial products on public blockchains, specifically risk-free products.
Spiko utilizes the Effect extensively from day one, for both backend and frontend development, particularly for managing asynchronous operations, API integrations, and smart contracts.
Learn more about & follow Samuel:
Effect is an ecosystem of tools for building robust, production-grade applications in TypeScript.
Song: Dosi & Aisake - Cruising [NCS Release]
Music provided by NoCopyrightSounds
Free Download/Stream: http://ncs.io/Cruising
Watch: http://ncs.lnk.to/CruisingAT/youtube
- (00:00) - Intro & why Spiko chose Effect
- (07:22) - Hiring engineers without Effect experience
- (10:06) - Spiko’s Architecture & Apps
- (15:12) - Event-Driven Systems
- (22:45) - Choosing RabbitMQ over Kafka or NATS
- (28:26) - Why they built effect-messaging
- (39:53) - Effect Schema, Versioning, Async Messaging
- (46:44) - Agents & Effect AI
- (53:21) - Closing Thoughts & Looking Ahead
Transcript
00:00we've been a small team so far
00:03but we've been working
00:04with Effect from scratch.
00:07when I decided
00:08the stack for Spiko
00:10I chosed Effect because it was the natural
00:12successor to fp-ts.
00:15And for hiring people
00:17most of the
00:18employees didn't know Effect
00:20before they came at Spiko.
00:22and I trained them
00:24and they also trained
00:25with the documentation
00:27and so far it's been okay.
00:30I think the
00:30documentation has really improved
00:34there is still a learning curve
00:35that is a bit higher than other libraries,
00:38but I think it's okay.
00:43Welcome to "Cause & Effect," a podcast
00:46about the TypeScript library and
00:47ecosystem called Effect,
00:49helping engineers to build
00:51production-ready software. I'm
00:53your host Johannes Schickling,
00:54and I've been building with Effect for
00:56over four years. With this
00:57podcast, I want to help others
00:59understand the powers and benefits of
01:01using Effect. In this episode, I'm
01:04talking to Samuel Briole,
01:06CTO at Spiko, which is a Paris-based
01:09fintech startup trusted
01:10with over 400 million euros
01:12under management. In this conversation,
01:15we dive deep into Spiko's
01:16event-driven architecture,
01:18built entirely with Effect and how they
01:20leverage RabbitMQ as an
01:22event broker to scale their
01:24backend systems. Let's get into it.
01:27Hey, Samuel, it's so great
01:28to have you on the podcast.
01:29How are you doing?
01:31Hi, I'm good.
01:32How are you?
01:34I'm doing fantastic.
01:35I'm quite impressed with
01:37what you all are building
01:38and I want to learn a lot more.
01:39But before getting
01:40into what you're building,
01:41would you mind introducing yourself
01:43and sharing what you do?
01:45Sure.
01:45So I'm Samuel.
01:47I'm CTO at Spiko and we are a French
01:50startup based in Paris.
01:52So we are a FinTech,
01:55especially in the finance sector.
01:57And we are building a new infrastructure
02:01for issuing regulated products,
02:03regulated financial
02:04products on public blockchains.
02:08And we especially focus
02:10on risk-free products.
02:13we allow people to buy a bond of,
02:17I don't know, from, of the French State
02:19or of the US States.
02:22So they can have a risk-free rate.
02:27They can earn a
02:28risk-free rate on their cash.
02:31So basically if you're based in the US,
02:35it's a very common thing to
02:37have like a risk-free rate
02:39yielding on your cash, but in Europe,
02:42it's not very common to have that.
02:44our mission is to
02:46democratize this kind of product.
02:49we launched Spiko one year ago
02:52and the company was
02:53created one year before,
02:54so it was been working for two years.
02:57And we launched our products one year ago
03:00and today we have
03:01around 400 millions of euros
03:05under management, which
03:07is quite a nice milestone.
03:11And also last month we raised a Series A.
03:16So we raised with Index Ventures,
03:19which is famous for
03:21supporting the Revolut
03:22and the Robinhood in
03:24the same financial sector.
03:26That is awesome.
03:26Well, first of all, congratulations
03:28on those amazing milestones,
03:30like 400 million under management
03:33or whatever is the best way to put it.
03:35That is impressive.
03:36And then also congrats on the series A
03:38with Index Ventures.
03:40I'm curious to learn a little bit more
03:42about like both your personal background,
03:44how you arrived there, but also about
03:47how you should think
03:48about Spiko as a company,
03:50how many engineers are working at Spiko.
03:53So maybe you can give a
03:54little bit more detail here.
03:56Sure.
03:56I did an engineering school
03:58where I learned physics
03:59and I almost started a PhD
04:02in quantum physics.
04:03But at some point I
04:06decided that I didn't want
04:08to do research and I switched to
04:10software engineering.
04:12So during my studies,
04:13I was building like an
04:16app for discovering electronic music.
04:19So I learned to code a bit by myself.
04:22And I was like, okay, it's so cool.
04:23I want to do this as my job.
04:27I started my career
04:29in a small service company.
04:31And quickly I wanted to
04:33build my product from scratch.
04:36So I worked for a company called Inato
04:39so I was maybe the second engineer.
04:43And then I tried to launch my own startup
04:47and I failed two times.
04:50And then I met the founders of Spiko.
04:53And I found the mission really exciting.
04:57And also the technological,
05:00challenge at Spiko
05:02using the blockchains
05:03and also innovating both
05:05on the tech side and
05:06on the financial side
05:08was very exciting.
05:09So that's how I arrived at Spiko.
05:12That is awesome.
05:13And there's quite a bit of overlap there
05:15with other people in
05:17the Effect community.
05:18You mentioned the folks at Inato.
05:21I had the pleasure of
05:22meeting some of them
05:24throughout the last
05:25two Effect Conferences.
05:27And it seems like there's
05:28quite the active Effect scene
05:30in Paris, which I'm sure we'll talk
05:32a little bit later about as well.
05:34But yeah, that journey so far sounds
05:36really, really cool.
05:37I see a bunch of
05:38parallels and similarities.
05:40I'm currently still
05:42working on a music app
05:44for me personally
05:45focused on electronic music.
05:46So I wish it certainly
05:48seemed to share that passion here
05:50but now you're a CTO at Spiko.
05:53So what does that entail
05:55and how many other
05:56engineers are you working with?
05:58the engineering team at Spiko
06:00is currently five people
06:01and there is one joining soon.
06:04And our goal is to reach around, I think
06:0810 people next year.
06:09So we'll split into probably two teams.
06:12we've been a small team so far
06:15but we've been working
06:17with Effect from scratch.
06:19So everyone in the team is using Effect
06:23and everyone is doing
06:24TypeScript and Effect in the team.
06:26That's awesome.
06:27So you mentioned to me before
06:29that you joined as the first employee
06:32besides the founders.
06:33So you were there from the very beginning
06:36and the other engineers
06:37who joined afterwards,
06:39did they have any prior Effect experience
06:41or any other relevant
06:43experience besides TypeScript?
06:45Or I'm just curious
06:47like how I should think
06:48about the sequencing
06:50now Spiko has been around
06:51for two years.
06:52You've probably written
06:53some of the first lines of code
06:55and then the other
06:56engineers you brought on,
06:58was it a prerequisite for
06:59them to already know Effect?
07:01Did you train them?
07:02No, it's not a prerequisite.
07:04I discovered the functional
07:07programming at Inato
07:08when I was working at
07:09Inato and we used fp-ts.
07:11And then when I decided to choose
07:15like the stack for Spiko
07:17I chosed Effect because it was natural
07:19successor to fp-ts.
07:22And for hiring people
07:24most of the
07:25employees didn't know Effect
07:27before they came at Spiko.
07:30I think none of them knew
07:33Effect and I trained them
07:36and they also trained
07:37with the documentation
07:39and so far it's been okay.
07:42I think the
07:43documentation has really improved
07:46there is still a learning curve
07:48that is a bit higher than other libraries,
07:50but I think it's okay.
07:52Also at Spiko, we have, I mean,
07:54we are all Full Stack
07:55and Dev Ops and we,
07:58we do all kinds of things
07:59and we also develop smart contracts.
08:02And so far we don't have
08:04specialized roles in tech.
08:06So when we hire people,
08:08we try to hire people
08:10who can quickly jump
08:11on new technologies and
08:13learn this technology.
08:14So, so far everyone has
08:18been learning Effect.
08:20That is awesome.
08:21Congratulations on that.
08:22I share that similar path to you
08:25that I found my way to Effect through
08:27functional programming,
08:28but I do think that this
08:30won't be what brings most people
08:33to effect these days and
08:34certainly in the future.
08:36So I'm curious, like what is your
08:39teaching methodology
08:40when you bring on
08:42someone new to the team?
08:43Are you precisely
08:45already looking for someone
08:46with functional programming backgrounds
08:48or are you rather like
08:50not mentioning FP at all?
08:52Like how you're
08:53explaining Effect to new engineers?
08:56It's a good question.
08:57I don't mention
08:58functional programming.
08:59I just say it's inspired by functional
09:02programming concepts,
09:04but I don't think it's
09:06a good way to put it to,
09:07to say it's functional
09:08programming because it's not.
09:10I prefer
09:10to describe Effect as
09:13a framework
09:14that has type level error
09:17and included dependency management.
09:20So basically in TypeScript by default,
09:22you have, you would have the success type
09:25of a function that is correctly typed,
09:28but here you have, you have
09:29also errors and dependencies.
09:32So I think it's a simple way to put it,
09:36like you will have also success,
09:38the error and the dependencies.
09:40So maybe we can switch gears slightly
09:43and talk a little bit
09:44more about the architecture
09:46of the product that you're building.
09:49So maybe we can start by you describing
09:53which sort of like
09:54consumer facing parts there are.
09:56Is there maybe a mobile app?
09:58Is there a web app?
09:59Are there some other primitives?
10:01And then switching gears
10:03and describing more of like
10:04the backend architecture.
10:06So, so far we have three apps,
10:11three front-end apps written in
10:12TypeScript and React.
10:14we are building a platform
10:15that connects financial
10:17product issuers, distributors,
10:20and also we have like
10:22the admin interface.
10:25So we have three types
10:27of users in our system,
10:30investors, distributors, and admins.
10:35So basically we built three apps
10:37and they are all connected to three APIs.
10:42And all these APIs are
10:44connected to backend services.
10:47And so the way we
10:50designed our backend is we built,
10:53I would say not microservices,
10:55but I would say macroservices.
10:57It's not a monolith, but
10:59we have like a few services.
11:01One service to manage, I don't know,
11:03fund operations, one to manage orders,
11:07one manages the blockchain stuff.
11:12So that's how we designed our backend.
11:15Got it.
11:16And so given those different APIs,
11:19I assume you're using
11:20the various building blocks
11:22that Effect provides for that.
11:24For example, the Effect HTTP
11:27packages from the Platform.
11:29Absolutely.
11:30So at the beginning, when we started,
11:33there was no platform package in Effect.
11:35I think it was two years ago.
11:38And Platform
11:39almost did not exist, I think.
11:41So the first thing I did was wrapping
11:45Express into Effect,
11:48which came up as a very bad idea
11:51because it was a mess
11:53to manage dependencies, and
11:54I had like some
11:55non-Effect code in my code base,
11:57which wasn't really nice.
12:00So basically my entry
12:01point wasn't an Effect.
12:03It was an Express server.
12:05And Effect was in each endpoint handler,
12:10I had to Effect.runPromise my code.
12:14But later the Effect
12:16platform package came in Effect
12:18and we switched to it.
12:20And at the time there
12:23was no HTTP API module.
12:26So we used effect-http
12:30which was a lib
12:31written by Milan Suk.
12:32And we used it for a while actually.
12:34And I even met Milan at
12:36the first Effect Conference
12:37and it was awesome to meet
12:42everyone at this conference.
12:44But yeah, we used it for a while
12:46and then HTTP API module arrived
12:52and we switched to it.
12:53We migrated everything to it then.
12:55Yeah, I mean, I
12:57remember those transitions well
12:59similar to you where,
13:01like I also used fp-ts in the past
13:04where you got those like much more fine
13:07granular primitives,
13:08but not as high level of a thing
13:11such as like an HTTP module.
13:14And I still remember,
13:15getting something really nice,
13:17like getting those
13:18primitives from Effect,
13:20but having to give up on
13:22higher level primitives
13:23where you like imagining Express or Koa
13:27or like all of those different things,
13:29you arguably had better primitives,
13:31like you had middlewares, et cetera.
13:34And now you kind of needed to,
13:36you got one thing,
13:37but you had to give up of another thing.
13:39And now you're kind of wondering,
13:40what should I do about this?
13:42And you did a similar thing as I did,
13:44which is like trying to rebuild
13:46those higher level primitives.
13:48But now it's so nice
13:50that it was just always
13:51just a matter of time
13:52until the Effect ecosystem
13:55and the Effect Core
13:57which give us some of those primitives.
13:59And now finally the time has come over
14:01the last year or so.
14:04if you're going through that transition,
14:06it's just so nice how
14:07much code you can delete
14:09and how everything
14:10just fits like a glove.
14:12So I wanna learn more about the backend,
14:14but maybe briefly to wrap
14:16up on the front end first,
14:18how much or how little are
14:20using Effect on the front end?
14:23Are you stopping by
14:24basically consuming the HTTP APIs
14:27or are using Effect
14:29in a more involved way
14:31on the front end?
14:32No, we use it only for data fetching.
14:34So, but it's very convenient
14:37because with the HTTP API module,
14:39you can write APIs in
14:41the declarative way.
14:43So you have auto-generated
14:46clients, which is type safe.
14:48So basically we have a lib
14:52that exports the HTTP API
14:56specification and we have the clients,
14:59we just import the client from the app.
15:02But maybe in the future we could use it
15:07even in a more involved
15:09way, but it's not the case yet.
15:10Got it, that makes sense.
15:12So yeah, let's talk a little
15:13bit more about the backend.
15:14So you mentioned already
15:15those like three pillars
15:17of your application, like
15:19if I remember correctly,
15:20like the investor facing ones,
15:22then the distributor
15:24facing ones, and then the admin,
15:26which I suppose is like your company.
15:28so at the
15:28minimum for each of those,
15:30there seems to be an HTTP API,
15:32but then you also
15:33need to interface somehow
15:35with like external services.
15:37So is that all done from those HTTP APIs
15:40or are there some
15:41other parts in the system,
15:43maybe like some message buzz or yeah,
15:46maybe you can explain a little bit more
15:48of like how the backend
15:49works and how Effect helps there.
15:51So as I told you, we have multiple
15:53services now backend
15:55and they sometimes need to communicate
15:58between one and another.
16:00So the first thing we did
16:03was to build an internal API.
16:06So for each service, we
16:07built an internal HTTP API
16:10because it was the same
16:10technology as external APIs.
16:14So it was easy for one
16:15service to call another service.
16:17We had the same,
16:18auto-generative client,
16:20everything was type safe.
16:22So it felt very nice.
16:24And at some point we had to,
16:28run some, I would
16:29say some background jobs,
16:30for instance, an investor
16:33wants to create an order
16:35and we need to send him an email.
16:37But before sending the
16:38email, we need to generate PDF
16:40and like a lot of stuff has to happen
16:43and it will take maybe several seconds,
16:47maybe 10 seconds. I don't know
16:49But we want to do it as a background job.
16:51So we can answer with
16:53the HTTP response quickly.
16:55So the first thing we did
16:58was using our internal API
17:02but we used something to
17:04avoid blocking the HTTP response.
17:08so we used Effect.forkDaemon
17:11so we would fork the
17:14execution of an internal API call
17:16into the, I would say the global scope.
17:20So it would not be blocking for answering
17:24or for responding with the HTTP status
17:27but the server would
17:29still make the request
17:30to another internal service.
17:33And we did that for a while
17:35especially for the emails,
17:37every kind of background jobs.
17:39And at some point
17:41we had a lot of internal API calls
17:43and we still have some, but at some point
17:46I wanted to bring a new,
17:49a better kind of
17:50architecture to do these kind of things.
17:53And what people tipically
17:55do is bring a message broker
18:00that is connecting to all microservices
18:02so service can just
18:04drop events or commands
18:06in the message broker
18:07and service can also listen
18:09to events or commands and handle them.
18:13Got it.
18:13So let me just see
18:15whether I understood correctly.
18:17So you basically have
18:19those HTTP endpoints.
18:20You're running them, are
18:22you running them on something
18:23like AWS Lambda or on stateful servers?
18:27Yes, everything runs in Kubernetes.
18:31Okay, so you're
18:32running that in Kubernetes
18:33so you have the benefit
18:35of long running servers.
18:36So it's one of the
18:37questions I would have had otherwise
18:39is when you say fork deamon, now you
18:42basically need to think
18:43about like the request lifecycle
18:45but otherwise also the
18:46lifecycle of a server.
18:48So if you have the lifecycle of a server
18:50that makes that certainly easier.
18:52But the more complexity
18:54that you had sort of like
18:56as your app develops and evolves,
18:59typically it gets more
19:00complex than simpler over time.
19:02And so you always had that trade off of
19:04like you want to
19:06if the client does something, it should
19:08get a request response
19:10as quickly as possible.
19:12And now you have that
19:14tension where on the one hand
19:17you want to keep the response snappy and
19:19as quick as possible.
19:20And on the other side, you
19:22had more and more complexity,
19:23more and more stuff and
19:25features that should be handled.
19:27So far, you've always kind of let that
19:29run in the background
19:30by basically through the
19:33request response lifecycle
19:35would already create a
19:37Effect fiber for like a fork.
19:41The details here don't matter too much.
19:43But basically now the
19:45question is like the little job
19:48that now somehow runs through the event,
19:51like JavaScript event loop that maybe
19:54gets out of control.
19:56Maybe like what if you need
19:57to do error handling there,
19:58maybe how many of those
20:00are running at the same time.
20:02So this is the situation you were facing.
20:05And now you thought about, okay, what is
20:07like the architectural,
20:09architectually proper way to deal with that.
20:12And that is where you arrived at.
20:14Let's look into event
20:15buses. Is that correct?
20:16Yeah, absolutely.
20:17So we wanted to have a bit more control
20:20with this kind of background jobs.
20:22Also for
20:23the structure of the code base,
20:27we wanted to avoid like
20:29dependencies between services,
20:31like some service, they
20:33just want to drop an event
20:34and they don't have to know
20:36about the API of other services.
20:40So it made more sense
20:41to just drop an event
20:43and not calling the service directly is
20:46also like a best practice
20:48I think in terms of software design.
20:50When your system gets complex,
20:52you want to have like better architecture
20:56and dependencies enforcement.
20:58That makes sense.
20:58And so I just want to get a better
21:00understanding of like the nature
21:02of the things that should
21:03kind of happen in the background.
21:05Are those purely things
21:06that need to be kicked off
21:08but are no longer relevant for the user
21:11experience in the front end?
21:13So like if you click a
21:15button in the front end,
21:16you're like getting the request response
21:19cycle back and a thing happens
21:22and then you don't really care too much
21:24what is happening in the background.
21:26Or is there stuff that
21:27happens in the background
21:28that the front end might now be like
21:30pulling for in some way?
21:32Most of the time we don't call responses.
21:34It's just like background like I don't
21:36know, send an email or we call.
21:41So we're also connected with blockchains
21:43and blockchains are complex systems.
21:46But basically, the registry of our funds
21:49are written on the blockchains
21:50and the blockchains evolves.
21:52But yeah, a lot of events are
21:54happening on the blockchains
21:55and we basically pull the
21:57blockchain continuously.
21:59We index all these events
22:01and sometimes when a
22:03specific event happens,
22:04we want to trigger some code.
22:06So for these kinds of scenarios, it's
22:08really nice to have an
22:11event-driven architecture.
22:13Got it. Yeah, so you're
22:14mentioning an event bus.
22:16So let's dive in. What did you do there?
22:20Suppose there are many different options.
22:22You're not the first one to embrace an
22:25event-driven architecture.
22:27So how did you go about that?
22:29Did you build something like that before?
22:31Did you have any sort
22:32of relevant experience?
22:34Walk me through sort of like the
22:36You getting up to speed on that topic
22:39and then evaluating different options to
22:41what you've ultimately implemented.
22:44So we had like a
22:46discussion with all my team
22:47discussion with all my team and there are many options,
22:49but for us, we prefer to
22:52use an open source project.
22:55So we didn't want to be tied to a
22:59specific cloud provider,
23:01like I don't know, SQS from Amazon, from
23:04AWS or Google PubSub.
23:07And basically, at the end,
23:09we had three main choices.
23:11So the first was Kafka, because Kafka is
23:14widely used, big enterprise.
23:17And it's probably the most used
23:20in the financial sector.
23:22So it would have been a good fit for us.
23:25The second choice was
23:26RabbitMQ, because it's battle-tested,
23:30it's well-used, plenty of plugins, etc.
23:34And the third was like
23:36the new to the game is NATS.
23:39It's more recent, but it's very
23:41lightweight and
23:43It seemed very nice.
23:44So we decided to eliminate
23:47Kafka because it was
23:48heavy to run locally,
23:50not easy to run on Kubernetes.
23:51I think they are a great option to run on Kubernetes,
23:56but it was not the lightweight option.
24:00Overall, we chose RabbitMQ
24:02because it was more battle-tested,
24:04because it had a UI
24:06interface to play with.
24:09So we were more confident with something
24:12that is a bit battle-tested.
24:14And we didn't need to have
24:16something that is super fast
24:20or that can handle
24:21millions of message per seconds.
24:23Also, it's not a very critical choice for
24:28us because
24:29All the data that goes into
24:30RabbitMQ is not persistent.
24:32It's temporary because once the message
24:35is consumed, it disappears.
24:37So at some point, if we want to switch to
24:40another message broker,
24:42we could. It wouldn't be so hard for us to switch.
24:45So we were like, "Okay, RabbitMQ seems
24:48nice. Let's go and let's try it."
24:50But the adventure was a bit
24:53rough after we made that choice
24:56because we discovered a lot of things
24:59that were not so nice
25:00about RabbitMQ and AMQP.
25:04And maybe if I knew them before, maybe I
25:07would not have chosen something like that.
25:10Let's definitely go into that.
25:13For those folks who have never used
25:15Rabbit, you also mentioned AMQP.
25:18Would you mind giving a
25:20quick explainer of what those are
25:22and how they relate to each other?
25:25Sure.
25:26I'm not an expert, but
25:27here is my understanding.
25:29AMQP is a protocol, like HTTP.
25:32It's a protocol that is turned out for
25:35sending and consuming messages.
25:38And there are several brokers that
25:42implement this protocol.
25:44RabbitMQ is one of
25:45them, but
25:46You have other types of message brokers
25:50that implement AMQP.
25:52For instance, I think ActiveMQ.
25:56Everything that has an MQ in the name is
25:59probably an AMQP client and server.
26:04Actually, it's quite nice
26:05because AMQP is widely used.
26:08RabbitMQ is one of the most popular
26:11implementation of an AMQP server.
26:14I think I've used RabbitMQ in the past,
26:17but it's been a little while.
26:19And I'm dealing with other kinds of
26:21event-driven systems these days
26:23where I'm exploring event sourcing more
26:25in a front-end context.
26:27Some similarities, but also very
26:30different when it comes to the use case
26:32that you're describing.
26:34So very curious to
26:35learn more about yours.
26:37You mentioned that you were running into
26:39some issues with RabbitMQ.
26:42So maybe you can describe a little bit
26:44more chronologically
26:45how you went about it,
26:47what you were aiming for, which sort of
26:50assumptions or hopes
26:51that maybe turned out different than you
26:53thought, and what did you arrive at?
26:56It's an interesting question.
26:57So when we started to implement,
26:59the connection between RabbitMQ
27:02and our back-end services, what we did
27:05was we basically wrapped
27:07the AMQP library into Effect.
27:10So we used the official
27:12library called AMQPlib.
27:15the official implementation for Node.js.
27:18basically when you connect to RabbitMQ
27:21from Node.js, you create two objects.
27:23One is the connection, so
27:24you initiate the connection,
27:26and then you initiate a channel which
27:28represents a connection channel
27:31between the client and the server.
27:33So you initiate these two objects, and
27:36what we did is we put
27:38them into, I think, a layer,
27:40an effect layer that was scoped so when
27:43the dependency lifetime was ended,
27:46we could close the
27:47connection, close the channel.
27:49So that's pretty basic.
27:51Then we could just call our channel to
27:54send message and subscribe to it.
27:56What was a bit challenging is that at some
28:00point, we discovered that our services
28:03were not consuming messages anymore
28:05or they were not sending
28:08messages anymore to the server.
28:11Why?
28:12Because the connection and the channel
28:14between client and server
28:16was closed at some point
28:17for any reason.
28:19Maybe the connection between the service
28:22and the server had a problem.
28:25So we are running a
28:28RabbitMQ in the Kubernetes cluster.
28:30It's quite safe because we use
28:31RabbitMQ as a cluster in
28:33our Kubernetes cluster.
28:35So we always have the persistence of
28:38messages and replications of events.
28:42But sometimes one node of
28:44the RabbitMQ cluster goes down,
28:46and then the connection between the
28:48service and the RabbitMQ node goes down.
28:52The problem with the AMQPlib is that
28:55when the connection object is failing,
28:58there is no autoreconnection.
28:59So basically, the connection just drops.
29:02It was a challenge for us to implement an
29:05automatic reconnection of the service.
29:07So we iterated a bit on this problem.
29:10The first thing we did is that
29:11In our Kubernetes
29:14health check endpoint,
29:15we would add a check to see if the
29:17connection was alive.
29:18If the connection was not alive,
29:20so Kubernetes could just kill the bot,
29:22we would start a new one
29:23with a fresh connection.
29:25But still, it was not a good solution
29:29because we were using some events.
29:31We had a lot of issues with that.
29:33What we did is we tried to implement a
29:37way to reconnect automatically.
29:40That's how I ended up writing a library
29:43called effect-messaging
29:44to wrap the AMQP
29:47connection, the AMQP channel,
29:48and add autoreconnection to this.
29:51In Effect, when you create a connection
29:54object and you provide it in a layer,
29:57then you can't mutate the object.
29:59that is in the layer.
30:01But in the AMQPlib
30:03there is no function,
30:05there is no method of
30:06connection to refresh.
30:08So you have to replace
30:11the object completely.
30:12This was an interesting problem.
30:15When I started implementing
30:17the effect-messaging library,
30:20I digged a little bit into Effect
30:23and into what data structure I could
30:26choose to represent this.
30:29I ended up using something
30:31I think I would never use
30:35in standard application.
30:37I ended up using the Ref module to
30:40represent a mutable state.
30:42So what I did is instead of providing a connection object
30:46from the AMQPlib
30:47I provided a Ref of the connection.
30:50Then I could listen to events and to
30:54close events from the connection
30:57and then refresh the
30:58connection and mutate the Ref.
31:01People using the layer
31:03would have the new connection.
31:05I love that you walked us
31:07through from the high level
31:10to the lowest level
31:11what you need to understand
31:13and leverage some lesser known pieces of
31:17Effect to implement this.
31:18Even though it's very specific,
31:20what I think is universal about that
31:23experience that you went through
31:25is that Effect allows
31:27you to model the realities
31:30of your application at this
31:32point of your infrastructure.
31:34Since this is what for me most of software
31:37development is about.
31:39You're trying to
31:40figure out what is the goal,
31:42what are the building
31:43blocks, what are the materials,
31:45what are the pieces that we can use here.
31:48Then you need to develop what is
31:50sometimes referred to
31:51as mechanical empathy for the building
31:54blocks that you have.
31:55And here mechanical empathy is understanding,
31:59"Okay, there's a connection.
32:01In an ideal world, this
32:03connection would suffice
32:05and would never be super stable
32:07and there's just one
32:09computer is never going down.
32:12Super duper easy."
32:13But the tricky part here is we're dealing
32:16with distributed systems
32:18and with distributed systems
32:19there arise a ton of problems
32:21and now you need to
32:23model for those cases.
32:25The specifics is one thing here,
32:28but I think what is
32:30applicable to almost any situation
32:33is you need to model your reality of your
32:36application, of your system.
32:39This is, I think, now
32:40that you've figured out
32:42what are the primitives of
32:43Effect that you need to use,
32:45you could use them and
32:46model your domain perfectly.
32:49In a way where it's
32:52kind of a one-to-one fit
32:53to reality and implementation.
32:57Maybe the first time you
32:58look at this you might think,
33:00"Why are all of those things needed?"
33:01But now that you understand
33:03it, understand
33:05each piece here has a reason.
33:07This is what I also
33:08appreciate about so many pieces
33:11in the Effect
33:12platform packages, et cetera,
33:14where maybe you look at it
33:15the first time you think,
33:17"Why are those three things needed?"
33:19Isn't one sufficient?
33:20But then you realize,
33:21"Oh, actually, those are here
33:23for those various scenarios."
33:25And they're split up into three that is
33:28not a leaky abstraction.
33:30So many things in the
33:31JavaScript ecosystems
33:32are designed in a leaky, abstracted way.
33:36And this is what I
33:37appreciate about Effect.
33:38It kind of absorbs the reality of the
33:41materials you're working with.
33:43And I think this is
33:44exactly what you've done
33:45in the effect-messaging layer.
33:47And also just on a personal note,
33:49this definitely
33:50resonates in regards to the work
33:53I've been doing over
33:54the last couple of years
33:55where I'm diving very
33:56deep into distributed systems
33:58where reconnection, et
34:00cetera, is the bread and butter.
34:02So kudos to you to next to
34:05building a FinTech product,
34:07also dealing with distributed systems
34:10and building a library in that regard.
34:12So congrats, that is awesome.
34:15Actually, it was super interesting.
34:17And I think it's also part of our job to
34:22have a resilient system.
34:24And it's very important for our product.
34:27But it was a really funny experience
34:29to also do this kind of lower
34:33I mean, it's lower than stuff that we
34:38rarely do in a FinTech startup
34:41because we often use existing libraries.
34:44It's really fun, actually.
34:47And now for someone else, there is an
34:49existing library for that
34:51where someone can look at this
34:54and already absorb your
34:56mental model of the world.
34:58You had to do the modeling
34:59carefully yourself before
35:02where you needed to figure out,
35:04"Is this kind of
35:05abstraction modeling correct?
35:07Should there be more pieces?
35:09Should there cannot be condensed more?"
35:11And now someone can look and say,
35:14"Okay, event-driven sounds great,
35:16RabbitMQ sounds great. Let me try that."
35:19And if you've designed it well, they are
35:22falling in a pit of success.
35:24So that is amazing.
35:25And now that you've created this
35:27environment for your fellow engineers,
35:29what is it like now to
35:31work with that system?
35:32You mean the developer
35:34experience with that library?
35:37Exactly. You've created the
35:39world that you wanted to see.
35:40You wanted to have
35:42RabbitMQ as a message broker.
35:44You wanted from your API servers,
35:47wanted to publish events
35:49onto the message broker.
35:51So how is that working?
35:53And then we should also
35:54talk about the consumer side
35:55of how those events are being processed
35:58and any sort of interesting
36:00things emerging from that.
36:02Sure.
36:03I think the library of effect-messaging
36:05at the moment,
36:06it mainly solves the
36:07problem of reconnection
36:09and having a strong connection layer.
36:11And I also created
36:13higher-level interfaces,
36:15like publisher interface
36:17and subscriber interface.
36:19That can allow people to
36:23subsctibe to events and publish.
36:26So at the moment, I kept
36:27this interface
36:29very generic.
36:30So we could add more
36:32message brokers to the library
36:34in the future if we wanted to.
36:36But at Spiko, we use RabbitMQ,
36:39so we used AMQP
36:40publisher, AMQP subscribers.
36:43I mean, we don't have any
36:44more problem with connections
36:46and failing connections.
36:48but those are abstractions on top of this
36:51publisher subscriber
36:52where we have a strong
36:55typing of events and payloads,
36:57like event types payloads.
37:00this is not in the
37:01effect-messaging library.
37:02It's something we built
37:03on top in our codebase.
37:06basically
37:06when we want to create a new event,
37:08we declare a new interface
37:11with an event type and payload.
37:13And that's it.
37:14But maybe in the future
37:16If we find the good abstraction for it,
37:18maybe we could add these kind of features
37:22to effect-messaging
37:23the vision I
37:24have maybe for this library
37:26would be to do something that
37:27is similar to HTTP API module,
37:30but to describe events.
37:32So I think it's a bit of an ideal world,
37:34and it's not done yet,
37:36but we could imagine
37:38something where you declare
37:39your messaging API
37:41with events and payloads
37:43in a declarative way,
37:44and it would create both a subscriber
37:47and the publisher for these events.
37:50Also, another part
37:51that is very interesting
37:53is that with HTTP API module,
37:56you can generate open API specs.
37:59And I've done a bit of research,
38:01and there is an equivalence
38:02in the Async messaging world.
38:05There is an equivalent of
38:07open API called AsyncAPI,
38:10I don't think it's very popular yet,
38:11but it's a way to
38:14expose Async messaging APIs.
38:17So it could be interesting in the future.
38:19Maybe we want to drop events from a
38:23Node.js application,
38:24and there is a new team
38:26using the NOSO language
38:27that wants to generate codegen,
38:30a subscriber for these events.
38:32It could use a generic interface. It could use a
38:36specification to generate a client
38:38for these events.
38:39Maybe in the future we'll have that.
38:41No, this makes perfect sense,
38:43and I do hope that Effect Platform
38:47and more primitives inside
38:49of the core Effect ecosystem
38:52are moving in that direction,
38:54similar to what we've seen already
38:56with the Effect Platform package,
38:58with the HTTP package, etc.
39:00So I think there are many scenarios
39:02where events are super
39:05valuable primitive
39:06here in combination with RabbitMQ,
39:09but I think they're
39:09more universal than that.
39:11And I think there's
39:13many, many related domains
39:16and problems that
39:18should be explored as well.
39:20I'm not sure how relevant schema
39:22evolution, for example,
39:23here would be.
39:25There are a lot of
39:25problems and a lot of challenges
39:27that are explored at scale
39:30in the Kafka ecosystem, etc.,
39:32and bring that more into
39:34default with core Effect
39:36I think would be super valuable.
39:38So kudos to you for
39:39already pioneering in that space.
39:43I'm actually curious for the
39:44events that you can publish
39:46and consume through effect-messaging.
39:49Are you already embracing
39:52effect schemas for that?
39:53I think the way we
39:55describe events is with schema,
39:57because we always use
39:58schema for decoding everything
40:00that comes from the outside world,
40:02because we can't trust the outside world,
40:05so we always decode everything,
40:08everything that comes from the database.
40:10We have a persistence response SQL, so we
40:13use also Effect SQL,
40:15but we decode with schemas,
40:17and for messaging, we describe all the
40:22events with schemas,
40:24and make sure that we encode it and
40:26decode it with schemas.
40:29Got it.
40:30Is the problem of schema evolution
40:32already becoming an issue?
40:34So you mentioned that you're not really
40:37persisting the events
40:39over a longer period of time,
40:40but they're rather
40:41like going onto the broker
40:42until they're taken off, and that's it.
40:45So the only schema evolution scenario
40:48that I could think of
40:49is that you have version
40:50one of the service rolled out,
40:53and then you're rolling out version two.
40:55Somehow there's a period
40:57of time where you still have
40:59a backlog of many messages, which are
41:01still of version one,
41:02and now you've rolled
41:03out version two of the app,
41:05and now that expects
41:07an event of version two,
41:09but gets one of version one.
41:11Have you faced that problem already,
41:13and have you modeled around it?
41:16Yeah, we already faced that problem,
41:18and we're still a young company,
41:21and we don't have millions of users,
41:23so sometimes we take the risk to publish,
41:27to do a breaking change,
41:28because
41:29we don't have a big use of message that
41:32are being consumed,
41:33and if we deploy a breaking change,
41:36we have a very low
41:38probability that we get an error
41:41by breaking the schema.
41:43If we really want to be careful,
41:45what we do is we try to
41:47avoid breaking changes,
41:49and then we have, I think, two options.
41:51Either we modify the existing
41:54schema in a non-breaking way,
41:56like for instance we
41:57add a new optional field,
41:59or we could just create a V2 or another
42:05version of the event.
42:06That makes perfect sense.
42:07I'm just personally very interested in
42:09that problem domain,
42:10and so for example to say we just don't
42:14do breaking changes,
42:15I think that's a super valid way to go,
42:18but I would like to get more
42:21safety rails and guard rails
42:23to make sure that that is actually done.
42:26Maybe you're deep into some refactoring,
42:30and you just have a type error over here,
42:33and you jump to the schema
42:35definition, you add something,
42:36and you don't even think about it,
42:39and TypeScript is
42:40happy, the tests are passing,
42:42and you do that
42:44deployment on a Friday afternoon,
42:46and poof, the weekend is ruined.
42:49How do we get a little
42:50bit more systematized help
42:53that we say, "Hey, this is a schema,
42:55and the schema doesn't
42:57just exist in a vacuum,
42:59but it exists in a chronological context,
43:03and needs to be protected
43:06with some special rules."
43:07This is something that
43:08I'm very curious about,
43:09and I think it's
43:10generally applicable to many,
43:12many parts where you need a schema.
43:15Obviously, to databases as well,
43:17databases have their own
43:18way of dealing with that,
43:20but let's say you will have a little bit
43:22of client-side storage
43:25that you have a schema for
43:27and you always test your app, or
43:30rarely test your app
43:31going from version 1 to version 2.
43:33You test it from scratch in version 1
43:36and from scratch in version 2,
43:38but you rarely test those transitions.
43:41This is just something
43:42that I hope we get to explore
43:44with effect schema as well.
43:47That sounds super cool.
43:48How ready is effect-messaging right now
43:51if someone outside of Spiko says,
43:54"Heck, I want to use that."
43:56How ready for production is it?
43:58Can people already use it?
44:00I just want to rebound
44:02on what you were saying,
44:03because I think it's
44:04also an interesting problem,
44:05and then I answered your question about
44:07the readiness of effect-messaging.
44:11But avoiding breaking change
44:13is something really important.
44:16We discovered something called Optic
44:19that can compare two
44:21versions of an open API file
44:23and that can prevent
44:26breaking changes on open API specs.
44:29I think I was really
44:30happy to discover this
44:31because in Effect you can
44:34generate open API specs.
44:36It's taken to the library.
44:39Already we can profit off
44:42the whole open API ecosystem,
44:44including this tool.
44:46It's a way to leverage the
44:49existing tools in open API
44:50to avoid breaking change.
44:52I hope we can do this for
44:54messaging in future as well.
44:57To answer your questions
44:58about effect-messaging,
45:02I think we are probably the only company
45:05using effect-messaging in production,
45:08at least I don't know
45:09any other companies who do.
45:12But it's working fine for us.
45:16It's production ready in the sense that
45:18we are using it in production.
45:20But of course, I think if
45:23more people want to use it,
45:25maybe we discover some
45:28issues and it's open source.
45:29so anyone can post
45:32issues and contribute.
45:33That is awesome.
45:34It seems like you have a public roadmap
45:37there laid out as well,
45:39which I assume are ideas you
45:41had along the way of thoughts
45:43that it would be cool if at some point,
45:46probably not promises that you're going
45:47to ship those in the near future.
45:50But that sounds very enticing.
45:52It seems like you even considered
45:54supporting other message brokers,
45:56the one that we talked
45:57about initially, Kafka,
45:58NATS, Google PubSub, et cetera.
46:02That is super interesting.
46:03I think the vision is interesting.
46:05Maybe I won't have time to implement.
46:08Actually, I don't think I
46:09will have time to implement this.
46:12But it's open for everyone.
46:15And if someone is interested
46:17in writing a PubSub subscriber, PubSub publisher
46:21I think I laid out a
46:24very generic interface.
46:25So it might be a good way to start.
46:30But I see you already have one
46:32contribution pending
46:33from trusty GitHub Copilot
46:41I launched the Copilot.
46:44So I'm experimenting a
46:46bit this week with agents
46:51Yeah, so maybe let's use that as a segue
46:54to talk about that topic specifically.
46:57I mean, AI agents, coding agents, et
47:00cetera, all the rage these days.
47:02I'm spending a lot of time myself
47:05exploring them, seeing
47:07where they're good at,
47:08where they're not yet good at, how can I
47:11use them for my own benefits,
47:13like for productivity reasons or just
47:17quality of life improvements.
47:19So I'm curious where you
47:20found them most helpful,
47:22and particularly also
47:23in the context of Effect,
47:25maybe areas where you found them to be
47:27not great with Effect yet.
47:30Good question
47:30I can speak a bit about my
47:32time with the agents.
47:33So it's quite recent.
47:36I was not a fan of Closo and not a fan of
47:40AI until a few weeks ago,
47:43a few months ago, I think,
47:45because I didn't find it very productive.
47:49Everything that was written
47:51by the AI, I had to change it.
47:55It was more a loss of time, I think, than
47:58productivity benefits.
48:01But after investing a bit of
48:03time into these agents,
48:05and I think good instructions.
48:09So I've tested a few agents, a few things
48:13like VS Code with Copilot, Closo,
48:18and also like CLIs now, there was a
48:22really interesting
48:23project called opencode.
48:25I don't know if you heard of it, but I
48:28find it very interesting.
48:29Actually, I start to see
48:31some benefits using agents.
48:33I mean, I don't delegate
48:36everything to an agent,
48:38but now it really helps
48:40me to write code faster.
48:42I read everything and I make sure that
48:44it's written as I want it to be.
48:48It comes with some
48:49things way better than before,
48:52because now with agents,
48:55before doing an edit to a file,
48:57it really researches a lot,
48:59like it goes on the internet,
49:00it fetches documentation.
49:02So I set it up for the
49:04Effect MCP for Effect docs.
49:07I'm not sure it's very
49:08often used by my agent,
49:10but if the agent wants
49:12to know more about Effect,
49:15it has access to the
49:16documentation, theoretically.
49:18And now the fact that the agent is able
49:22to grab a lot of context
49:24from the code base from the internet,
49:27it makes it really,
49:28really better than before.
49:30It's quite impressive now, I think.
49:33Yeah, and I mean, it's worth keeping in
49:35mind that this is
49:37developing super quickly
49:39and we're now recording
49:40this at late August 2025.
49:43If someone is watching this
49:45like a couple of months later,
49:46God knows what sort of AI
49:48models are available by then.
49:51And I'm pretty certain this is the worst
49:53it's ever been as it is right now.
49:56So things are developing and
49:58improving very, very quickly.
50:00And this is a very important topic to me
50:03because I think it fundamentally changes
50:06what programming means.
50:08I hope that it will leverage us and
50:11elevate us as system designers
50:14to spend more time about trying to
50:17understand the reality
50:19and modeling the reality,
50:21like expressing which parts of the
50:25reality do we need to model
50:27and which ones can we leave out.
50:30Is it important to modelling that the
50:34server that you're running this on
50:36is like whatever, in a silver
50:39casing or in a black casing?
50:40No, that part of the
50:42domain is not important.
50:43But does it sit in that
50:46one cluster or another?
50:47You get the point.
50:48And I think this is
50:49more like modeling things,
50:51expressing intent, that is
50:54where I see things going.
50:56And the act of implementation is probably
51:00being taken away from us over time.
51:03That's at least my
51:05understanding of the future right now.
51:08But I'm seeing it as glass half full
51:10because what I see as the same as today
51:13is that we're still looking for better
51:15materials to build applications with.
51:18And I'm convinced that AI is going to
51:21have a very good time
51:23building with Effect in the future
51:25just because it is the best material to
51:28build applications with.
51:30But I think we still have a long way to
51:32go to really help AIs
51:34to be very good at working with Effect.
51:38So you mentioned the Effect
51:39MCP, that already helps a lot
51:41and it will get a lot better
51:43because we've recently
51:44switched over the Effect docs search
51:48to use a service and a
51:51tool called Mixspread,
51:52which does smart AI
51:55indexing and retrieval.
51:57So if you now look for interruption,
52:00you actually now should
52:01get a much more useful result
52:03and we're going to wire up the same thing
52:04for the Effect MCP as well.
52:07So that will mean that if an AI needs to
52:10look up interruption,
52:11it gets better context. So
52:13that's going to help a lot.
52:14But the other thing
52:15that AI really relies on,
52:18delivering good results is that it's
52:20trained on various little snippets
52:23and examples similar to us humans.
52:26Some of us are going to go to the source
52:28and read the source code
52:30and apply a lot of mental horsepower to
52:32figuring out how
52:34something should be used.
52:35But I think the majority much rather
52:38prefers to look for an example,
52:40like how can I use Effect
52:41Schema? How can I use Effect HTTP?
52:43Or how can I use effect-messaging instead
52:46of reading the source,
52:47reading an example,
52:48trusting that is correct
52:50and then kind of like transfer applying
52:53that on your own domain.
52:54And this is where we still have a long
52:56way to go to like cover all the API
52:59use cases of Effect and
53:00helping the coding agent,
53:02giving it more training
53:03material. And once we have that,
53:05I expect a huge uptick in
53:08how good the results are
53:10that we're getting from
53:11coding agents for Effect.
53:12So that's something I'm
53:13keeping a close eye on
53:15and expect to see a lot of improvements
53:19there in the future.
53:20So as a last topic,
53:22we can look back on many
53:24years of Effect getting better,
53:26but it's barely getting started. What are
53:29you looking forward most
53:30to in the future of Effect?
53:32Okay. It's a good question. So first, I'm
53:35excited to know that the
53:37team is working on the V4.
53:39I'm not sure what impact it will have
53:42for us, but I'm excited.
53:44I understand that it will have like
53:46performance benefits.
53:48I'm also excited to
53:50try out the Effect AI.
53:53So we were talking about about AI for
53:56productivity with agents.
53:59But I think we could also use AI for
54:03application stuff like for automating
54:06stuff in our business.
54:07For instance, I give you a very simple
54:09example, but when an investor wants to
54:14withdraw
54:14money from the platform,
54:17he fills his bank number and
54:20he has to send us a PDF or something from
54:24a document from the bank to
54:26justify that the correct number.
54:28And at the moment, we have to
54:30verify each document manually.
54:33And this is something we
54:35could automate with AI.
54:37So in the future, we could, for instance,
54:40have a service that calls an LLM sending
54:43the bank account to check with the PDF.
54:46And we could ask for a structured
54:49response, like yes or
54:51no, how confident are you?
54:53And based on this response, we could take
54:56a decision and programmatically validate
54:59all of the bank accounts.
55:00I mean, I'd highly
55:02recommend you giving it a try.
55:04I've used the Effect AI package for very
55:07similar things myself.
55:09I've built a little invoicing system for
55:12myself that I use when I work with other
55:15people or send out invoices myself.
55:18I've used the Effect AI package,
55:20particularly for that use case.
55:22And it just worked first try.
55:25So it is incredible
55:26what is possible with that.
55:28It's nowhere comparable to the amount of
55:31effort it takes to build
55:32effect-messaging from scratch.
55:34But this will be a thing.
55:36You'll be done after 15 minutes and it's
55:39going to significantly improve the
55:42capabilities of your products.
55:44So don't sleep on that.
55:45Yeah, definitely.
55:46And actually, I think it's
55:48a big shift of mindset also.
55:50And I think I didn't realize it until a few
55:54weeks ago, actually.
55:56the way I was interacting with
55:57LLM so far was just sending texts,
56:00receiving texts, but
56:02not structured texts.
56:03Not texts I could decide to put in my
56:06code and do an if-else on the results.
56:10But here it's completely different.
56:12It's a very shift of mindset.
56:15And I think it's a
56:16revolution for programming.
56:19Most certainly.
56:20Well, I'm looking forward to the next
56:22time we have a chat to hear more about
56:24how you're leveraging the AI packages of
56:27Effect to ship new features.
56:29But Samuel, I want to thank you for
56:32taking the time and
56:34sharing your entire journey here.
56:37What you're building at
56:38Spiko sounds super, super cool.
56:40So it seems like you all are hiring.
56:43So if you're based in Paris or want to
56:46move to Paris, I think Spiko is probably
56:48an interesting company
56:49to consider applying at.
56:52And yeah, also huge congrats on all the
56:56milestones you've reached recently on
56:58shipping effect-messaging.
56:59I hope that a lot more folks are using
57:03messaging patterns like that.
57:05I think it's a wonderful fit with Effect.
57:08And thank you for coming on the show.
57:11Thank you so much as well.
57:12It's a real pleasure for me to
57:15participate in the podcast and also to be
57:18part of the community.
57:20So I want to thank you as well.
57:22Perfect. Salut.
57:25Salut.
57:26Thank you for listening to the
57:27Cause & Effect Podcast.
57:29If you've enjoyed this episode, please
57:31subscribe, leave a review
57:32and share it with your friends.
57:34If you haven't done so already, you can
57:36join our Discord community.
57:38And if you have any questions, feedback
57:40or suggestions about this episode or
57:42about Effect in general,
57:44don't hesitate to get in touch.
57:46See you in the next episode.