Scala Days 2025↑
Edit September 2025
You can now find the talk’s slides here!
The recording should follow by end of this year.
I’ll be speaking at Scala Days 2025 in Lausanne, Switzerland! I am very exited about this. The talk title is “Using Scala in a Go-First Company”.
Abstract:
How do you write Scala that’s accessible, maintainable, and doesn’t try to be too clever, especially in a company where Go is the default? And why would you introduce Scala in the first place? We’ll cover the rationale for starting a Scala project in a go shop and how we manage to maintain it.
Tickets & Details↑
➡️ Details here: scaladays.org/editions/2025/talks/using-scala-in-a-go-first
➡️ Tickets: scaladays.org/tickets
Context↑
We run a small, but very impactful Scala codebase @ work. It’s a set of self-hosted Flink jobs (using ngrok + Kubernetes), written in Scala 3 with the wonderful flink-scala-api.
They are responsible for ingesting a lot of important, real-time networking and metadata we use for support, analytics, GTM and other stuff. We average ~20k events/s, made of up various, rather complex data streams. I wrote a bit more about it here not too long ago.
Introducing a new language (and one that isn’t Python) into our engineering stack was certainly a big decision. The Data Engineering world is tightly integrated in our regular product engineering stack and not a separate silo. I advocate heavily for this and hightly suggest ever Data Engineer to do the same. I’m talking about using the same repo(s), CI, tooling, developer CLI, formatting etc. as the main codebase, as well as using the same processes (on-call, deployments) and auxillary tooling (observability etc).
And Scala offer a lot of neat features that make this work, especially around writing generic, safe code. Writing truly generic code - using more than just basic invariant generics and simple interaces, like go - that has actual compiler support to help us enforce it (unlike Python) is one of the few ways we keep a large Data Platform maintainable by a team of one (and we do use Python, too, just not for real-time stuff).
This isn’t a “my language is better than yours” conversation. A language is just a tool.
I like go and Python and use both at work, the former more than the latter. This talk isn’t about “which language is better”, but rather “why was Scala for this in particular?” and, more specifically, how we make that work.
Speaking of: I’m the only full-time DE, and I don’t even spend my entire time on the Data Platform, so making things generic and re-usable is one of the top priorities to keep this maintainable.
Almost all code is expressed as Thing[A : MaybeATypeClassOrTwo] extends MaybeATrait (no effect system), which makes extending it reasonably easy - it’s impossible for me to maintain hand-grown, artisinal schemas like shops with large DE teams might. It is possible to maintain a largely generic, self-hosted ecosystem with good observability and tooling.
Since the rest of the code base is almost all go, this could have gone horribly wrong. But there’s a lot of lessons learned over the past year and a half or so that I found worth sharing. And in this talk, I’ll try to do just that.
Hope to see you there!
New Website↑
If you’ve been here before, you’ll notice that I moved from hugo to Astro.
What↑
I’ve been dabbling in frontend/TypeScript stuff - you’re looking at it - and essentially heavily customized the default Astro theme with almost all features of ink-free, namely:
- Table of contents
- + Floating TOC
- Tag support
- Tag search
- Word count + reading stats [1]
- Several internal components, such as the preview boxes, dynamic icons with gradients and such
- Some custom
mdxcomponents, such as images with proper attribution, admonitions etc - Correct Open Graph support, I think?
- Reasonable mobile support
If you find something is broken, please reach out.
[1]: Those aren’t 100% correct, since I use pandoc to do it and I think I need a Lua script to filter code blocks…
Markdown translation↑
It’s still using markdown for the blog articles, but I was forced to turn them into mdx to support all the existing features.
I figured writing a full article about this wouldn’t be too interesting, since the most interesting thing (other than me figuring our “modern” web dev) was translating all 35 articles (150k+ words) into mdx with the new syntax. This was done with throwaway Python.
While most things are the same, things like admonitions, need a different syntax:
735,736c742,744
< > [!TIP]
< > Code diff [here](https://github.com/chollinger93/bridgefour/pull/6/files).
---
> <Aside variant="tip">
> Code diff [here](https://github.com/chollinger93/bridgefour/pull/6/files).
> </Aside>
1250c1258
< {{< figure src="assets/raft_state_machine.png" caption="State transitions for leader election" attr="Howard, Heidi, Malte Schwarzkopf, Anil Madhavapeddy, and Jon Crowcroft. 'Raft Refloated: Do We Have Consensus?.' *ACM SIGOPS Operating Systems Review* 49, no. 1 (2015): 12-21." attrlink="https://api.repository.cam.ac.uk/server/api/core/bitstreams/c9bcee5b-a1cb-4147-9281-1a05632f5aa3/content">}}
---
> <ImgAttr src={img_raft_state_machine} caption="State transitions for leader election" attr="Howard, Heidi, Malte Schwarzkopf, Anil Madhavapeddy, and Jon Crowcroft. 'Raft Refloated: Do We Have Consensus?.' *ACM SIGOPS Operating Systems Review* 49, no. 1 (2015): 12-21." attrlink="https://api.repository.cam.ac.uk/server/api/core/bitstreams/c9bcee5b-a1cb-4147-9281-1a05632f5aa3/content" />
I also used it to automatically parse all images -  syntax or Hugo’s {{ < figure}} - and turned the first match of either into a OpenGraph title image (which the previous blog usually didn’t have).
In the same pass, I also made sure all slugs matched (by querying the old blog while it was live for each to ensure we have no dead links):
< slug: "a-distributed-system-from-scratch-with-scala-3-part-3-job-submission-worker-scaling-and-leader-election-consensus-with-raft"
---
> slug: "2025/05/a-distributed-system-from-scratch-with-scala-3-part-3-job-submission-worker-scaling-and-leader-election-consensus-with-raft"
> img: /assets/og/2025-05_bridgefour-part3.png
8a10,12
> import img_raft_state_machine from 'assets/raft_state_machine.png';
> import img_dependency_2x from 'assets/dependency_2x.png';
Similar things happened for GitHub gist support and others.
Why↑
Largely because I wanted more features, flexibility, and a consolidated website that is independent of LinkedIn as a “professional” presence.
I also noticed my web dev knowledge was growing more and more outdated - my most up-to-date JavaScript book is from 2011 and I own a largely unread CSS book from 2012 - which caused me some pain when working with Slidev, for instance. I have no interest in doing FE or full stack full time (I like my backend work, thank you very much), but I think it’s a useful thing to know.
It’s still static site generation at the end of the day, so I didn’t quite jump into the React (and others) rabbit hole.
Minimalism↑
It’s also a bit of a departure from my previous minimalism. My old website was a plain html file with some inline-CSS that could be maintained via ssh + vim with a dial up connection from the middle of the woods. The point here was to avoid the “modern web” bloat of shifting content that trick you into clicking something, scrolling popups, auto-playing videos, endless trackers and CDNs (…).
However - one can do this with modern tools. I have some work to do to make this site more accessible to slower connection speeds (it’s pretty managable via simulated 3G, but not great), but I don’t think using tools made after the year 1991 is a bad thing - provided I’m not the one actively supporting dark patterns.
That said, in the interest of transparency: I am using a self-hosted umami instance for web analyitics. It does respect your “Do not track” header, but I figured I’d let you know. Using goaccess is great and all, but this is better - it’s nice to know what people like to read, I guess. This seemed like an implementation that’s compatible with my privacy standards.
ink-free & Hugo↑
Hugo is great and I plan to support the previous blog’s template project (ink-free - contributions welcome), but working on a private theme allows me to make changes that might not be suitable for a re-usable theme, which makes this a lot easier to maintain for myself, since I can e.g. hard-code text or URLs without having to think about how I can make this configurable for others. While writing this, I adjusted the <blockquote> margins without having to create a PR, test it, cut a release etc.
I’m not necessarily happy about that (I love Open Source), but I unforunately have to spend my time on side projects somewhat wisely.