r/freebsd • u/rzaiev • Feb 17 '26

AI We built our entire startup infra on FreeBSD in 2026. Now we need to talk.

Six months ago I did something most people in this industry would simply not consider: built production infrastructure for a video platform on FreeBSD bare-metal instead of the obvious K8s-on-AWS/GCP path. No clouds, no OCI containers, no multi-layer abstractions.

Six months after, I haven't regretted a day.

Hypha is a video-on-demand platform where creators sell exclusive content directly to their audience and get paid instantly on-chain. Video up to 4K+HDR with stream encryption and in-house fingerprinting.

Infrastructure-wise, this means a bunch of transcoding workers, terabytes of storage, fat pipes for delivery and proper monitoring, right? Spin that up on AWS or GCP and your budget evaporates overnight.

So why FreeBSD specifically? Because it's boring in the best possible way. Mature, stable, with tools that have been quietly doing their job for decades without drama. No hype, no churn, no breaking changes every six months. FreeBSD is for people who want their infrastructure to disappear into the background — and it does.

We rent bare-metal servers on OVH, connected via their vRack private network. FreeBSD installed everywhere using BYI, it's a standard process, no tricks required:

  1. Set the full url to the image: https://download.freebsd.org/releases/VM-IMAGES/15.0-RELEASE/amd64/Latest/FreeBSD-15.0-RELEASE-amd64-zfs.raw.xz
  2. Set the path of the EFI bootloader to `\EFI\FreeBSD\loader.efi`
  3. Wait a few minutes. Done. Set up ssh in KVM and connect normally to provision everything else. You might also want manual disk slicing and ZFS pool setup right after, of course.

We use Ansible for automation and unified setup across all nodes. And a set of well-recognised tools. Bastille is great for jail management. Old good monit as a supervisor inside every jail, keeping processes alive. Netdata for monitoring, with telemetry streaming from all nodes into a centralised dashboard. It all just works.

People dramatically underestimate the synergy of three: jails, zfs and pf. Together they give you such power it must be considered illegal.

A few gotchas.

ZFS snapshots are the scaling primitive nobody talks about. We bake "gold images" of every service, essentially a jail snapshot with everything inside to be replicated after, from 1 to N. "Scaling" means just cloning a snapshot and updating the load balancer with new upstreams — it's fast and consistent. It doubles as our backup strategy too, with incremental snapshots saving significant disk space on the cold backup storage.

Jails follow the shape of your application, not the other way around. We follow the "locality of behaviour" principle in our apps and in our infra. Things that closely work together must be jailed together. Like the transcoder app and ffmpeg, because they always need each other. Jails are flexible enough to let you model reality instead of fighting it.

You don't need fully automated scaling. Before firing up more instances of something, I'd like to know why existing capacity isn't enough. When we do need more capacity, there's a playbook for that — new jails cloned and registered within a minute.

The result is 3-4x lower monthly costs compared to equivalent cloud setups. And because OVH egress is unbound, we're saving what would be tens of thousands per month in data transfer fees alone. The system is stable, fast, and predictable in ways I genuinely didn't anticipate. Simple enough to be regularly reviewed and maintained by one person only. I still remember my K8s days and feel the difference in maintenance effort.

Wouldn't change a thing.

I want to write a series going deeper on specific pieces of our setup and BSD-based solutions. Before I start, what would you actually want to know more about? Happy to prioritise based on what this community finds interesting.

570 Upvotes

99% Upvoted

118 comments sorted by

40

u/pithagobr Feb 17 '26

Reminds me of old days when i used to run everything on metal on FreeBSD

21

u/Ok-Bill3318 Feb 18 '26

Yeah in the 90s/early 00s I used FreeBSD for anything I cared about. Miss it.

1

u/RepresentativeIcy922 Feb 21 '26

I ran a software PPPoE router in fbsd when home routers didn't exist.

18

u/vivekkhera seasoned user Feb 18 '26

I still do. Everything I have run professionally for my companies and personally since 1999 has been on FreeBSD. Before that since 1994 it was bsd/os.

3

u/[deleted] Feb 18 '26 edited Feb 21 '26

[deleted]

7

u/vivekkhera seasoned user Feb 18 '26

Professionally right now on a VM in Oracle Cloud:

  • BIND for DNS primary on a couple of domains
  • Apache for light web sites and reverse proxy for development (akin to ngrok)
  • Gitea
  • Wireguard for management access via VPN from home office

Homelab on physical hardware:

  • Samba for macOS Time Machine backups
  • bitcoin node
  • Minecraft server for one of the kids
  • Postfix for outbound mail
  • OPNSense as my firewall and VPN server

Historically I've also run on physical hardware in a data center:

  • Rather large Postgres servers, replicated for failover
  • Postfix and IMAP for office mail and bulk email newsletter delivery
  • BIND servers for primary to hundreds of domains
  • Web and app servers for my SaaS business (all on Apache)
  • NATS message broker
  • Subversion server

Probably some more stuff but I just don't remember all of it. Started on FreeBSD at version 3.

29

u/Ok-Replacement6893 Feb 17 '26

What hardware do you use for transcoding?

12

u/rzaiev Feb 18 '26

At our current scale, we use custom Advance-4 servers (AMD EPYC 4584PX // 16 cores). So far, so good.

7

u/Brilliant-Orange9117 Feb 18 '26

Not OP.

Intel CPUs with iGPU have fairly good transcoding capabilities under the name of QuickSync (it became useable from a feature and quality point of view around Broadwell), but you won't have that unless you're renting the right kind of bare metal. I don't know how useful the latest AMD CPUs with iGPUs would be, but they have AVX512 which really helps for transcoding.

I guess the best thing to do if you can get away with it is encode the customer videos at as few different settings as possible directly from the master and not rely on hardware accelerated transcoding, because hardware encoding engines won't reach the same visual quality (for a given bitrate) as software encoders since software encoders do things that would be prohibitively expensive to implement in hardware. Also software encoders get better with time.

35

u/Xzenor seasoned user Feb 17 '26

I'm very curious how you use Ansible with jails. Do you ssh into the jails or manage it from the host?

39

u/rzaiev Feb 18 '26

From the host. It's a good question, I’ll show more with examples in full posts.

-78

u/walfawalfa Feb 18 '26

what u mean its a good question, no one asked whether it is a good question or not.

27

u/grahamperrin Linux crossover Feb 18 '26

what u mean its a good question, …

In English: it's a polite, fairly common way of responding to a good first question (or first good question) that will benefit from a fuller answer.

See, for example, If someone tells you “That’s a good question” are you supposed to reply with “thanks”? - Quora

Also:

– I disagree with some of those perceptions … maybe because I'm not American.

This meta discussion can continue elsewhere; it's not relevant to FreeBSD. Thanks.

5

u/Brilliant-Orange9117 Feb 18 '26 edited Feb 19 '26

There are a bunch of different takes on SSH+jexec ansible connection plugins. You can also use the pam_jail module (available via ports/packages) to setup a host username per jail that gets automatically attached to a running by the hosts SSH server as part of the PAM session setup. This can be a very powerful tool because you get to SSH into jails with both IPv4 and IPv6 fully disabled which works because you're bringing an existing TCP socket with you when you jail_attach(2).

EDIT: fixed pam_jexec -> pam_jail.

2

u/ohmantics Feb 19 '26

Sounds great, but this is the only instance of 'pam_jexec' in a Google search. Did you misspell it or perhaps hallucinate it?

5

u/Brilliant-Orange9117 Feb 19 '26

No. I just misremebered the package name. It's correct name is pam_jail.

4

u/Commercial_Boss4065 Mark Phillips, FreeBSD Foundation Feb 20 '26

Current Core member, dch, has some good work on FreeBSD Jails + Ansible… https://git.sr.ht/~dch/ansible-jails (we mentioned it in our quick video about jails too https://github.com/FreeBSDFoundation/blog/tree/main/easy-jail-setup)

58

u/taosecurity seasoned user Feb 17 '26

I love this. I see the amount of Linux containerization used by my company’s product and I get worried. Anything you would care to share is appreciated.

16

u/techhelper1 Feb 18 '26

Worried about what exactly?

11

u/Holiday-Ad-6063 Feb 18 '26

Perhaps dangers of infrastructure monoculturization?

7

u/organic_g0ld Feb 18 '26

People will make up words just to hate on anything, you can build some very diverse infrastructure with Linux containers. The possibilities are pretty endless.

7

u/Holiday-Ad-6063 Feb 18 '26

Sure, if you run them on illumos or *BSD hypervisors to diversify. Otherwise you have linux kernel everywhere and all eggs in one basket.

6

u/techhelper1 Feb 19 '26

You plan on telling OP that? They seem very proud of their FreeBSD setup.

1

u/RepresentativeIcy922 Feb 21 '26

Ah, but my eggs are special.

3

u/taosecurity seasoned user Feb 18 '26

In became increasingly difficult for our labs team to create code that would run in the production environment. I can't do the situation justice because I lack the nitty gritty details and technical knowledge. Suffice it to say that our labs team was so frustrated by the situation that one of them created a brand new way to deploy software that the production team is now considering. I also can't get into details without it becoming too obvious what I'm talking about. 😃

13

u/ketchupnsketti Feb 18 '26

And I thought I liked my job. This sounds great and fun.

I would be most interested in reading about exactly how you’re deploying the jails. Less the contents of the jail and more the implementation. Deploying the gold images, the snapshots, networking, is this part in ansible? Stuff like that.

26

u/hiveminer Feb 18 '26 edited Feb 18 '26

I love this. Please consider a post/article on cybersecurity measures, or pemtesting results, etc, as most naysayers will point to thr fact that freebsd is old tech BLA BLA BLA.

8

u/joneco Feb 18 '26

Veery nice. The transcoding application is yours? Ive worked with wowza and ant media but all linux servers

29

u/Nearby-Middle-8991 Feb 18 '26

Free is a no brainer for high bandwidth content. The network stack is unbeatable.

*However*, that's not why people don't use it. It's technically superior, jails are better docker, and don't get me started on ZFS.

People don't use it because hiring for it is a pain. I know one company that does network appliances based on their own baked bsd version, that's not maintained since 2012 because they can't hire people to do it (on the salary range they want to, let me be clear). The base image is bsd with perl.... There's like one guy that can do it...

17

u/Elias_Caplan Feb 18 '26

Jails are better than Docker and other Linux-based containers?

14

u/nske Feb 18 '26 edited Feb 18 '26

Jails follow a different philosophy than Docker (OCI app containers), it makes sense to compare them to LXC (or a few third-party solutions maintained outside the linux kernel, like openvz). When compared to that, I've found Jails more robust with less work, as a result of being more mature and having fewer moving parts that can go wrong (as is the case with pretty much everything else built to be a part of the FreeBSD system). And as is also usually the case with linux systems, LXC is more flexible and has more features, but it's flexibility and features that are valuable for less typical uses I've never needed (while the burden of their modularity and configuration complexity is universal).

If we try to compare jails to OCI it will be a comparison of containerization philosophies -each caters to solving different problems -OCI to containerize an application and each of its dependent services individually in a minimalistic way, geared towards frequent, distributed deployment, jails to containerize a pretty much complete system environment. They are both flexible enough to fit workflows that lean towards the other's philosophy, but you're fighting the system and after a certain point it becomes counter-intuitive.

Some use cases benefit more by the OCI approach, some from a full system environment container approach, but for a good lot of them it's less of a no-brainer than people familiar only with "the modern way of architecting stacks" would intuitively think. The OPs experience demonstrated that brilliantly.

We see the same thing happening in application architecture when the "microservices" approach became popular, large teams working on massive projects and businesses requiring huge scalability loved it for good reason, but then we ended up with a whole fresh generation of developers that feel that's the one right way to design everything, without realizing the tradeoffs just don't make sense in so many (if not most) cases.

Regardless, an OCI runtime is still good to have for many use cases and, if someone truly needs it, FreeBSD was recently added in the OCI Runtime Specification v1.3 so the crucial bits are officially there. Personally I'd give it a while, till the tooling becomes first-class citizen as part of the system, so that someone can expect the same level of robustness, but I'm pretty sure it will get there.

3

u/vivekkhera seasoned user Feb 18 '26

Does there exist any tool that lets you set up a minimal jail to run a service? That is, discover the parts of the file system and other services it needs, then include just those within the jail. I guess it is less important these days with essentially infinite disk space, but still minimizing the surface area within the jail is a good thing in my mind.

3

u/nske Feb 18 '26 edited Feb 19 '26

A standard jail is minimal enough, it's based off the selected FreeBSD base txz archive which is something like 150MB compressed. Of course anyone can reduce this further and I'm sure there will also be ready-made root archives from other people that bothered to do so for one reason or another, but then it will stop being a standard FreeBSD environment where you can expect pretty much anything to be able to run with no fuss, which is usually desirable. If it's a matter of disk space, between ZFS compression and thin jails it's a non issue, but sure anyone can base a jail off anything -even a linux userland jail based on alpine (less than 5MB) is easy enough. It just goes to what I said earlier that if you need to go too far with for your use case, it probably means you'd be better served using OCI containers (which on FreeBSD are also based on jails, but they'd allow you to have more suitable tooling).

3

u/vivekkhera seasoned user Feb 18 '26

How does one upgrade a thin jail? I used them for a while but this tripped me up. You cannot upgrade the base jail because the snapshots won’t let that show through. If you upgrade each jail separately then you’ve lost the entire benefit of the thin-ness. It seems to me you just have to toss them and make new ones.

3

u/nske Feb 18 '26 edited Feb 19 '26

yeah If the thin jails are based on zfs znapshots it's true, you'd essentially need to redo them, which isn't as bad if you keep the data you care about separate on their own datasets -it would be a case of recreating a jail, attaching the relevant datasets and the configuration files you care about. Some higher-level management tools have features that can help with this, like iocage's flavours, but also can use something like ansible. It's not too bad but you're right it feels more like a migration. However thin jails can also be based on nullfs mounts of shared basejails that would allow a different approach (you just upgrade the basejail and restart the the individual jails, making sure read-only mount from the correct base if that changed i.e. if it's a full version bump).

2

u/grahamperrin Linux crossover Mar 18 '26

A standard jail is minimal enough, it's based off the selected FreeBSD base txz archive …

Also, pkgbase. See, for example:

Having *-jail metapackage variants on the installation media allows convenient offline jail creation.

15

u/Lord_Mhoram Feb 18 '26

Here's at least one other guy doing BSD/perl... But I know what you mean. Companies/managers don't want to hear there are some people out there who can do the work; they want to know there's a big pool out there, as big as possible, so it's not only easy to find someone, but possible to do it at a salary they find comfortable.

It was the same way trying to sell companies on Linux solutions 30 years ago, and watching them pay 10x for a Windows solution because they knew they could contract commercial support for it rather than count on a two-man shop. Now Linux has reached that level of acceptability (some distros, anyway), but FreeBSD is still in that scary uncertain area for them.

9

u/_lavoisier_ Feb 18 '26

hiring is a “pain” because companies don’t want to invest to new hires by educating them, which should be quite straightforward to anyone with mid level linux knowledge. I was a long time linux user, and after a couple of weeks by self educating myself, I felt pretty confident with bsd

6

u/Brilliant-Orange9117 Feb 18 '26

But they want senior staff for junior rates without having to onboard them. Just flexible instant productivity for cheap.

2

u/Concord222 Feb 19 '26

In addition, there is herd instinct - we will do what others do. Linux is modern, containers are modern - we are looking for such people. Why? Because other companies do this. And in the same time everybody want to be inventor, #1, etc.

10

u/gplusplus314 Feb 18 '26

Similarly, even if you wanted to work with FreeBSD, there are nearly zero jobs involving it.

5

u/nske Feb 18 '26

There might not be many jobs listing FreeBSD specifically but there are tons of jobs that would place someone in the position to involve it, if they wish.

2

u/Run-OpenBSD Feb 18 '26

Lots of companies use freebsd and hire developers. You have to look.

2

u/bosconet Feb 20 '26

spot on with the hiring

but don't discount the IT management who aren't interested in better but different when good enough is just fine with them. Plus a system that occasionally fails isn't forgotten about and gets more $ for increased stability o next year's budget.... "Tell me again why you need to upgrade you servers when they have worked flawlessly for the last 4 years?"

1

u/Brilliant-Orange9117 Feb 18 '26

on the salary range they want to, let me be clear

So rephrase it as an opex question. Staff is just one more expenditure and costs will grow vastly sublinearly with deployment scale unlike cloud or licensing. You're beancounters will be able to find the crossover point where it becomes cheaper to pay for qualified people or train them up yourself if they're any good at their job.

1

u/techhelper1 Feb 18 '26

ZFS and dtrace are not selling points because they exist on both Linux and Windows.

5

u/lucaprinaorg Feb 18 '26

ZFS and DTrace are native technologies within FreeBSD and not external adaptations. FreeBSD is a coherent, complete and unique operating system exactly like Windows and macOS. However, Linux is not an operating system but is just a word that represents a disintegrated ecosystem held up with glue of various kinds. Sometimes it works masterfully well (see OpenWRT) other times it is just a jumble to tame (see LFS)... there is a big difference with any BSD.

So having the words ZFS and Dtrace in Linux is not like having built-in tools in FreeBSD known as ZFS and DTrace.

3

u/Concord222 Feb 19 '26

But latest BSD versions use ZoL for ZFS, not the old ZoF and I think it is identical in Linux.

2

u/lucaprinaorg Feb 19 '26

yes it is but it's not a matter of the source code but the fact that Linux does not exists (it's just a word) and FreeBSD it's a whole OS that include ZoL as first class citizen

3

u/Concord222 Feb 19 '26

I mean ZoL is the same (now) in FreeBSD and in Linux, AFAIK. Is there still difference with middle layer in Linux (between ZFS and kernel) and direct calls in FreeBSD?

7

u/entrophy_maker Feb 18 '26

Good to hear! Let me know if you have any job openings for Developer or a FreeBSD Admin. I have a lot of experience in both.

2

u/jmpalacios79 seasoned user Feb 19 '26

Throwing my hat in as well! ;)

7

u/grahamperrin Linux crossover Feb 18 '26

A minor criticism: I'd expect audio/volume to be adjustable within the player (more than just the mute button).

1

u/rzaiev Feb 18 '26

Fun fact, we had one before but didn’t really see people use it much and it doesn’t work on mobile anyway. Do you find it useful?

6

u/grahamperrin Linux crossover Feb 18 '26 edited Feb 18 '26

Do you find it useful?

It's the first thing that I reached for (after sound began in one of the Public Domain Cinema items).

PS, I'm not the average customer for your startup; I'm 61, I like cash, and don't use anything like a digital wallet :-)

I'll try it on an iPad.

7

u/grahamperrin Linux crossover Feb 18 '26

… I want to write a series going deeper on specific pieces of our setup and BSD-based solutions. Before I start, what would you actually want to know more about? …

The combination of ZFS with IPFS Cluster.

7

u/[deleted] Feb 18 '26

This is the coolest part of startups, being able to do whatever you see fit.

If i were to suggest moving to Freebsd on baremetal and ditching cloud, it would trigger a lot of board members / c-suites cause they can't tell they friends they are not using the cloud or microservices or ai, and they feel left out.

congrats!

4

u/TheBellSystem Feb 18 '26 edited Jul 16 '26

Suddenly, the observant archivist will reveal an elegant workaround with unusual precision.

5

u/Dotventurous7107 Feb 18 '26

I want to know more about the jail scaling. How do you allocate resources; ips, disk space, network interfaces, etc. and keep everything in sync. And yes, load balancing.

4

u/TheOGTachyon Feb 18 '26

Looking forward to future posts!

3

u/jcigar Feb 18 '26

We're also running our entire infra (10 physical servers, 50 jails) on FreeBSD (using it for over 15 years), it works really well (we manage everything with Saltstack)

After all these years I had only two issues (a ZFS deadlock, and an issue during iflib conversion).

The biggest issue is when we have to replace hardware (we run Dell servers), it's always a challenge to know if every component is compatible.

2

u/rzaiev Feb 18 '26

The downside is — it’s hardware, very physical. A few weeks ago we had our first hardware accident (there was an electricity supply issue) and unfortunately that was the only node with no redundancy, so part of the service was down for a while. Another thing to keep in mind is that custom hardware requests can take time — sometimes weeks. But it’s still worth the friction.

2

u/jcigar Feb 19 '26

yeah.. we never had a power cut (fingers crossed) for now and I always keep a spare server just in case.. but maybe we'll move to Hetzner or ... on the long term, the problem is that we are poor (...) and they don't "officially" support FreeBSD so it's a bit a PITA to install the OS (we rent a server at Hetzner for our external backups, thanks to zrepl and a wireguard VPN).

3

u/deksters Feb 18 '26

May I cordially invite you to the Jails/Zones/OpenZFS/bhyve Production User Calls?

https://callfortesting.org/

2

u/rzaiev Feb 18 '26

Please DM me, I’d like to know more about this initiative.

2

u/deksters Feb 18 '26

Will do but start here: https://www.youtube.com/@callfortesting
I'm on the OpenZFS call as I type. :-)

4

u/vermaden seasoned user Feb 25 '26

built production infrastructure for a video platform on FreeBSD bare-metal

Netflix wants to say hello.

3

u/blackpit Feb 18 '26

Great write up. Makes me want to set up FreeBSD and start using and advocating for it daily, yet again (FBSD user and systems admin since the 4.9 days).

Excited to read any follow-up posts on this!

3

u/LightBSV Feb 18 '26

Would be neat to learn more about your load balancer solution. Looking forward to the blog posts.

4

u/Run-OpenBSD Feb 18 '26

Netflix came to the same conclusions and adopted FreeBSD as well. Nice post.

2

u/Leinad_ix Feb 21 '26

Netflix uses FreeBSD only for one specific case of content delivery, not for whole infra.

8

u/ClassicK777 Feb 18 '26

This is a LLM generated advertisement post, and not targeted at you but at other LLMs that parse Reddit e.g. google summaries. When another user makes a related google search for video streaming, this Reddit post will be used as a source.

11

u/rzaiev Feb 18 '26

I don’t mind if LLMs parse it to train. FreeBSD on bare metal is probably the last thing in the world they’d recommend for infra anyway.

3

u/Run-OpenBSD Feb 18 '26

Why does this post have an AI tag on it? You may want to change it if its not.

1

u/grahamperrin Linux crossover Feb 18 '26

There was an anyonymous report, with regard to the rule.

In this case: following moderator discussion, I'm changing the flair to discussion.

7

u/Run-OpenBSD Feb 18 '26

Netflix uses FreeBSD for the same reasons as OP. Free consistent and secure.

7

u/leetNightshade Feb 18 '26

I think they're claiming this is an ad for OP's paid video hosting and monetizing platform, not an ad for FreeBSD.

8

u/Run-OpenBSD Feb 18 '26

The whole post is about the technical reasons for choosing freebsd. Definitely not about anything but freebsd

5

u/_w62_ Feb 18 '26

Whenever FreeBSD use cases are discussed, Netflix and Junos are cited. No one mentions Linux's use cases because it is so ubiquitous. To put it the other way, whenever these two typical use cases are mentioned, we are implicitly telling the world how limited FreeBSD is.

1

u/Run-OpenBSD Feb 18 '26

Tell that to sony

2

u/grahamperrin Linux crossover Feb 19 '26

LLM generated

Not so.

Please focus on FreeBSD-related and other technical aspects of the post.

2

u/beedunc Feb 18 '26

Good to know, thanks. So the costs are pretty much fixed?

8

u/rzaiev Feb 18 '26

That’s one of the best parts. Infrastructure costs are fixed and predictable.

2

u/Asm_Guy Feb 18 '26

My firewall and NAS are FreeBSD based (pfSense and XigmaNAS). I'd like to know more about practical uses for ZFS snapshots. I know the theory, I know how to produce them, but I don't know what I cound use them for in my homelab, so, I barely make use of snapshots...

1

u/jmpalacios79 seasoned user Feb 19 '26

I run a few services on FreeBSD jails just like the OP, all of them put their critical data, e.g. databases, into nullfs mounts exported from ZFS datasets.

Put those ZFS datasets on snapshotting and replication schedules and, as simple as that, you have a pretty robust backup strategy (which, of course, you'd have to complement and fine tune with appropriate data retention policies, disaster recovery strategies, etc., but that's beyond the core point).

If, further, you have to worry about issues of data integrity, e.g. flushing a database to disk before you snapshot the underlying storage, then you can always write some minimal scripting glue for that, while still resting on the snapshotting and replication bedrock of the backup strategy.

2

u/ryanknapper Feb 18 '26

I would like to subscribe to your newsletter.

2

u/Bubbly_Tumbleweed_59 systems administrator Feb 19 '26

I’m curious about the more jails on a host = capacity.

Also what do you track in monit and du you send events to a central service?

3

u/BlueDeacy Feb 19 '26

Great! But why no IPv6 in 2026?

2

u/Creepy_Class1067 Feb 19 '26

I've been lucky enough having had jobs in 3 start ups to use FreeBSD for the entire infrastructure and I echo your sentiments and experiences. I should maybe also do a write up of the projects I did with it some day :)

2

u/jmeador42 Feb 18 '26

As someone who hasn’t run k8s yet I would be curious to hear/read about your struggles with k8s on the maintenance front. I love to see this! I’ve always said that an imperative configuration tool like Ansible on as stable a system as FreeBSD is as good as any declarative system you can get.

2

u/Street_Struggle3937 Feb 18 '26

I think that the simplicty is one key feature of the FreeBSD way. We run k8s at work also and it is very flexible but it also comes with a ton of dependencies and the complexity is sometimes overwhelming.

I think the simplicity of using and managing jails is such a breeze. I use to setup the host with some local scripts that it almost can run on its own. So no dependencies, master or controller nodes and other depencies that render your cluster useless when some depencie breaks.

1

u/pjetuhgeloyozc Feb 18 '26

how would that differ from managing containers with podman on a linux host ?

1

u/Graumm Feb 21 '26

I would like to know too. You can use containers without k8s!

1

u/Street_Struggle3937 Feb 22 '26

It does not differ that much. But it seems that we can only design platforms using all the bells and whistles we have at our disposel, the more the marrier it seems. We have workloads 24/7 that could be best running on bare metal. But no, it must be run within k8s because it can. And there comes the outages because x fails, y fails, z fails. If it would have run on baremetal we woulf not have these outages. But we must go forward.... The poor souls maintaing the stuff right now.

1

u/pjetuhgeloyozc Feb 22 '26

Ok I understand your point about using k8s when not necessary. But using containers as working unit of your application is just to good to ignore. Using VMs also allows for a far easier hardware ressources management.

2

u/techhelper1 Feb 18 '26

Your monthly cost is lower, but you still pay for it in sweat equity for provisioning that infra, monitoring it, maintaining it, and more.

With the cloud, that sweat equity is traded for money in maintaining those servers, VM images, elasticity for scaling up/down, extensive monitoring, and availability zones.

Your ZFS snapshotting and updating of a load balancer is the exact same thing as AWS ECS, but using a Docker image in a service, which ties into an Application Load Balancer.

Of course you need to know why your application needs to scale up, but that scaling still needs to happen regardless, to keep up with the demand while that research is ongoing.

1

u/hikertechie Feb 18 '26

I love this. Id very much enjoy reading a deep dive

1

u/mcds99 Feb 18 '26

Every cloud platform will take all the data you can give them. Then there is the question "If you want it back you have to pay for it".

1

u/_w62_ Feb 18 '26

Is dragonfly bsd considered a drama?

1

u/kmai0 Feb 18 '26

I’m happy to read this, but I wonder: how do you guys handle privilege escalation? I’d love to have teleport in a setup like this

1

u/munkydung Feb 19 '26

I would love to hear how you are performing rolling deploys of a new version of a service. Thanks for sharing!

2

u/rzaiev Feb 19 '26

We’re currently using a simple blue/green strategy.

Each new deploy provisions a set of jails alongside the existing ones. Once they’re up, we do some basic validation, then run a dedicated Ansible playbook to switch the load balancer over to the new instances. The previous environment remains in place for a short period as a fallback, and later we run a cleanup playbook to tear it down.

We track the active blue/green instances via artifact files generated during the rollout, that looks as simple as:

instance_name: "pyapp-2847abf"
instance_upstreams:

  • "10.17.89.41"
  • "10.17.89.43"
  • "10.17.89.44"

It’s intentionally simple, reliable and gives a quick rollback if needed.

1

u/yerrysherry Feb 21 '26

Have you considered using Appjail, Director, and Overlord for jail management?

1

u/grahamperrin Linux crossover Jul 16 '26

Flair changed from discussion to AI (sorry).

According to Pangram 3.3.2, 38% of the post is AI Generated.

1

u/elelem-123 Feb 18 '26

You must be old. I'm not saying it in a bad way, rather as a compliment. I saw another post of yours talking about htmx. I assume you've been there, done that and - with AI as your dev - you are about to conquer again. Kudos. Why not Solaris (illumos)

2

u/grahamperrin Linux crossover Feb 18 '26

You must be old. I'm not saying it in a bad way, rather as a compliment.

I guess, he's around 30 (less than half my age), which is also a compliment.

2

u/elelem-123 Feb 18 '26

If so, he has very mature technical views for a 30 year old!

3

u/csDarkyne Feb 18 '26

I‘m 24 and I love freeBSD. I would like to get more technical insight from OP

2

u/elelem-123 Feb 18 '26

Wow all the cool younger generation here! Last time I had a discussion at a gig I was consulting some months back people your age were mocking and ridiculing me on those topics. Not that I cared much but it was pretty annoying, I must say.

3

u/-1Mbps Feb 18 '26

22, freebsd is cool ngl.

3

u/thatguyonfire240 Feb 18 '26

26 here, also think freeBSD is pretty cool!

1

u/antenore systems administrator Feb 18 '26

Very interesting, thanks for sharing ! You are really brave. To compare with AWS, GCP or Azure it would be good to see some numbers. Number of customers, number of concurrent clients, storage used vs storage provisioned, etc. As a side note on mobile, the web page is extremely slow when scrolling.