moving to freebsd / jails / bhyve (part 9): fail

i really gave it the college try, but i realized something towards the end…

  1. i used proxmox on this same server for 3 years without any major issues. the only reason i replaced it was because i wanted a 1u for proxmox that can run an a310 for jellyfin. i never had to grind through proxmox like i did with freebsd.
  2. on this same hardware that worked flawlessly for years with proxmox, i find that freebsd will simply freeze up under high network load. i suspect a nic that isn’t playing well with the fbsd driver but there is zero information i can see from logs, debug logs, or dmesg that says “that’s the problem” so i can fix it. right now i have five jails and they run great for like an hour or two but then it goes under high load and just freezes up everything.
  3. every single thing i have done mostly felt like i was fighting it. there are a lot of “nicky nack” things that you bump into that you need to constantly address that simply work in linux. this was not the case with freebsd.
  4. what the hell with needing a libc6 shim to make ffmpeg work? 😂
  5. how on earth is there no sane way to mount smb / cifs? the only way i could do it was with rclone which is wonky in its own right. in proxmox all i had to do is tick a box to allow a container to use it. rclone also auto mounts in an lxc container. i could not figure out how to get it to mount in a jail without doing the @reboot thing as a crontab job.

i learned there are a ton of things in linux that are taken for granted. fbsd has an amazing development team and i’m sure for someone who has a job maintaining freebsd servers its is great. but for someone trying to run a home lab on one server using fbsd + jails…. just use proxmox until it is in enshitified.

i will say one thing though. i have an ancient isilon that runs freebsd but it is super basic and only does storage. for that use case… it can’t be beat. but getting it to do the same things that vmware, proxmox, and maybe even smartos do? right now i think it’s unrealistic.

either way, i’m going to mothball that server until i figure out something else to do with it. maybe i’ll give openindiana (and zones) a go when i have time.

a side note: i was trying to make a linux jail just to see if it worked. there isn’t much of a selection. i think ancient versions of ubuntu or debian (maybe fedora? i don’t remember). either way they ran into a usr is share error so i gave up on that as well. i was considering running everything in bhyve but i think i lost my will. hope the stuff i did write helps someone some day in the future. good luck if you’re trying.