And there you go, you can install and run this locally now! I’ve previously tried a few
other projects that try to simplify the setup as well,
but this seems quite the nice project.
Note that I did need to use a workaround
to get the disk to be recognized correctly, but the comments say that you might not need
that.
I was recently trying to run a big docker-compose.yml setup using podman-compose and it took ages
to spin up and then didn’t work. But apparently you don’t have to, because podman system service serves up a Docker API replacement
which then can transparently be used by docker-compose.
First, start up podman system service --time=0 unix://$PWD/podman.sock somewhere to have podman
serve up the api.
Then, run DOCKER_HOST=unix://$PWD/podman.sock docker-compose up to start up docker-compose as
usual.
I didn’t play around with it for too long, but it seems to work quite well at a glance. Much
faster and prettier (colourful output, better separation between docker-compose vs container output).
Apparently this has been a thing since 2021
and this article has some info on how to
start this up as a systemd service. (I personally did not want to do that because I want podman
to run as non-root/my own user, but even that should be possible using systemctl --user.)
Note that 5.2 seems to be the latest version available, I am not sure if that is compatible with rocm 5.4,
which is what the AUR has.
And then you can check if torch thinks it has gpu acceleration support:
# should print 'True', otherwise something is up
$ HSA_OVERRIDE_GFX_VERSION=9.0.0 ./venv/bin/python -c 'import torch; print(torch.cuda.is_available())'
$ HSA_OVERRIDE_GFX_VERSION=9.0.0 TORCH_COMMAND='pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/rocm5.2' ./venv/bin/python launch.py --precision full --no-half
Python 3.10.8 (main, Nov 1 2022, 14:18:21) [GCC 12.2.0]
Commit hash: 685f9631b56ff8bd43bce24ff5ce0f9a0e9af490
Installing gfpgan
Installing clip
Installing open_clip
Installing requirements for CodeFormer
Installing requirements for Web UI
Launching Web UI with arguments: --precision full --no-half
No module 'xformers'. Proceeding without it.
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading weights [7460a6fa] from /home/luna/t/stable-diffusion-webui/models/Stable-diffusion/sd-v1-4.ckpt
Applying cross attention optimization (Doggettx).
Model loaded.
Loaded a total of 0 textual inversion embeddings.
Embeddings:
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Don’t forget ./venv/bin/ in front of python, that tripped me up a couple times.
Generating a batch of 10 images, this is what I got for gothy witch in the forest with a large pointed hat with stars on it, 4k, detailed, gloomy, cat sitting on a tree branch:
Which shows that writing prompts is not trivial either, I suppose.
And that might be how you can get this to run on a laptop. Worked for me. The only thing left is to contemplate
the ethics/motivations for replacing artists with “ai” magic.
I prefer humans, so here are some whose art I like:
These are the best cinnamon buns I have ever made, and the softest home-made dough I have ever tasted. Really flaky and soft like store-bought stuff, only made at home and without preservatives and other magics.
You will need a bunch of time, because the dough rises once overnight in the fridge and then 1-2 hours before baking the next time.
Author’s note: Bon Appétit had a big implosion of structural racism in 2020, with most of their POC and women on-video people leaving, including Sohla El-Waylly. If you want to support them, you can find those people doing things on their own now! For example, Sohla does regular things with NYT Cooking and elsewhere now.
The recipe
This is the adjusted recipe as I am making it, read below for how I arrived here from the original one.
7g active dry yeast (about 2¼ tsp., one small package in europe)
¼ tsp. baking soda
1 tsp. salt
375g all-purpose flour
filling:
180g dates
2 tsp. ground cinnamon
glaze (optional):
3 tbsp yoghurt
1 tsp. vanilla paste or extract
80g powdered sugar
Instructions
tongzhang
add 100g of yoghurt and 20g of flour to small pot
whisk at medium heat until it has a sticky pudding-like texture
be careful here, this burns quickly if not paying attention
take of heat immediately, transfer to different container to not cook it further!
make the dough
warm remaining yoghurt and 6 tbsp oil to 36°C (about body temp, should not feel hot or cold)
combine with yeast and 50g sugar
mix dry ingredients (remaining flour, baking soda and salt) for dough
combine yoghurt mix, dry ingredients and tongzhang; mix until a ball forms (will look very wet at first)
knead & fold
put on unfloured surface (dough should still be wet and sticky)
knead, pushing away from you and then pulling it back towards you; until a smooth ball forms (~3 minutes)
if too sticky, oil hands (and maybe surface)
don’t add flour!
roll into 20cm square, fold twice in half to get a 10cm square
roll back into 20cm square, fold again like before to 10cm square
let it rise!
oil bowl with 1 tbsp. oil
add dough, turn over to coat in oil
cover tightly with cling foil
rest overnight for 8 hours (up to 1 day)
make the filling
???
simpler alternative:
cover rolled out with a nice layer of butter
sprinkle with sugar and cinnamon to taste
prepare the dough
punch down dough! (ouch?)
roll out dough on unfloured surface to 20cm square
fold twice again to 10cm square
roll out to 30cm square, about 0.6cm thick
assemble the buns!
grease round cake tin/ceramic thing with remaining 1 tbsp oil (9 buns, ~25cm diameter should work)
cover rolled out dough with filling from above; leave 1cm or a bit more of free space at the farthest end (to “close” the buns at the end)
if using the date filling, sprinkle remaining sugar on top
roll into tight roll from closest edge to the one with the free space
cut off ~1cm at each end for clean finishes with serated knife
i like to bake the ends with the regular buns and then use them as samples
slice into three equal sections, then each section into three buns
wash knife in between with hot water for cleaner cuts
place buns side-down into greased cake tin
cover with aluminium foil or lid, let rise for 1-1.5 hours until doubled in size
should spring back and leave a small indent when poked
buns should be pretty close together in the tin and touch so that they grow taller when baked
bake ‘em!!
pre-heat oven to 180°C
bake covered for 20 minutes until “puffed, pale and mostly set”
remove lid/cover
bake 15 minutes uncovered until golden brown for soft and squishy buns
Add glaze if you want one; and then serve still warm! Eat soon (same day) for the nicest texture!
Modifications
I like to change things a bit, or somethings have to:
if I don’t have dates, some butter + cinnamon + sugar works just as well as a filling
I omit the glaze from the original recipe, because I like things less sweet and less sticky
I recommend mixing the dough by hand and not in a food processor as suggested in the original recipe. I broke my food processor because it was not powerful enough, as did several other people in the comments.
The Story
The original recipe uses buttermilk and eggs, which are not a thing in vegan baking. So we need to replace them somehow.
First, Tangzhong to the rescue! Tangzhong is basically a little pudding made from some of the water and flour in the dough, which somehow binds liquid and keeps it in the dough, making it softer and fluffier than it might otherwise be. This replaces the egg in the recipe, which seems to play a similar role. (I am guessing here, )
The flakyness comes from using a good amount of oil in the dough, which is worked in after kneading by covering the dough in oil and then folding it a couple of times. (Twice!)
And finally the buttermilk we can just replace with vegan yoghurt, which seems to work well enough.
Wait! Actually the most troublesome part is converting the measurements from metric to grams and then calculating how much liquid and flour we are using in the tangzhong. This amount we then need to subtract from the original recipe because tangzhong does not add or take away any mass from a recipe, it just uses some of the liquid and flour in different ways.
So you’ll need a scale as well because this seems to be a rather precise thing.
The Math
Replace egg and add tongzhang while keeping hydration the same.
One large egg = 56.7g, has ~75% water content => $56g * 0.75 = 42g$ water that needs to be added into the recipe.
Our hydration: $42g / x = 0.6$ => $42g / 0.6 = 70g$ flour for tongzhang taken from total amount of flour. (Thank you to J for listening to me talking this through, which made me realize this calculation made no sense at all.)
When we use tongzhang we need the hydration to be 75% instead, so we need to increase the amount of water in the recipe.
So: $375g\ flour * 0.75 = 281g\ water$ to get 75% hydration. So we need $281g - 222g = 59g$ additional water in the recipe. But because we don’t have the egg we also need 42g more to add the water content of the egg back in?
Finally we usually use 5-10% of the flour in the recipe for the tongzhang and the tongzhang is 1 part flour to 5 parts liquid. So $375g * 0.05 = 18.75g$ of flour to $93.75g$ of water, or up to $375g * 0.10 = 37.5g$ of flour to 187.5g of water.
It might be simpler, quoting from the same article again:
I’ve now made this standard slurry often enough that this is what I use for any yeast recipe calling for between 3 and 4 cups of flour: 3 tablespoons (23g) of the flour in the recipe + 1⁄2 cup (113g) of the liquid.
Remember, you’re using flour and liquid from the recipe, not adding extra flour and liquid! Take that into account when you’re measuring out the remaining flour and liquid for the dough.
Our recipe has 375g = 3 cups of flour, so maybe let’s go with that. Now only to decide whether to add the additional liquid for the higher hydration in water, milk or yoghurt. Let’s go with yoghurt because that is what we are using anyways.
I have a service running in Kubernetes
on my server that needed some tending debugging, so here’s how that
went and the little trick that was needed for it.
Usually I’d just use kubectl exec and be on my way, but there were three issues with that:
I wanted to access the db and sqlite was not installed in the container
The service runs as a non-root user, so no installing anything in addition
The live container was pretty locked down, containing only busybox and my binary
$ kubectl --context live debug -it numblr-c67cd998f-69ktm --image=alpine:3.15 --target=numblr --share-processes`
# try accessing the data of the live pod
/ # ps aux
PID USER TIME COMMAND
1 1000 0:25 /app/numblr -addr=0.0.0.0:5555 -debug-addr=0.0.0.0:6060 -db=/app/data/cache.db -stats
67 root 0:00 sh
76 root 0:00 ps aux
/ # ls /proc/1/root
ls: /proc/1/root: Permission denied
# replicate the live user
/ # apk add --no-cache shadow && useradd --home-dir / --shell /bin/sh numblr && apk del shadow
...
# run sqlite as that user for access!!
/ # apk add sqlite
...
/ # su - numblr -c 'sqlite3 /proc/1/root/app/data/cache.db'
SQLite version 3.36.0 2021-06-18 18:36:39
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE feed_infos ( name TEXT PRIMARY KEY, url TEXT, cached_at DATE , description text, error text);
CREATE TABLE posts ( source TEXT, name TEXT, id TEXT, author TEXT, avatar_url TEXT, url TEXT, title TEXT, description_html TEXT, tags TEXT, date_string TEXT, date DATE, PRIMARY KEY (source, name, id));
CREATE INDEX posts_by_author_and_date ON posts (author, date);
CREATE INDEX posts_by_author_and_id_and_date ON posts (author, id, date);
And off I was, with access to the live db and able to run some EXPLAIN
QUERY PLANs and so on!
Two key things here:
Access to another container’s files is possible through /app/$pid/root.
Container filesystems are visible to other containers in the pod
through the /proc/$pid/root link. This makes debugging easier, but it
also means that filesystem secrets are protected only by filesystem
permissions.
To access those files you need the same user as that other container,
being root inside your debug container is not enough.
… say no more, I have one now, here’s what I did to make that happen!
Whyy?
Well. Because.
At least mostly. Here are some reasons:
i want to try it
i want some more experience with Kubernetes
i can
Basically I want to see how difficult it is, if it is viable for me. And
maybe it might even be nicer than the manual server management that I have
so far.
I may also have been at KubeCon 2022
virtually this last week and may have wanted some more experience before we
restructure things at more. But the “automated the setup” aspect is pretty
tempting as well.
Some constraints
As usual, I have some odd choices and constraints:
no publishing ports open to the internet
i use caddy and a firewall, i don’t want to
open up new ports and neither do i want to replace caddy (yet)
as little as possible admin
and as little as possible setup hassle
How?
In short, the cluster runs k3s, the ports are published on a dedicated
internal ip using kube-vip; and all of this took one evening of
frustration and a morning with a clearer head to figure out.
Note: ⚡ This setup has been running for a few hours only, I don’t know if
it is secure enough, fast enough, whatever enough. But it runs and now I
can try some more things. ⚡
k3s
k3s is a small-ish Kubernetes distribution deliverd as one binary and I
think easier to set up than a full blown cluster. E.g. the database is a
simple sqlite database and there’s no fun distributed etcd stuff to setup.
Running k3s server gives you a Kubernetes cluster.
I installed k3sfrom the AUR.
Do note that the k3s-bin package is lacking the k3s-killall cleanup
script, which you will need to clean up the networks, iptables stuff and
containers that k3s starts up. I pretty much always ran the
k3s-killall script when I was changing IPs, testing network settings and
similar stuff like that. If in doubt, get a clean (network) state by
running k3s-killall.
The devil is in the (network) details. Because I don’t want to publish
ports on my public IP, which is not something that k3s seems to support
out of the box.
I tried a lot of things, perused the docs over and over, and in the end
just used kube-vip which was linked in a GitHub issue. But here are some
of the steps that I tried:
--disable traefik (traefik listens on :80 and :443, which was
overriding my caddy server for reasons)
various --flannel-backend and --flannel-iface settings
--https-listen=other-port, which did not work at all and is another saga in itself
block ports using ufw
For some reason the default load balancer that comes with k3s goes
around ufw, likely by being earlier in the iptables chain.
(#2059)
and more
In short, mostly I don’t understand how networking works properly, iptables
even less; but there was also something odd going on.
kube-vip
I had enough.
Luckily, kube-vip seemed to be doing what I want: Allowing me to specify
an interface an IP address that ports will be published on. In particular,
instead of the builtin LoadBalancer implementation that comes with k3s we
use kube-vip, setup according to their docs.
Add a new internal ip for Kubernetes: sudo ip addr add 192.168.0.101 dev lo
Note the IP and the interface (lo) are custom and you can choose what
you need there, e.g. for my live setup it has a different IP and listens
on the actual network interface. (But because it’s a different IP ports
on that IP are not reachable from the outside.)
Setup some permissions that kube-vip needs when running as a daemonset:
Finally, run k3s server with the correct parameters to use that IP:
$ k3s server --node-ip 192.168.0.101 --advertise-address 192.168.0.101 --disable traefik --flannel-iface lb --disable servicelb
INFO[0000] Starting k3s v1.23.6+k3s1 (418c3fa8)
INFO[0000] Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s
INFO[0000] Configuring database table schema and indexes, this may take a moment...
INFO[0000] Database tables and indexes are up to date
[...]
With all of this we disable traefik and the default load balancer
(servicelb), and replace the latter with kube-vip which listens on the
IP and interface we have specified.
ufw
We do need to expose one port on our firewall if we want to manage the cluster from the outside using kubectl:
To route things to the outside, we can expose a port to the host
on the IP we have set up:
# Expose on port 15555 to the host.
#
# With this setup the port is then routed outside the host in some other way,
# e.g. using Caddy outside of Kubernetes.
apiVersion: v1
kind: Service
metadata:
name: numblr
spec:
selector:
app: numblr
ports:
- protocol: TCP
port: 15555
targetPort: http
type: LoadBalancer
I have regular services running already and Caddy set up, so I just want to
add services to that for now. In the future I might play with setting up
traefik + cert-manager so that subdomains and certificates are exposed
automatically, replacing Caddy completely.
What now?
Now I have it running live, serving a staging instance of
numblr that has a copy of the live
database. It runs okay so far, but there seem to be some wrinkles I have
to investigate still.
What’s neat is that I can now say kubectl apply -f kubernetes/deployment.yaml when
I want to deploy a new version, and I don’t have to do a little scp + ssh
manual service restart dance. And I can add new services in the same way,
only having to tell caddy that there’s a new port to proxy on some new
domain.
I think that’s pretty nice, let’s see how it turns out.
Edit from the future: A week later I am pretty happy so far. Better
tooling, easy deployments, easy (and fast) access to logs, … And a
really nice way to debug things, using ephemeral containers.
Quite the nice workflow so far, even when used in the home.
Fin
That’s it! Have a nice day, I have some flowers to plant now.
So I’ve been making fancy coffee for a little more than a year two
years (?!) now, here are some quick tips and tricks I’ve learned.
Quick note: These are my experiences and are what I enjoy, written more
opinionated than I actually am. Please do ignore me if your experience
does not match up! If someone tells you they know how things are and you
are doing things wrong – they are likely wrong and probably not fun to be
around!
Timing is everything!
have a recipe, follow that
time things, gives me guidance and a nice sequence to fall back on if confused
especially for espresso, go by time first, then taste (if
beginner/having trouble with tasting for what you want/like)
for flair signature: 15s preinfusion (after first drops), pull shot at
6-7bar and aim for ~30s shot time (without the preinfusion)
for v60: 17g coffee + 250ml, heat filter, add grounds, preinfuse (60g
water, 45s), slowly pour until 250ml; aim for 4:30min total time
(timing less important here than with espresso)
Over time I’ve discovered that when I make espresso + mjölk things I
usually don’t need sugar. With filter (+ mjölk) I usually want sugar, but
sometimes less.
ignore that it has to taste of a wistful feeling at sundown, just try
to find out what you like
i have trouble identifying what coffee tastes like, but I still have
fun with it and discover things over time
don’t require yourself to taste what it says on the label (I can’t in most cases)
Keep at it
there might be fun jumps in knowledge/experience over time
e.g. I got much better at latte art after a few months, kind of from one day to the next
Have fun
That’s what makes it enjoyable for me, and keeps me making elaborate coffee
things. Having these odd coffees with friends is also really fun for me,
because they tell me fun things about what the coffee tastes like, often
much more interesting things than I can taste.
For more, see the other things I have written about coffee-related topics.
Recently, I wanted to play Stardew Valley
with a friend. However, said friend lives a while away and thus we don’t
have a LAN. But now I have a VPN, and we can play together, no matter
where we are! (Sadly Stardew Valley co-op mode does not work on mobile,
that would be even neater.)
With all of the things below, I mainly followed the instructions on the ArchLinux
wiki. There was some fiddling required, but all-in-all this was the work of
an afternoon to set up.
Setting up ufw
For some reason I did not have a firewall running on my server yet. That’s
a bit irresponsible and was mentioned on the ArchLinux wiki, so I did set
it up:
$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), allow (routed)
New profiles: skip
To Action From
-- ------ ----
<REDACTED>/tcp LIMIT IN Anywhere # SSH port
443/tcp (WWW Secure) ALLOW IN Anywhere
Anywhere ALLOW IN 192.168.0.0/24
<REDACTED>/udp ALLOW IN Anywhere # WireGuard
80/tcp (WWW) ALLOW IN Anywhere
Note the <REDACTED> ports, those are non-standard ports for both SSH
and WireGuard, redacted for some security by obscurity here. Do be careful
with that SSH port though, because you can lock yourself out of your
own server. Luckily I did not.
However, my server needed a restart to make these settings take effect. Not
sure why, but that’s what it needed.
Setting up the WireGuard server
As we only want to connect to each other, not provide a full VPN, this is
the config file for WireGuard. The keys were generated using the
instructions on the wiki.
$ sudo cat /etc/wireguard/wg0.conf
[Interface]
Address = 10.0.0.1/24
ListenPort = 51820
PrivateKey = SERVER_PRIVATE_KEY
# if the server is behind a router and receives traffic via NAT, these
# iptables rules are not needed
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o ens3 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o ens3 -j MASQUERADE
[Peer]
# client1
PublicKey = CLIENT1_PUBLIC_KEY
PresharedKey = SERVER_CLIENT1_PSK
AllowedIPs = 10.0.0.2/32
[Peer]
# client2
PublicKey = CLIENT2_PUBLIC_KEY
PresharedKey = SERVER_CLIENT2_PSK
AllowedIPs = 10.0.0.3/32
(Note that all values for PublicKey, PrivateKey and PresharedKey have been redacted,
you’ll need to fit the actual values in if you want to replicated it.)
Setting up WireGuard clients
Network Manager has built-in support for WireGuard, which is pretty neat. Here’s
how to connect a client.
First, set the network name (wg0) and private key for your client that
is used to encrypt all traffic to the VPN server:
Secondly, set your IP address in the VPN to 10.0.0.x and use network mask 32.
Gateway will be marked yellow but works fine when left empty.
And finally configure the actual server (called “peer” here) you are
connecting to. “Public key” is the public key of the VPN server, “Preshared key”
is another secret you’ll get from the server for an additional layer of security.
“Endpoint” is the host and port of the VPN server, written as host:port.
You should then be able to save and activate the connection. You should now be in
the VPN and able to connect to other clients in the network using 10.0.0.x IP addresses.
And with that we could play Stardew Valley together! One player is hosting the
farm and the game, and others connect to it using the game hosts IP address in
the VPN.
![screenshot host]()
![screenshot player join]()
(TODO: I know those screenshots are missing. I did play as pictured above,
I just forgot to take the screenshots and then did not do it and now I
want to get this post out.)
Conclusion
All in all this was much simpler to set up than expected! I am kind of amazed
this is even possible to set up in a reasonable amount of time for someone who
does not do sysadmin on a regular basis.
I now know they’re called hub gears, but that makes no sense to me.
A while back I forgot to fix the bar that makes my backpedal break work to
the frame of my bike. That was… no fun, because when I tried breaking
the next day it whacked into the bike frame and got stuck. I could not
drive anymore at all because my back wheel would not spin at all anymore.
A few days later I found a fix that temporarily
“solved” this problem using a hammer. My bike was driving again!
But it was driving weirder. I felt like my back wheel was wiggling much
more along its axis and it often did not want to switch between gears
properly. It just felt very fragile, but I was scared of trying to open
the hub gear and what I might find in there. Or if I could ever put it
back together.
But come this weekend, I have taken it apart entirely, I had a nice
afternoon walk to my local hardware store, cleaned the parts, and will put
it back together tomorrow.
What helped me was nlzs youtube channel. I found several videos
detailing how to recenter the axis to fix the wiggling, how to take the hub
gear apart, how to clean it, and how to put it back together. All in nice
short videos that succinctly show how to do all this with clear images and
tips for the trickier parts.
I may have gotten quite deep into coffee in the last year.
And thus I had a problem: Given that my information is mostly from
english-speaking regions, when I’m lucky from the UK, and when I’m unlucky
from the USA. Most shops and suppliers that are talked about there are at
best impractical here, so what do I do? As an aside, I try to avoid amazon
if possible, so the easy options there are out.
This is my little collection of tips and shops that I found useful so far.
First, your city may have local roasters and coffee shops that sell coffee
beans and equipments! I found some locally and now have “my own” roaster
from whom I get most of my coffee.
If you know me and you are interested, feel free to ask which roasters and
café’s I’ve been to!
Given that I want to avoid amazon, I had to relearn a bit how to find
things On The Internet. Funnily enough I found https://idealo.de quite
helpful. I can sometimes find shops there that are difficult to find using
regular search, and they let you search by product name.
To find local specialty cafés I found https://europeancoffeetrip.com/?s=Germany
to be quite helpful, they listed a few cafés that I did not find via
searching and have some pictures of the cafés they list.
Shops that deliver using DHL (because “Packstationen”):
They also sell coffee and some equipment. Have bought equipment from
them which was nice, they even sent a small hand-written card, so I
think they’re still a small shop.
They seem to roast only a few times a year and then have that coffee on
their website for a long time. I am rather confused by that, because
as far as I can tell coffee “degrades” as soon as it was roasted, even
if it packaged.
I have not tried their coffee though, so they may still be a nice
option. I certainly liked the small touches in their deliveries.
They mostly sell equipment related to doing latté art. They seem to be
the only german seller of “WPM” milk pitchers, which are supposed to be
fancy and things. And they sell those “loveramics” cups that they use
in barista competitions.
Oh, and if you want certain Timemore products (like their scale), you may
be better off using https://aliexpress.com rather than amazon. I ordered
my scale there for half the price that was shown on amazon. It took a few
weeks, but arrived without issue.
And that’s it, so far! I’ll update this when I find new interesting things.
So I’ve been trying out Neovim again. I’ve been interested a few
times in the past, but when setting up a new laptop I decided to try
it out for good.
In the end I was inspired to try again by lots of README’s for Vim
plugins mentioning it and by an unrelated post somewhere that also
mentioned Neovim.
So here I am, a little more than a week later, with regular vim
removed from all my systems to properly get used to it. It’s been
fun!
What’s been especially fun was finally making my setup reproducible
again by putting the various Vim plugins into my regular dotfiles
as git submodules.
And also, my fresh Neovim config is much shorter than the fun
collection of random things my .vimrc was before (which was 219
lines of things accumulated over 10+ years, most of not used anymore
for a long time):
-- looks
vim.api.nvim_command('colorscheme peachpuff')
vim.api.nvim_command('set number')
-- highlight character that was jumped to correctly after lightspeed motions (https://github.com/ggandor/lightspeed.nvim/issues/66#issuecomment-952888479)
vim.cmd[[highlight Cursor ctermfg=NONE ctermbg=NONE cterm=reverse]]
-- fast navigation with fzf
vim.api.nvim_set_keymap('n', '<Leader>q', [[ <Cmd>b#<CR> ]], {noremap = true})
vim.api.nvim_set_keymap('n', '<Leader>b', [[ <Cmd>Buffers<CR> ]], {noremap = true})
vim.api.nvim_set_keymap('n', '<Leader>e', [[ <Cmd>Files<CR> ]], {noremap = true})
vim.api.nvim_set_keymap('n', '<Leader>s', [[ <Cmd>Ag<CR> ]], {noremap = true})
That’s all of it. Some colorscheming so that it works on terminals
with a light background [^1], a fix/thing for one of the plugins I
use, and fun keybindings for fzf.vim which I use to switch
navigate around quickly.
Do I like neovim? Yes! Is it much different than vim? Not that
much. Do I want to switch back? Not at the moment.
I think the nicest thing overall about Neovim is that it has nicer
defaults, e.g. when pasting from elsewhere it seems to pick that up
automatically and paste things without indenting all over the place.
There’s probably lots more things to configure and play around with,
but so far I haven’t needed to, and that has been quite nice.
[^1]: Siderant: Why do some programs need extra config for that, it’s a bit annoying.
So, you just watched James Hoffmann doing another glorious oh no face
and naturally you need this in your life.
However, you need subtitles, an acceptable file size, and more. Let’s
get to it!
You download your video using youtube-dl, possibly in a more
sensible file-size (--format '[height <=? 720]') and then also the subtitles
using --write-auto-sub --skip-download.
And then comes the magic, which will result in fun things like this:
Which was generated by calling ./gif-it.sh 3:06 3:13.5 Coffee\ Substitutes\ -\ Tasted\ and\ Explained-KArQ3mBzWC4.mp4 ooh-no.
I could not figure out how to make ffmpeg output a GIF and video, so the first
step is a reencode to render the subtitles. It also cuts the video to size, because
we only want that part anyways.
The filter pipeline is interesting, if impenetrable to debug for me if things break.
See the post by Giphy below for a detailed explanation.
Finally there’s an MP4 of the GIF for smaller file sizes.
Note that here we have -vf format=yuv420p, because Firefox does not play yuv444
videos, which was what mpv selected by default when converting from the GIF.
Speaking of file sizes, for the 8.5 second GIF we get the following:
$ ls -lh ooh-no.*
-rw-r--r-- 1 luna luna 3.2M May 5 17:45 ooh-no.gif
-rw-r--r-- 1 luna luna 195K May 5 17:45 ooh-no.gif.mp4
-rw-r--r-- 1 luna luna 760K May 5 17:45 ooh-no.mp4
Not too bad, especially the .gif.mp4, and given that I am not a GIF professional.
That’s it, enjoy, this time with sound:
Resources:
most importantly, James Hoffmann’s
YouTube channel! He makes delightful nerdy videos about all things coffee.
This allows me to turn on my laptops monitor, put the window I want
to record there, and then select that section of my desktop to
record from.
I could not get that to work in obs-studio running on Sway.
However, it works really nicely in GNOME when obs-studio is nudged
to actually run on Wayland using QT_QPA_PLATFORM=wayland obs.
However, obs-studio seems to use much more resources, so the incantation
is still the best way to go for me.
Note the Cache-Control header manipulation: I could not get Jitsi
to enable caching for everything on its own, so I am having Caddy
do it. I could probably cache more resources, but the JavaScript
files are the heavy ones (>1mb) and make things fast enough for now.
And that’s it! You now have a working Jitsi setup!
Is it working better though?
Maybe. Subjectively yes, but the video will still not update sometimes for a bit
while it catches up to the network.
It does! It’s pretty neat, even at version 0.3, and it makes this
whole “setting up JACK” thing actually possible for me.
As you may recall,
getting JACK to work is a bit of a trial on a good day. So much so,
that I did not use it at all, even after getting it to work, because
turning it on pretty much means turning off PulseAudio [^1].
You can put a reverb or a delay on all incoming sounds, making those same meetings much
more entertaining.
There are kind of no limits. You start up carla or catia, reroute audio at will,
and then off to the giggles again.
Fin
So, pretty neat! It’s still quite buggy sometimes, but also quite
awesome. I really like it, and the exciting part is that it makes
me want to do more music things, actually record something, just
play around, …
🎶
[^1]: Yes, I know it’s possible to combine them, but it was still a
hassle to get to work every time. It’s possible, but was too much
for me.
This is an interesting approach to teaching complex topics, in this
case Quantum Theory. It is both a text that is teaching the topic,
but also an essay about the educational approach behind it.
I really like these types of experiments with mediums, like Bret Victor
did famously several times. Very interesting. Especially because it
is about teaching mathematics, which is often presented in difficult
to approach manner.
To quote the quote within the text:
Mathematics is a process of staring hard enough with enough
perseverance at the fog of muddle and confusion to eventually break
through to improved clarity. I’m happy when I can admit, at least to
myself, that my thinking is muddled, and I try to overcome the
embarrassment that I might reveal ignorance or confusion. Over the
years, this has helped me develop clarity in some things, but I
remain muddled in many others.
William Thurston, Fields-medal winning mathematician; as quoted by
qcvc
To math is to stubborn, one might say, in a silly enough mood.
You know those TODOs and FIXME left throughout your projects,
never to be uncovered again and only to be wondered about by
librarians far into the future?
No? Maybe just me.
Anyways, since a few months I made my git status shortcut remind me about them:
$ git st
## main...origin/main
M explorer.go
explorer.go: // FIXME: restrict paths to only boards/
static/board.js: // FIXME: potential confusion because query overrides config, even if config is more recent
static/board.js: // TODO: notify about override in yaml somehow
static/board.js:// TODO: support refreshing automatically
static/board.js: // TODO: always display timestamps in utc
static/board.js: // FIXME: support display of all datasets (only displays one so far)
(Note that TODO and FIXME would be highlighted in red.)
This works using a git alias and a bit of shell fun:
Find packages that have been explicitely installed. Wait…
Oh yes, as I am writing I realize that pacman does this already (as it should) using pacman -Qe.
Ah well, learn something new every day or something. Coming up,
neat git tricks and finally writing down what I do (differently)
when installing ArchLinux. (I forget every time.)
Display packages taking up more than 100MiB on your system.
Most of this incantation is concerned with bringing both the name of the package and its size
onto one line. The last grep then filters out the big packages. So you could adjust the size
threshold by adjusting that last regular expression.
I have been working on some Arduino-related things recently, which involved displaying some things
on an OLED display (this one). However, I was very disappointed by how slow it was.
It took about 30ms per character to update, which was really disappointing. Knowing not much
about actual hardware, possibilities, or how slow/fast it should be, I just continued on, assuming
I’d just have to work with it.
It turns out I do not.
The display in question is an I2C display, connected on the A4 and A5 pin on the Arduino Uno
board I’m using. And initially, it really was connected on those two pins, because that is what
I found in the code sample from the seller:
#include <U8x8lib.h>
// this will be hella slow, mind the _SW_ constructors
U8X8_SSD1306_128X32_UNIVISION_SW_I2C u8x8(/* clock=A5*/ A5, /* data=A4*/ A4);
The problem is that it was doing the I2C communication in software, which apparently is really
slow. The funny thing is that on the Arduino the pins actually have multiple purposes,
and A4 and A5 in particular are also special I2C pins, then called SDA and SCL respectively.
And using a different configuration for the u8x8 display library, it suddenly was about ten
times faster with updates. About 30ms for a “full screen” update now, and 2ms for smaller
updates.
So… if you want your OLED display to be really slow, do as I demonstrated. Otherwise, mind the
u8x8 constructors with SW (i.e. software) in the name, and stick with the ones that use the
actual capabilities of the hardware.
I’ve been having problems with getting external USB audio interfaces to work on my
computer. I have a guitar and wanted to plug it into various external interfaces,
and it didn’t work properly.
In the end, the trick was to tell jack to use a different audio interface for input
than the one for output.
This is how the config looks in cadence:
In the above, note that “Input device” is hw:1, whereas “Output device” is hw:0. For me, that’s the trick that worked in the end.
In addition to this, set up realtime audio, by installing
the realtime-privileges package and adding yourself to the realtime group. (Don’t forget to also set it in jack/qjackctl/cadence.
Hints:
use aplay -l to figure out if your sound card is recognized correctly
have a look at the “Input Devices” in pavucontrol, if your audio interface appears there and is shown
receiving sound (via the level that is displayed there)
look at dmesg -w for kernel/driver messages
(Note to self: The image above was taken using scrot -s, and then compressed using tinypng.com.
convert image.png smaller.webp also worked, made the image go from ~21000 bytes to ~10500 bytes. But tinypng brought
it down to ~8500 bytes.)
I used this to find out which libraries that a certain program (or
library) is linked to have missing dependencies. In my case I ran a
partial update for icu, and that broke lots of things (mostly qt),
and then I had to run out and find out which libraries still used
the old version of icu.
Replace the argument to the ldd call at the beginning with the
program you’re trying to get running again.
Combine with:
| less at the end to scroll and search through (search for “not found”)
pacman -Qo with the library that has missing dependencies
But there was also another article, about writing an strace-equivalent (and quite a bit more) using ptrace (2): (It gets into emulating syscalls for different systems and faking/disallowing system calls even.)
And switching topics, entirely, here’s an article about returning to ballet and learning. I really liked that one, because I also struggle trying to learn new things, which seemed to be easy when I was a bit younger. But often when I try to practice something, I do learn a little bit, even if it doesn’t really feel that way.
Today geary couldn’t start up, saying the database was potentially corrupted. There was an option to just rebuild the database file, but as I like my history of emails and didn’t want to download all of them again, I gave up.
At first. :)
As it turned out, there was a weird empty geary.db-journal file right alongside the database, and it was owned by root. As geary (and sqlite) couldn’t access it, it couldn’t do any database modifications. (According to https://sqlite.org/tempfiles.html, that file is used for “rollback journals”.)
In the end, I just deleted the file (geary.db-journal), and geary worked again.
Here’s my approximate debugging session:
# start geary, see an error
# hm...
$ geary --debug
[...]
unable to open database [...] 'CREATE TABLE CorruptionCheckTable (text_col TEXT)'
# hm, let's try it manually
$ sqlite3 ~/.local/share/geary/<my-account>/geary.db
sqlite> CREATE TABLE CorruptionCheckTable (text_col TEXT);
unable to open database
# oops? no idea what's going on, let's do something else/more important
# ...
# search for "sqlite create table error unable to open database file" on the web
# find http://fredericiana.com/2014/11/29/sqlite-error-open-database-file/
# file permissions?!
$ ls -la ~/.local/share/geary/<my-account>/
-rw-r--r-- 1 lu lu 374161408 Feb 2 14:43 geary.db
-rw-r--r-- 1 root root 0 Feb 1 10:18 geary.db-journal
# why is there an empty root-owned file here?
# don't know, let's delete it
$ sudo rm ~/.local/share/geary/<my-account>/geary.db-journal
# and it works again
“But their narratives are not yours. And just like they get to have their own, YOU get to have YOUR own. And your narrative is that they are unreasonable, selfish, and frankly being jerks about this whole thing.
Those two narratives are not mutually exclusive. They exist simultaneously. Nothing needs to be done about them; they simply are.” – Laura at captainawkward
Mentions many of the tropes that people in tech write about, but it’s also a good collection of arguments and reasons for working less.
I still work too much, but I am happier when I work less. Getting used to taking regular breaks is what I’ll try next, because usually I try to push through problems (and then push more), ending up tired and fatigued at the end of the day, not even having accomplished much.
Addressing a group of students (or colleagues) as “ladies and gentlemen” would subtly imply that their gender was their defining feature: that before all else, I regarded them as gendered. Whether or not my language was suitably inclusive, I was rather missing the point. I wouldn’t address a lecture hall as “Hullo, English Studies students, and those of you taking Joint Honours degrees!” I wouldn’t begin a seminar with “I’m delighted to see you, British and EU citizens, and also our International students from other regions.” I certainly wouldn’t usually begin a speech “Dear gay and lesbian students, heterosexual students, bisexual students and asexual students”. In each case it would sound as if I was making a particular point. And my audience would be entirely justified in raising an eyebrow to question exactly why I was implying that those distinctions mattered about them.
Which is to say, another time for a few links to interesting stuff.
Magic Paper looks like an interesting approach to exploring math and other abstract concepts. There are connections to Bret Victor’s and Seymour Papert’s work, which is always neat to see.
Create Rust games easily by Lisa, a very nice talk about creating games with ggex (”good games ezily”), a library written in Rust inspired by löve. The talk was really fun to watch, and it inspired me to write down an idea for a little game of my own. So… watch this space, that might be a thing soon.
By quitting mpv with Shift+Q instead of just q you can continue watching where you stopped later.
The --save-position-on-quit option enables this by default. However, the position will only be saved when quitting, not when skipping to the next file or anything else.
This may have been the most exciting cooking magic I’ve learned this year so far!
If your pancakes haven’t been properly fluffy, have you tried putting the lid on? Because you totally should! The recipe is fairly standard (basically flour, sugar and baking soda), but this worked out great. (I don’t have photos, because we ate all of them.)
Or: An Incremental Approach to Compiler Construction
I appear to be writing a compiler. It’s not entirely by accident, but it’s not entirely intentional either. I’ve been interested in compilers for a long time, but I haven’t learned assembly, so most of my experiments have been compiling to different languages (like C), and interpreters.
But now I’ve found a paper that builds a compiler in 24 incremental steps:
An Incremental Approach to Compiler Construction, by Abdulaziz Ghuloum
It’s about writing a simple compiler for a sizable subset of Scheme (up to an interpreter), to raw x86 (32 bit) assembly.
I think it’s awesome!
My compiler targets x86_64, because that’s what my laptop is running. For now that seemed to amount to using the wide registers whenever pointers/memory locations are in play, which means at several steps I got loads of segfaults.
I’m currently at step seven, which introduces heap allocation, and with that several types that aren’t representable with just a stack. (Or maybe they are, but not without many difficulties, at least in this compiler.)
It’s fun, it’s challenging, but it is also doable, which means that I (mostyl) understand what it is doing, without having much experience with assembly. I did know Scheme and compilers, though, but in theory this should also be possible if it were a compiler for Python. (In fact, I might port the first few sections to Python, so people will look at that, and then hopefully continue with the rest of the paper.)
A taste of the compiler
So, how do things work?
In the beginning, there was nothing. Except, the paper starts with just numbers, just a function returning a numeric constant:
While that may seem like a whole lot of gibberish, what’s important are three lines:
// scheme.s
scheme_entry: ; define a label called "scheme_entry"
movl $42, %eax ; move the number "42" into the "eax" register
ret ; return
And now, we can call it from C:
// driver.c
#include <stdio.H>
extern int scheme_entry();
int main(int argc, char **argv) {
int val = scheme_entry();
printf("%d\n", val);
return 0;
}
And sure enough, it prints 42:
$ gcc scheme.s driver.c -o scheme
$ ./scheme
42
To me, that was amazing! I didn’t know a thing about assembly, and here I was, writing a compiler, which about a week later supported ifs and let. (That was yesterday, today it’s learning about heap allocation.)
It’s not all sunshine
Here come the caveats.
You will have a much easier time if you can work in a 32 bit x86 environment, because that’s what the compiler in the paper targets. With a little more effort and debugging segfaults it is possible to port it to 64 bits, but if you can get a compiler for a 32 bit environment, I’d recommend that.
The paper is rather sparse in places. For example, it explains how to encode numbers and shows code for that, but then it leaves you on your own to write the code for booleans, characters and the other types. It’s doable, but it seemed a bit daunting to me at first. However, I think that was very much a didactic decision to not include all the code, because that will require you to think about what you’re doing.
I also think I may have found a few mistakes, but I’m not entirely sure about those, partially because I work on a differrent architecture. But it works.
Resources
Want to write your own compiler as well? Great, here are some pointers to helpful things:
Nada Amin’s version of the compiler, which includes a draft of an extended tutorial, that might have more hints
this also includes the full code for the compiler, for x86
the [edb debugger][], which is really useful to debug segfaults, as it value visualize the registers nicely, and even allows you to edit the code in place!
and my version of the compiler, unfinished and likely broken, but existing nonetheless
Note Taking During One-on-Ones, which is something that I might want to start. My notes are very unorganized (one might say random) so far. Also, I’m not a manager
As the title said, it’s a guided tutorial on how to build your own editor in C. It’s amazing! You’ll learn about C, the esoteric innards of consoles, text editors, memory, …
The tutorial looks really good, progress is being made in small steps, every new line is explained – I think it’s amazing!
So if you’re interested in C or always wanted to know how a text editor in a terminal might work, here’s your chance. It’s fun!
The predict_linear in Prometheus uses linear regression to predict how a time series will evolve in the future. So you can alert when there is still time to fix things.
This is something that the SRE book from Google talks about. Alerting when there is still time to fix things, when stuff isn’t broken yet.
In general, that book seems really interesting. I do find it a bit difficult to read because it is very abstract as well, but interesting nonetheless. (I think it is a bit abstract because the engineers writing it wanted to talk about generally applicable principles, and not the tools they use at Google. However, these tools are somewhat crucial, so it would help to get some hints what kinds of technologies would help implementing something like what they describe in the book.)
Let’s talk about JavaScript. I don’t admire it, I just use it. When I need to make a request to the server in a web page, then I just write out plain old JavaScript without any libraries to make an XMLHttpRequest. It’s not that pretty, but most of the time it works pretty flawlessly.
I know what to do. I create it, I .open it, I know how to send query parameters (the new URL object is pretty neat, but pasting the query string together also works for me), I know how to send POST requests (FormData is your friend).
Similarly, when I need a web service, I’ll write some Go. Usually without any libraries. In fact, one of the things I like about it is that i don’t have to. The standard library is fine (I’d say excellent), and it works pretty well for these things.
And again, I know my way around. I know the interfaces, I know how to lock something with a Mutex should I need to, and so on. It’s almost boring, except that it’s not.
The list goes on. I don’t hate PHP, I like plain old Ruby, I appreciate ObjectiveC a bit, I can write Python, I’ll even dive into C if I have to, or sometimes if I just feel like it.
And still, I have also written non-trivial amounts of code in Haskell, Rust, some Erlang, and plenty of Clojure. I also like those languages, but I’m not religious about them. Sometimes I want them, but often I am fine with this tooling I have around.
I just like plain old boring technology. In fact, I find it exciting, and interesting that these kinds of boring things lead to interesting results. There’s something there, I think.
So what’s my point here, then?
My point is that I don’t like fighting over technology. Highly opinionated pieces about how language X or library Y are the best thing ever, or worse, how Z is the worst thing ever – these kinds of articles aren’t really interesting to me. Sometimes I’ll read them, but I prefer reading different things.
I love hearing about language X helped solve a specific problem. Or how library Y helped the author develop a solution much faster. I like hearing about techniques and trade-offs.
So maybe that’s what I’ll write about next. Maybe I’ll write about Go and how its standard library helps me with all kinds of things. Or how I write these small (or sometimes rather big) pieces of JavaScript that enhance simple web pages. Or I’ll finally write Saved by the Shell.
Or not. After all, I have boring opinions about tech.
(Aside: I think what I’ve written here is already too opinionated. But alas, it was fun to write, and it does express a bit how I feel. So here we go.)
Clean up untracked files from the repository, including files ignored by git (via .gitignore). Rerun --force instead of --dry-run to actually remove these files. There is also an interactive mode (-i).
These are two “open” machine learning projects (kinda), which I think is an increasingly important topic. I hope these will help individuals and open communities use machine learning around things like voice recognition and speech synthesis, as opposed to only being able to use these by ceding control to big software companies.
This piece is not really only about the issue discussed, but more broadly about how to respond and react to criticism from less privileged communities, and how to factor in your own privilege in that response.
If you live in a place where you are guaranteed free speech, calls for space and respect like this aren’t censorship—they’re calls for consideration. You still have the power and right to make whatever decision feels best for you, but my hope is that you will prioritize the expansion of kindness and reduction of harm in the process.
And a personal note: I didn’t nod along with this article at first, but at the very least I really agree with the consideration towards criticismm, and the way it has been handled.
Bringing the web up to speed with WebAssembly surprised me by showing that WebAssembly is academically interesting (as well), and has a couple of neat properties, like a memory model that is designed for security.
Yachae Kalguksu (Korean Knife Noodles with Vegetables), which were delicious.
I needed a lot more flour than I thought I would (while folding) so that the folded noodles don’t stick together. I really liked this, and will cook something involving these noodles again soon.
For the past few weeks I’ve had only sporadic internet access, which resulted in many tabs accumulating, with articles preloaded to be read later. As it turns out, they only seem to accumulate further, so here I’m trying to clean up and collect the ones which I found interesting.
Growing up alongside tech, a nuanced view on learning, having opinions, and how these relate to tech and tech culture.
As a tangent, the sentence “We need extremism, but eventually we want nuance.”, popped into my head.
I don’t much like the way this article is written, but I do agree that the context package is quite invasive. I’m not sure if it needs a language change, but it is worth thinking about. On the other hand, the context package has only been helpful to me so far. However, it was only helpful because it was integrated into the standard library. So… I don’t really know.
How would one cancel an io.Copy(...) call, though. Closing the underlying io.Reader/io.Writer should work, assuming that the Read/Write calls then return an error. Hm, more to investigate, I guess.
Pascal at Apple, which has an amazing comment about the deeper history and many (!) more further resources.
I recently moved to new flat, and didn’t (and still don’t) have a full kitchen for a week. What I did have was a simple water cooker and a mixer. As it turns out, you can cook delicious things with them, you just have to kinda know what kind of recipes to look for.
In short, vegan and raw. At least, that’s how I ditched eating out (which I can’t afford doing more than maybe twice a month at most), and “cooked” neat things.
So, here is my current bag of tricks:
salad (kinda obvious, but I was too lazy/tired to properly do this for a while)
things that help: fresh herbs, grains, semi-fancy extras, like olives or tofu
simple soups
cauliflower as a raw couscous replacement
vegetable “noodles”
And if you have a fridge (which I finally have), you can make fancy cakes, deserts, …
I continue to use just-add-boiling-water type meals sometimes, but I am much happier since I discovered I can still cook fancy things without a kitchen.
Oh, and I am also planning on making some raw vegan cakes & cookies. Very curious how they will turn out. I might make some of the following:
Doing this with possibly millions of domains and even more millions of rules immediately becomes a bit more complicated.
They ended up with a simple yet fairly accurate approximation of the number of requests per ip, based around a counter stored in memcached, incremented asynchronously. There’s quite a bit of other fun stuff, like how they ensure traffic goes to the same server (anycast?), and the other algorithms they went through.
If the second tc qdisc add command does not work, you might want to use tc qdisc change instead. And when you’re done tc qdisc del is your friend as well!
In Go, a common complaint is the lack of generics, among other things. This article demonstrates that there are a lot of challenges to be solved if Go would get generics, and most of them is not about generics, but about the implications of having them.
These are some of the knock on effects that I see of adding generics or immutability to Go. To be clear, I’m not saying that it should not be done, in fact in my previous post I argued the opposite.
What I want to make clear is adding generics or immutability has nothing to do with the syntax of those features, little to do with their underlying implementation, and everything to do with the impact on the overall complexity budget of the language and its libraries, that these features would unlock.
(Emphasis mine.)
(This article is a follow-up to an earlier article about the concept of “Simplicity debt”, which is somewhat similar to technical debt, but occurs because something was implemented for the sake of simplicity, and later it turns out that things weren’t as simple. Another example of this is the GIL in python, as mentioned in the article.)
GraphQL, RPC, and bespoke clients are all pretty plausible ways forward to a post-REST world, but whichever we choose, we shouldn’t forget the lessons that REST taught us in that convention and widespread consistency are powerful things. If we do adopt something new, we should aim to make it as ubiquitous as possible so that we don’t worsen developer experience by fracturing technologies. Unfortunately though, realistically that might mean just sticking to REST.
A fascinating walkthrough for the new storage format that is going to be used in Prometheus 2.0. Starting with the requirements, it describes the current solutions, the problems with it, and the solution they implemented.
After all, it is synthetic and thus not worth much beyond a good first impression. Starting out about 20x above the initial design target, it was time to embed this into an actual Prometheus server, adding all the practical overhead and flakes only experienced in more realistic environments.
Some things are “standard” database techniques, like batching writes as much as possible, moving away from the one-file-per-series scheme, not touching old data at all, a write ahead log, using a data layout so mmap can be used, … But there is also an inverted index for the queries, which is usually used in text indexing, and a scheme for intersecting sorted ids that reminds me of the leapfrog triejoin algorithm.
This course is a very practical introduction to deep learning. It is focused on writing code and achieving state of the art results. Intended as a 7-week course, each lecture presents a different “end-to-end” project. For example, the first lecture implements a full image classification task, deciding whether a picture has a dog or a cat.
Some characteristics:
approachable for “non-math” people (and in the future non-programmers)
state of the art results
self-contained projects each week
better understanding by discouraging copying and pasting
each week is provided as a notebook
Intended to be read, and it is recommended to then try writing it out all on your own, only referencing the notebook when stuck. This way developing own intuitions and actively learning the material.
Notes for the first lecture
The code from the lecture uses an existing pre-trained model for the ImageNet dataset to create a classifier that is able to classify arbitrary user-specified categories.
The lecture uses a dataset from Kaggle, an algorithm competition platform, but that’s just one way to get a dataset with suitable pictures. (Kaggle is a pain to set up, because they are afraid that people cheat on their competitions. To get the dataset you have to get an account, agree to the rules of the dogs-vs-cats competition there, and give them your phone number so that they can send you a verification SMS.)
the lecture uses a very specific directory layout:
data/
<name-of-dataset>/
train/
<category1>/
<img1.jpg>
...
<category2>/
...
<more categories...>
valid/
<layout same as train/...>
For example, if you want to distinguish between cats and dogs, you’d have a layout like this:
data/
cats-vs-dogs/
train/
cat/
(images of cats here)
dog/
(images of dogs here)
valid/
cat/
dog/
How to get a development environment
The course seemingly requires a computer with an Nvidia GPU, and suggests to use an AWS instance, but you can run everything locally:
install anaconda2 & dependencies (requires ~2.5gb of disk space):
The title is a bit misleading, but I like this approach to versioning/dependency management.
In this system, every library has a name, which is a string like “foo” or “bar”, and a version, which is an integer. This is a marked contrast with most other systems, where the version is a string or some more complicated data structure. The Amiga’s rough equivalent of dlopen has a signature like this:
loadLibrary(name string, minVersion int)
The idea there is to be as simple as possible but no simpler.
And, to make updates work:
For this to work, there is a strict social contract that library authors have to abide by: don’t break existing users. The rules are, 1) all changes must be compatible, and 2) increment the version number in every release. […]
So what if your Amiga library does need to make a backwards-incompatible change? After all, it’s not always possible to predict how a library will be used in the future. What once was a good design might prove problematic some day. In that case, you pick a new name. Period. If your library is “foo” you can call it “foo2” or “bar” or whatever else you can think of. It is, mechanically, a whole new library now. Clients then must opt in to your new library by changing their loadLibrary call to specify the new name. They will pick up new versions implicitly and automatically, but they’ll only pick up a new library explicitly and consciously.
I like this very much! It’s simple, makes things explicit, and (tries to) avoid breaking libraries.
You don’t even need to make it a function, it’s enough if the version manager uses the same scheme. minVersion, and new library names on breaking changes.
This is a neat explanation of how the bayesian theorem works applied to real-world examples.
But it is also saying that “experimentation is key”, and that beliefs that you 100% believe in (like not feeling good about yourself, or earning a low wage) are not necessarily true, and that you might just have convinced yourself of them by (accidentally?) not seeing different examples.
Hm, that doesn’t sound quite right. But this quote might explain it better:
I didn’t say it explicitly in the video, but in my view the Bayesian trap is interpreting events that happen repeatedly as events that happen inevitably. They may be inevitable OR they may simply be the outcome of a series of steps, which likely depend on our behaviour. Yet our expectation of a certain outcome often leads us to behave just as we always have which only ensures that outcome. To escape the Bayesian trap, we must be willing to experiment.
In any case, the video was quite interesting, and the scenery kinda dream-like.
A neat little collection of tricks to make an interpreted, dynamically typed language fast.
The short version is: a compact value representation, fixed object layout, copy-down inheritance, method signatures, computed gotos and a single-pass compiler.
The entire thing seems to be inspired by Lua, which is also known to be quite fast.
The author of Wren, Bob Nystrom, also wrote a book about implementing languages, which looks to be really neat as well. The existing parts are already excellent, but for the unfinished low-level details you’ll have to look elsewhere for now.
When your child says “Why am I not allowed to do this thing?”
Instead of defaulting to “My house, my rules!”
Try actually communicating a legitimate reason, because children pick up on subtlety and on context and on the unspoken messages, and it’s better to teach children lessons like “You should think really hard before taking on new responsibilities” and “It’s important to show consideration for the needs of the people with whom you share a living space” than lessons like “It’s okay for people to demand your absolute obedience so long as you’re dependent on them for survival.”
Our goal is going to be to ask questions about technical concepts that are easy to answer. I often have somebody with me who has a bunch of knowledge that I’d like to know too, but they don’t always know exactly how to explain it to me in the best way.
If I ask a good series of questions, then I can help the person explain what they know to me efficiently and guide them to telling me the stuff I’m interested in knowing. So let’s talk about how to do that!
Or, “How to ask questions so that you get good answers”.
My favourite: “Ask questions that show what’s not obvious”, with the goal of unearthing hidding knowledge and assumptions.
Editorial note: I didn’t properly read this article for a long time, because I didn’t take the time to actually read it actively, not just hurriedly to go and read more things. Before you can even ask questions, you need to make space for your own thoughts. And for me, the “only” way to do this is to do nothing else. No listening to music, no news/aggregator/social media/…, and usually also no one else around. Walking away from the computer often helps, as does writing things down. In this specific case, sleeping and then only reading the article in the morning did the trick.
When you struggle with your mental health on a daily basis, it can be hard to take action on the things that matter most to you. The mental barriers anxiety creates often appear insurmountable. But sometimes, when you really need to, you can break those barriers down.
Really neat, may also be helpful with other kinds of calls. (Spoiler alert: It definitely is! At least for me similar things help with calls and other things.)
In Over The Garden Wall (which you should see, probably), the first episode starts with Greg enumerating rejected names for the frog:
Anawalt Guggenheim
Albert Salami
Giggly Jumpy Tom Thomas Tambourine
Leg-faced McColon
Artichoke
Penguin
Pete
Steve
“Wait, wait a second […]”
At the end of the first episode, the frogs name is “Wirt”, which is also the name of Greg’s brother, who will henceforth be called “Kitty”. (Which is also my name, by the way.)
“What, maybe I’ll start calling you ‘Candypants’.” - “Woah, yeah!”
The article is a bit all over the place, but it does give a good summary of several arguments on why productivity might be misleading, how important unallocated time is and more.
It feels especially relevant to me as a software engineer, where daily people are “discovering” how they could be more efficient and everyone else should be, too. I don’t believe in that anymore, I think it’s rather harmful. Some techniques might help, but I don’t think any particular one can help everyone. Or rather, the technique we have to learn is introspection, and self-awareness, not how to organize ourselves.
In the end, as in the article, the question is what we decide to do with our time. I don’t have a definitive answer for me, but I do know that I do not want to spend most of it “working”. I want to help people, for example, but that should not come at the cost of not being able to help myself.
"You could dump the boyfriend and get a cat. It wouldn’t do any chores, but least the cat would be cute and hang out with you sometimes."
(And from lots of other places, see youtube-dl --list-extractors for a list.)
# Download a video
$ youtube-dl https://www.youtube.com/watch?v=aZvDe3box70
# Loop the video
$ mpv --no-video --loop=inf --end 3:47 'Shaban - Ungleichung-aZvDe3box70.mkv'
Note the --loop=inf parameter. And --end <timestamp> is also very helpful.
Also, don’t forget that mpv can play any video that’s downloadable by youtube-dl directly! (But we can’t use that for looping, because then it would download the file repeatedly.)
One of our engineers came up with a useful script to grab all unique lines from the history of the repository and sort them according to entropy. This helps to lift any access keys or passwords which may have been committed at any point to the top.
That’s about what the commandline above does.
Here’s entropy.rb:
#!/usr/bin/env ruby
def shannon_entropy(s)
d = {}
s.each_char do |c|
d[c] ||= 0.0
d[c] += 1
end
res = 0.0
d.each_value do |v|
freq = v / s.length
res -= freq * (Math.log(freq) / Math.log(2))
end
res
end
if __FILE__ == $0
$stdin.each_line do |line|
e = shannon_entropy(line)
puts format("%.4f\t%s", e, line)
end
end
The comment is from a Hacker News thread about a recent disclosure of (very few) private repositories on GitHub.
Another comment in the same thread points out that Shannon Entropy was used for that, which I then ported to Ruby.
And now, you can search for “interesting” lines in your repository. Have fun with what you find! :)
It’s still hard to get started. I know Clojure (the language) well
enough, but learning the tooling is much more difficult for me. I’ve
tried to write simple APIs before, but the problems were similar each
time.
Documentation
It seems that documentation for Clojure libraries is hard to find
on the web. ring and compojure both have generated docs, but
they are simply listings of the namespaces and the symbols in them,
without top-level examples.
So what I mostly did was a combination of reading examples, glancing
at the source code of different libraries (ring-defaults, compojure-api,
ring-middleware-format, …), and failing to get anywhere and trying
again a few days later.
(I should have been using ciders support for displaying
documentation more, but that wouldn’t have helped with discovering
which libraries to use.)
Middlewares?!
There’s a bewildering choice of ring middlewares to try. ring
itself brings a lot of them with itself, but there are other useful
ones that come from other places.
However, finding them is mostly a matter of luck, I think. I’ve
started out with ring-defaults, but it doesn’t do content
negotiation and too much other things.
Where wrap-params and wrap-keyword-params come with ring itself,
and wrap-restful-format does content negotiation.
To wrap-restful-format uses the Accept and Content-Type headers
to decide how to interpret requests and responses. In your code you
simply set :body to some data, and wrap-restful-format will handle
the conversion.
How did I learn of wrap-restful-format? I stumbled upon it while
trying out compojure-api,
which in turn I randomly found by searching for “compojure” on clojars.
I haven’t yet found a good way to catch errors. There are some
middlewares for that, but I want one that does content-negotiation,
and I don’t know if any support that.
Also, I haven’t yet found out how to selectively respond with HTML if
requested, and otherwise API data. That would be very helpful for
API endpoints that should also have a UI.
Neat things
This is mostly an aside, but both compojure-api and Nightlight
are neat projects. With compojure-api you get automatic documentation
for your API, and can even try it out there easily. Nightlight gives
you an IDE in the browser. In theory that’s really cool, but it seems
to be lacking for documentation support at this point.
Assorted Emacs tips
By default, the macros from compojure get indented very strangely,
but put-clojure-indent can help. For example, to indent the GET
macro properly, use the following:
(put-clojure-indent 'GET '(:defn))
;; ... and so on for POST etc.
Here (:defn) is an indent spec
which allows properly indenting even complex macros.
Another thing that often tripped me up were how comments are indented.
A single ; comment is indented to the side.
This is the default and when using ;; instead
the comment stays at the indentation level of the code surrounding it.
Updating the dependencies in project.clj apparently does require
restarting the leiningen processes. In my case, that means rerunning
lein ring server-headlessand restarting the cider connection
in Emacs.
Additionally, if you’ve not used cider for a while you may still have
its plugin in your ~/.lein/profiles.clj file. This is not necessary
anymore.
Find .txt and .md files that were accessed within the past week.
The -atime -7 controls the time here. It can also be used to find
files accessed more than a week ago (-atime +7), or exactly a week
ago -atime 7.
The other interesting part are the parentheses, which are used to
group the -name options together, so that -or works only on
those two. If the parentheses were not used, -or would have
either matched everything before it or everything after it.
-maxdepth 3 is used so that the search space is small enough.
With it enabled, the command completes almost instantaneously:
0.07s user 0.06s system 98% cpu 0.139 total
Whereas -maxdepth 4 is already much slower.
0.31s user 0.97s system 37% cpu 3.403 total
This works on my system because my notes are in relatively high-up
directories.
Update:
Instead of the command above, I now use one that sorts the files
by access time and returns all files it finds, not only the more
recent ones:
ls -1t $(find $HOME -maxdepth 2 -type f \( -name '*.txt' -or -name '*.md' \) \! -path "$HOME/.*")
This runs as a cronjob and its output is redirected into $HOME/.recent.txt,
where it is then read by Emacs.
The concerns seem to be an uncertain roadmap, potential patent
problems with the license of react/react native, and having to
write all code in JavaScript.
Compiling to JavaScript is not really mentioned, but dismissed more
or less as follows:
To my mind, there is a generalised obstinate refusal to see JavaScript as inherently deficient and therefore needing replacement rather than polishing. The resulting proliferation of grafts and crutches is seen as signs of a vibrant ecosystem, whereas what it really shows is that the language is lacking in fundamentally important functionality.
Xamarin and Appcelerator are suggested as more stable, and likely
long-lived competitors. They might be more stable, but apart from
“acquired by …” and “got … in funding” no arguments are made why
they will exist (and be maintained) longer.
…
So, I don’t know. I’m still interested in ClojureScript, which
allows/inspires libraries like Om,
and can be used with React Native. And I definitely don’t buy the
“let’s write this in Swift instead” angle.
In my opinion React Native is at least an interesting idea, and it
being implemented in JavaScript might not be ideal, but it doesn’t
stop me from using interesting languages such as
Elm or ClojureScript on top of it.
pkgsrc can install a full UNIX system inside
another system in any location, as (I think) any user that can
access that location.
It might be more “practical” than Nix, and
might be interesting for deploying software with complex
dependencies? (Without using something like Docker.)
There are more interesting things, just one example:
Best of all, the binaries in /home/ourapp/root can reasonably be
expected to run on any version of Linux available in the past 15
years: simply extract a tarball of the build to the target machine.
blog is a tiny tool that generates your (link)blog.
It takes a YAML file as input, and produces
a single HTML file on stdout, which you could then upload to your
server, place on your Desktop, or pass along to friends.
blog is not meant to be a feature-rich program. It does the
bare minimum necessary to host a blog with different post types,
and not more. Whichever additional features you need you can
add to your version of it.
How to use it
All posts are written in a single file blog.yaml, which contains
a list of entries.
The most basic post type is text, written as follows:
- title: An example post
content: You can use *Markdown* here...
Optionally you can specify a date field.
If content starts with a non-alphabetic character, you need to
start the value with a vertical bar |:
- title: Special characters...
content: |
*This* post starts with a special character.
There are a few other types of posts:
shell, similar to text, but title is a shell command
link, with an additional url field
image, where url is the source of an image
song, where url is the source of the song
With the exception of the shell type, title and content
are optional.
Find infrequently used binaries on your system. ls --time=atime is the key here, it uses the access time instead of the modification time, which is the default.
