Accessing files using ephemeral containers∞
I have a service running in Kubernetes
on my server that needed some
tending debugging, so here’s how that
went and the little trick that was needed for it.
Usually I’d just use
kubectl exec and be on my way, but there were three issues with that:
- I wanted to access the db and
sqlitewas not installed in the container
- The service runs as a non-root user, so no installing anything in addition
- The live container was pretty locked down, containing only busybox and my binary
Ephemeral containers to the rescue!
Here’s what was necessary in the end:
$ kubectl --context live debug -it numblr-c67cd998f-69ktm --image=alpine:3.15 --target=numblr --share-processes` # try accessing the data of the live pod / # ps aux PID USER TIME COMMAND 1 1000 0:25 /app/numblr -addr=0.0.0.0:5555 -debug-addr=0.0.0.0:6060 -db=/app/data/cache.db -stats 67 root 0:00 sh 76 root 0:00 ps aux / # ls /proc/1/root ls: /proc/1/root: Permission denied # replicate the live user / # apk add --no-cache shadow && useradd --home-dir / --shell /bin/sh numblr && apk del shadow ... # run sqlite as that user for access!! / # apk add sqlite ... / # su - numblr -c 'sqlite3 /proc/1/root/app/data/cache.db' SQLite version 3.36.0 2021-06-18 18:36:39 Enter ".help" for usage hints. sqlite> .schema CREATE TABLE feed_infos ( name TEXT PRIMARY KEY, url TEXT, cached_at DATE , description text, error text); CREATE TABLE posts ( source TEXT, name TEXT, id TEXT, author TEXT, avatar_url TEXT, url TEXT, title TEXT, description_html TEXT, tags TEXT, date_string TEXT, date DATE, PRIMARY KEY (source, name, id)); CREATE INDEX posts_by_author_and_date ON posts (author, date); CREATE INDEX posts_by_author_and_id_and_date ON posts (author, id, date);
And off I was, with access to the live db and able to run some
QUERY PLANs and so on!
Two key things here:
Access to another container’s files is possible through
According to Understanding Process Namespace Sharing:
Container filesystems are visible to other containers in the pod through the /proc/$pid/root link. This makes debugging easier, but it also means that filesystem secrets are protected only by filesystem permissions.
To access those files you need the same user as that other container, being root inside your debug container is not enough.
That’s it, have a nice day!