Accessing files using ephemeral containers

I have a service running in Kubernetes on my server that needed some tending debugging, so here’s how that went and the little trick that was needed for it.

Usually I’d just use kubectl exec and be on my way, but there were three issues with that:

1. I wanted to access the db and sqlite was not installed in the container
2. The service runs as a non-root user, so no installing anything in addition
3. The live container was pretty locked down, containing only busybox and my binary

Ephemeral containers to the rescue!

Here’s what was necessary in the end:

$ kubectl --context live debug -it numblr-c67cd998f-69ktm --image=alpine:3.15 --target=numblr --share-processes`

# try accessing the data of the live pod
/ # ps aux
PID   USER     TIME  COMMAND
    1 1000      0:25 /app/numblr -addr=0.0.0.0:5555 -debug-addr=0.0.0.0:6060 -db=/app/data/cache.db -stats
   67 root      0:00 sh
   76 root      0:00 ps aux
/ # ls /proc/1/root
ls: /proc/1/root: Permission denied

# replicate the live user
/ # apk add --no-cache shadow && useradd --home-dir / --shell /bin/sh numblr && apk del shadow
...

# run sqlite as that user for access!!
/ # apk add sqlite
...

/ # su - numblr -c 'sqlite3 /proc/1/root/app/data/cache.db'
SQLite version 3.36.0 2021-06-18 18:36:39
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE feed_infos ( name TEXT PRIMARY KEY, url TEXT, cached_at DATE , description text, error text);
CREATE TABLE posts ( source TEXT, name TEXT, id TEXT, author TEXT, avatar_url TEXT, url TEXT, title TEXT, description_html TEXT, tags TEXT, date_string TEXT, date DATE, PRIMARY KEY (source, name, id));
CREATE INDEX posts_by_author_and_date ON posts (author, date);
CREATE INDEX posts_by_author_and_id_and_date ON posts (author, id, date);

And off I was, with access to the live db and able to run some EXPLAIN QUERY PLANs and so on!

Two key things here:

• Access to another container’s files is possible through /app/$pid/root.

  According to Understanding Process Namespace Sharing:

  Container filesystems are visible to other containers in the pod through the /proc/$pid/root link. This makes debugging easier, but it also means that filesystem secrets are protected only by filesystem permissions.

• To access those files you need the same user as that other container, being root inside your debug container is not enough.

That’s it, have a nice day!