Etam Software - Shared libraries in Linux

In the beginning there was C

The basic building blocks of programs in C are functions. Functions allow us to call the same part of code with different parameters.

example.c:

#include <stdio.h>

int add_one(int i) {
    return i+1;
}

int main() {
    int k = add_one(5);
    int l = add_one(k);
    printf("Hello! %d\n", l);
    return 0;
}

We can look at the generated assembly and identify corresponding parts. Lines starting with "#" are my comments. Unimportant parts are replaced by "[...]".

$ gcc -S example.c -o -
[...]
# this is a function name
add_one:
[...]
# this is i+1
        addl    $1, %eax
[...]
# this is return
        ret
[...]
# this is a function name
main:
[...]
# this is add_one(5)
        movl    $5, %edi
        call    add_one
[...]
# this is add_one(k)
        movl    %eax, %edi
        call    add_one
[...]
# this is call to printf
        movl    %eax, %esi
        movl    $.LC0, %edi
        movl    $0, %eax
        call    printf
[...]
# this is return 0
        movl    $0, %eax
        leave
        .cfi_def_cfa 7, 8
        ret
[...]

We can compile and run the program:

$ gcc example.c -o example
$ ./example
Hello! 7

We can inspect the symbols in the binary executable and find our functions (see man nm):

$ nm example | grep -E ' (add_one|main|printf)'
0000000000401156 T add_one
0000000000401165 T main
                 U printf@GLIBC_2.2.5

This shows that add_one and main are functions inside the executable, but printf is not here.

We can check which shared libraries are needed by our executable:

$ readelf -d example | grep NEEDED
 0x0000000000000001 (NEEDED)    Shared library: [libc.so.6]

We can use ltrace to run the program and see what functions are called from shared libraries:

$ ltrace ./example 2>ltrace.txt
Hello! 7
$ cat ltrace.txt
printf("Hello! %d\n", 7) = 9
+++ exited (status 0) +++

We can see which shared library files are actually loaded to our executable:

$ ldd ./example
        linux-vdso.so.1 (0x00007fc95b1c0000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fc95ae00000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc95b1c2000)

Note: linux-vdso.so.1 is a special thing, that is magically created by Linux kernel (see man vdso). And we'll talk about /lib64/ld-linux-x86-64.so.2 later on.

We can verify, that printf exists in /lib64/libc.so.6:

$ nm -D /lib64/libc.so.6 | grep -E ' printf$'
00000000000605ce T printf

Interpreters and executables

Let's start by a small detour into bash scripts.

Bash scripts can be run by launching the interpreter with the file as an argument:

$ cat >example.sh <<EOF
echo "Hello!"
EOF
$ bash example.sh
Hello!

They can also begin with #! (so called "shebang"), followed by the interpreter command. When made executable, the file can be launched directly:

$ cat >example.sh <<EOF
#!/bin/bash
echo "Hello!"
EOF
$ chmod +x example.sh
$ ./example.sh
Hello!

In this case the kernel itself is responsible for detecting #! and loading the interpreter.

Now back to binary executables.

A binary executable also has an interpreter:

$ file ./example
./example: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, [...]

OK, what is this interpreter thing?

$ file /lib64/ld-linux-x86-64.so.2
/lib64/ld-linux-x86-64.so.2: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, [...]

A shared library? We've seen it earlier in the output of ldd, that it's loaded as a shared library.

But it has a trick up its sleave:

$ /lib64/ld-linux-x86-64.so.2 --help
Usage: /lib64/ld-linux-x86-64.so.2 [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
You have invoked 'ld.so', the program interpreter for dynamically-linked
ELF programs.  Usually, the program interpreter is invoked automatically
when a dynamically-linked executable is started.
[...]

It's also an executable program and it nicely tells us what it can do.

We can use it to run our executable (just like we can use bash to run a script):

$ /lib64/ld-linux-x86-64.so.2 ./example
Hello! 7

When running a binary directly, the kernel, just like it detects #! and runs the interpreter, it also detects the ELF file and loads the specified interpreter.

BTW it's also known as ld.so command:

$ ls -l `which ld.so`
lrwxrwxrwx 1 root root 29 08-31 12:56 /usr/bin/ld.so -> ../lib64/ld-linux-x86-64.so.2

The `ld.so`

Quoting from man ld.so:

The programs ld.so and ld-linux.so* find and load the shared objects (shared libraries) needed by a program, prepare the program to run, and then run it.

Finding Nemo

Finding of shared libraries works quite similarly to how bash searches $PATH for commands. The file /etc/ld.so.conf contains a list of directories, that needs to be checked when searching for a file.

There are interesting cases, where multiple shared libraries have the same name and provide the same functions, but are in different directories and provide different implementations.

One example is mechanism called "hwcaps" (from "hardware capabilities"). A file /usr/lib64/libbz2.so.1.0.6 contains a generic implementation of BZ2 (it's not important right now what BZ2 is), that works on every x86_64 CPU. But over time new extensions were added to the x86_64 architecture and they can be used to build an optimized version of BZ2. So on my system there is a file /usr/lib64/glibc-hwcaps/x86-64-v3/libbz2.so.1.0.6 that provides exactly the same functions and it's picked by ld.so if the CPU supports it.

Another example is when a new version of something wants to keep being compatible with older version. SDL (Simple DirectMedia Layer) is a popular library for building games. After version 1.2 was phased out in favor of version 2.0, an sdl12-compat library was created, that provides libSDL-1.2.so.0, but internally uses libSDL2-2.0.so.0. Likewise after release of 3.0, sdl2-compat provides libSDL2-2.0.so.0. Yes, it is possible to run a game that requires SDL 1.2 on SDL 3 through both compatibility layers.

`LD_LIBRARY_PATH`

If an application is shipped with some shared libraries in non-standard directory, the environment variable LD_LIBRARY_PATH allows to specify additional places for ld.so to search.

For example:

$ cd /usr/lib64/firefox
$ ls -1 lib*.so
libgkcodecs.so
liblgpllibs.so
libmozavcodec.so
libmozavutil.so
libmozgtk.so
libmozinference.so
libmozsandbox.so
libmozsqlite3.so
libmozwayland.so
libxul.so
$ ldd libxul.so | grep libmoz
        libmozsandbox.so => not found
        libmozsqlite3.so => not found
        libmozgtk.so => not found
        libmozwayland.so => not found
$ LD_LIBRARY_PATH=. ldd libxul.so | grep libmoz
        libmozsandbox.so => ./libmozsandbox.so (0x00007f64502e5000)
        libmozsqlite3.so => ./libmozsqlite3.so (0x00007f6445c94000)
        libmozgtk.so => ./libmozgtk.so (0x00007f64502d1000)
        libmozwayland.so => ./libmozwayland.so (0x00007f64502cb000)

You can read /usr/bin/firefox file (it's a shell script) and find how it sets up LD_LIBRARY_PATH.

`LD_PRELOAD`

This one trick allows to inject code into existing applications and libraries, where no one expects it.

In our example.c we had a call to printf function from libc.so.6. Now let's imagine, that for some reason, we want to change what this function does, without modifying the example nor libc.so.6 binary files.

We can create a new file fake_printf.c:

int printf() {
    puts("hahahahaha!!!!!");
}

And compile it into a shared library (it will print some warnings, but ignore them):

$ gcc -fpermissive -shared -fPIC fake_printf.c -o fake_printf.so

Now run it:

$ LD_PRELOAD=./fake_printf.so ./example
hahahahaha!!!!!

What happened is the fake_printf.so was loaded before all other shared libraries. Our printf implementation replaced the one from libc.so.6. The puts function is still provided by libc and it's used by our fake printf just fine.

In the beginning there was C

Interpreters and executables

The ld.so

Finding Nemo

LD_LIBRARY_PATH

LD_PRELOAD

The `ld.so`

`LD_LIBRARY_PATH`

`LD_PRELOAD`