Jim Calabro

Building Statically Linked Go Executables with CGO and Zig

2025-03-28T00:00:00Z

Building Statically Linked Go Executables with CGO and Zig - Jim Calabro

Jim Calabro

About

RSS | Atom | JSON

---

Building Statically Linked Go Executables with CGO and Zig

This is a short post about how to create a statically linked Go executable that calls in to CGO dependencies using Zig. The full code for this post is available in this repo.

By default, if you're using CGO, the executable you generate dynamically links, but I frequently want to statically link to avoid runtime errors.

First, let's create a zig library, with zig init, then trim back the excess stuff it generates so we're left just with a simple static library. You can rm src/main.zig since we're not creating a zig executable.

Next, we can trim build.zig to just:

// build.zig
const std = @import("std");

pub fn build(b: *std.Build) void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    const lib_mod = b.createModule(.{
        .root_source_file = b.path("src/root.zig"),
        .target = target,
        .optimize = optimize,
    });

    const lib = b.addLibrary(.{
        .linkage = .static,
        .name = "cgo_static_linking",
        .root_module = lib_mod,
    });
    b.installArtifact(lib);
}

We can leave the build.zig.zon file alone.

Now, let's actually write a simple library function that uses the C ABI in src/root.zig:

// src/root.zig
const std = @import("std");

pub export fn my_zig_function() void {
    std.debug.print("Hello from zig!\n", .{});
}

And its corresponding C header file named zig_lib.h:

// zig_lib.h
#pragma once

void my_zig_function();

That's it on the zig side! You can build the library now by simply running zig build.

Let's write the Go program that calls it.

// main.go
package main

/*
#cgo LDFLAGS: -L./zig-out/lib -lcgo_static_linking -static
#include "zig_lib.h"
*/
import "C"
import "fmt"

func main() {
	fmt.Println("starting program")
	defer fmt.Println("done")

	C.my_zig_function()
}

We'll now build the Go executable and statically link it with this bash command:

CC="zig cc -target x86_64-linux-musl" \
CGO_ENABLED=1 \
CGO_LDFLAGS="-static" \
GOOS=linux GOARCH=amd64 \
go build -a -ldflags '-extldflags "-static"' main.go

Let's check our work:

$ ./main
starting program
Hello from zig!
done

$ ldd ./main
        not a dynamic executable

Looks good to me!

I'm incredibly grateful that I'm building at a time where our tools are getting extremely good. Go and Zig are both amazing!

If you find this useful as I do, perhaps consider making a donation to the Zig Software Foundation!

uscope Update #2

2025-03-20T00:00:00Z

uscope Update #2 - Jim Calabro

Jim Calabro

About

RSS | Atom | JSON

---

uscope Update #2

Mar 20, 2025

Hello again! It's been a bit since I've posted an update, and I've been pretty quiet in open source work during that time, despite a lot happening.

If you find this work compelling, please consider supporting my work. This is a strong signal to me that I'm on the right track.

Updates

Last month, I didn't make a ton of progress on uscope to be perfectly honest. Life has been busy. Here are the updates:

Upgraded to Zig 0.14.0. Congrats to the zig team on a great release!
Worked on adding Go support in the debugger (much more work is needed still, but Go DWARF parses okay now).
Added preliminary support for the C3 programming language.
Made a tiny bit of progress on debugging binaries built by the zig self-hosted backend. There's a good amount more to do here.
You won't see this in any uscope code yet, but I did a lot of experimenting with eBPF and can see a path forward on using it for many interesting debugging use-cases.

What's Next?

Big news: I accepted a new job! I haven't announced where yet, but I will soon. I start in a couple weeks and I'm very excited! It's at a small, fast-growing company and I think we're a great fit culturally and technically.

Consequently, I predict that I'll have substantially less spare time and energy to work on uscope, at least for a while. So, in order to make sure we all get the most out of it, I'm going to be re-focusing the project in the following ways to try to make the code I've written as useful as possible to the community ASAP:

I'm going to yanking the ELF/DWARF parsing code out in to its own standalone Zig library.
- I think a lot of people would be tempted to build more cool introspection tools if only there were an easy-to-use library for parsing debug info that was free of license obligations
- Gimli for rust looks great and pyelftools is great, but for folks writing C or Zig programs, there aren't any great options that are obligation-free as far as I can tell (I certainly could be missing some lesser-known libraries besides the major ones).
I'm going to focus mostly on remote debugging since this is how I will personally use the debugger.
- I'll be creating a headless "remote daemon" mode for the debugger that is accessible over the network.
- This is particularly important for a Linux debugger as most of the code I care about isn't actually on my machine; it's somewhere in a data center/cloud.
Since I'm focusing on remote debugging, I'm going to rip out the native desktop UI and instead create a web UI that will access the remote daemon from the browser (even if it's just running on localhost).
- I'm sure this decision will be controversial, but there are many reasons to not ship a native desktop UI for Linux.
  - It's the wild west, and I cannot reasonably support all flavors of Linux desktop as a hobby project.
  - It's already been an issue for myself and my very limited user base, and it will only get worse over time.
- At time of writing, the browser is the highest bang-for-your buck in terms of UI quality to time invested.
  - There are some really impressive native UIs out there (1, 2), but they take a ton of time and effort, and frankly I currently don't have the graphics chops to build a beautiful, professional native UI. I could learn these techniques over time, but that takes spare time and energy that I don't have.
  - I used to do a ton of web development, and I'm pretty confident I can build a reasonably good UI quickly.
  - It will be written in vanilla js and still be fast and light-weight!
- I have a strong preference for only distributing statically linked binaries, and shipping an executable with a native GUI that's GPU-rendered is a really tall order.
  - Perhaps it's impossible? Not sure. In any case, it would be going against the grain.
  - CPU rendering is of course possible, but that comes with its own downsides (poor performance and battery draining mostly).
- The user-supplied visualization plugin story becomes so much simpler when you're working in the browser.
  - It's important that writing plugins is approachable by the programming population at large, otherwise very few people will write them at all.
- The uscope code is set up in such a way that adding a new UI is actually quite easy, and we can ship multiple UI types in the same binary.
  - For example, we can simultaneously support "remote daemon" mode as well as something like "cli" mode which would be akin to a more traditional gdb/lldb cli.
  - I'm mentioning this because it may be a good time to add a cli debugger, though no promises. I still think that the visual debugging story is dramatically more important than cli debugging.
  - The community can still maintain a native UI if they desire; it just won't ship as part of stock uscope.

Final Words

Those are the updates, and they're pretty major ones! Come discuss on Discord if you have feedback.

I've been doing these short updates about once a month so far. Definitely expect that to slow down - there's no regular cadence to my writing on this site.

That's all for now! Thanks for reading!

uscope Update #1

2025-02-18T00:00:00Z

uscope Update #1 - Jim Calabro

Jim Calabro

About

RSS | Atom | JSON

---

uscope Update #1

Feb 18, 2025

Hi! I wanted to give a quick update of the things I've been working on in uscope, the from-scratch graphical Linux debugger I've been building.

If you find this work compelling, please consider supporting my work. This is a strong signal to me that I'm on the right track.

For starters, I demo'ed the project on Zig Showtime! Thank you to Loris for having me; it was a lot of fun.

Next, here are the project updates in no particular order:

I completely re-wrote the variable rendering code so now it's displayed as an expandable tree. This means that things like arrays/slices, nested structs, and structs with circular pointers back to themselves are rendered properly. This is similar to how other debuggers render their data.
I also fixed a boat-load of bugs related to extracting variable data from the subordinate process, and folks are reporting some amount of success using the debugger on their real-world programs now!
I added the ability to debug multi-threaded subordinates (there's still a lot of bugs and things to do here, but it's a start)
Added support for Odin data types as a first-class citizen.
The debugger now renders very, very basic information for C++, Rust, and Jai data types. It's ugly, but can be usable if you're not doing anything complicated (i.e. Jai looks OK, but C++ and Rust not so much)

That's all for now. Thanks for reading!

P.S., Here's a screenshot of uscope rendering some very basic Odin types:

uscope: A New Debugger and Introspection Toolchain

2025-01-26T00:00:00Z

uscope: A New Debugger and Introspection Toolchain - Jim Calabro

Jim Calabro

About

RSS | Atom | JSON

---

uscope: A New Debugger and Introspection Toolchain

Jan 26, 2025

uscope (pronounced "microscope") is a new graphical debugger for Linux. Its goal is to enable you to understand what your system is actually doing as quickly and effectively as possible.

All debugging-related code is written from scratch, i.e. parsing ELF/DWARF, handling control of the subordinate process (launch, kill, step, continue, etc.), reading data from the subordinate's memory/registers and displaying it in a user-friendly manner, stack unwinding, etc.

Here's a screenshot of the debugger stopped in a simple zig program that just prints out a bunch of basic variables. You can see that a fair number of table-stakes debugger features have already been implemented, but there's still a ton more to do!

Why Are You Doing This?

I feel strongly that the state of the art for introspection tooling needs to improve, particularly on Linux. Figuring out what your program is doing (often called observability) is a critical aspect of developing all modern software, whether it's a command line application, giant distributed system, video game, database, and so on. There are loads of graphical debuggers that are powerful and fast to use on Windows, but it's a missing piece of the puzzle on Linux.

Fundamentally, it is the process of collecting data, possibly transforming or aggregating it, then visualizing it to bring light to something you didn't previously understand. This feedback loop needs to be easy and fast for developers, and unfortunately, the existing tools are not good enough.

The available Linux debuggers are gdb and lldb. When I use them, they crash a lot, they don't understand the data types I care about particularly well, and they don't make the data I need available at my fingertips as quickly as possible (which is really a problem with all CLI-based debuggers).

Consequently, most Linux devs I know don't use a debugger and instead just printf-debug, which is often the correct choice because what's the point of a debugger that doesn't save you time? We've got to improve the situation because without a great debugger, our software quality suffers and it takes longer to deliver.

What's the Plan?

It's very early days now and it's going to be a while before it's usable as a daily driver. The table-stakes debugger features are off to a good start, but there's still a long way to go.

Currently, only C and Zig have any shot of being debugged nicely, but support for more languages is coming. Support for other platforms (at least macOS and Windows) is also planned, but very far off. There's more detail on specific features I want to implement in the README.

The debugger collects data from various sources (currently mostly ptrace), transforms those raw bytes to a common data format, then provides that to a UI for visualization. Both the backend (data collection) and frontend (data visualization) will be hackable and extensible.

For instance, I work on chess engines for a living, and it would be pretty slick if I had a debugger that understood my exact data layout and automatically rendered an interactive chessboard when I want to inspect a variable that contains a bitboard for a position. The possibilities are limitless for bespoke debug tooling for your specific problem domain once we have a simple, extensible toolkit to work with.

How do You (yes you!) Get Involved?

Reach out! We've got a Discord, and my personal email is jim at this domain. I love talking debug tooling.

Contributions are absolutely welcome, though I'd recommend reaching out before tackling a large chunk of work to make sure we're on the same page and effort isn't wasted.

You could also consider sponsoring my work. This is a very strong signal to me that I'm on the right track.

How DWARF Works: Debug Information Entries

2024-09-28T00:00:00Z

How DWARF Works: Debug Information Entries - Jim Calabro

Jim Calabro

About

RSS | Atom | JSON

---

How DWARF Works: Debug Information Entries

Sep 28, 2024

This is part of the series on DWARF.

Debug Information Entries

So far, we've parsed enough of the contents of an ELF executable to get the various debug info sections as raw byte arrays. Now, we'll use all these sections together to make progress towards properly inspecting a running instance of the target program.

This post's goal is to construct the Debug Information Entry (DIE) tree. This is a long task with several side quests along the way.

A DIE is a piece of information about a particular node of a compiled program, and they are stored in a tree. When compiler authors write parsers/lexers, the first step in the pipeline is generally attempting to turn the source text of a program in to an Abstract Syntax Tree, and you can sort of think of the DIE tree as a similar thing for DWARF debuggers. All your compile units, functions, variables, struct definitions, etc. each get an entry in the DIE tree. A DIE tree is really more like a forest in that each compile unit in your executable gets its own tree.

We will use these DIEs later on to ascertain key facts about a running process such as "what is the data type of the variable named x, and where in memory/registers are its bytes?", which is critical for building a debugger.

Perhaps it's easiest to show what it looks like. Here's the relevant section of cloop's DIE tree using dwarfdump cloop:

< 1><0x000002dd>    DW_TAG_subprogram
                      DW_AT_external              yes(1)
                      DW_AT_name                  main
                      DW_AT_decl_file             0x00000001 /home/jcalabro/local/data/cloop/main.c
                      DW_AT_decl_line             0x00000004
                      DW_AT_decl_column           0x00000005
                      DW_AT_type                  <0x00000058>
                      DW_AT_low_pc                0x00401156
                      DW_AT_high_pc               <offset-from-lowpc> 86 <highpc: 0x004011ac>
                      DW_AT_frame_base            len 0x0001: 0x9c:
                          DW_OP_call_frame_cfa
                      DW_AT_call_all_tail_calls   yes(1)
                      DW_AT_sibling               <0x0000031c>
< 2><0x000002ff>      DW_TAG_variable
                        DW_AT_name                  pid
                        DW_AT_decl_file             0x00000001
                        DW_AT_decl_line             0x00000005
                        DW_AT_decl_column           0x0000000b
                        DW_AT_type                  <0x0000009d>
                        DW_AT_location              len 0x0002: 0x9164:
                            DW_OP_fbreg -28
< 2><0x0000030d>      DW_TAG_variable
                        DW_AT_name                  ndx
                        DW_AT_decl_file             0x00000001
                        DW_AT_decl_line             0x00000006
                        DW_AT_decl_column           0x00000018
                        DW_AT_type                  <0x0000031c>
                        DW_AT_location              len 0x0002: 0x9168:
                            DW_OP_fbreg -24

There's not a ton to see in our cloop program because it's so simple, but there is a function (aka subprogram) named main and two variables, ndx and pid, just as we'd expect. It tells us on what lines they appear, and what their data types are, and DW_AT_location actually tells us how to find the value of a variable at runtime! There's a lot of good stuff in here.

You can see that all these entries have a single tag that indicates the type of entry (each prefixed with DW_TAG_), then some number of attributes, or metadata describing that particular entry (each prefixed with DW_AT_). If you look at the full output, you can see a lot of DIEs for primitive C types and the standard library.

It's a bit tough to tell because we're only one level deep, but this data is stored in a tree. You can see it in the indentation of the tags and attributes if you look carefully, or you can use the numbers on the far left as reference of depth (< 1> and < 2>). This is just how dwarfdump renders the DIE tree; the actual format on disk is not of this form.

Abbrev Tables

All that being said, the very first thing we have to parse actually is not the DIE tree. No, we will start by reading the full set of abbreviation tables. The abbrev tables give you a lot of information that's used in constructing the DIE tree. It tells you, for each potential entry in the tree, what is its tag (AST node type), and what attributes each node has. For each attribute, it tells you its data type via its form (is it a string, is it a constant integer value, and so on). The set of abbrev tables lives in the .debug_abbrev ELF section.

Parsing the abbrev table is pretty straightforward: it's an array of declarations (decls), each with a code, a tag, and some number of attributes. To see it in action, you run dwarfdump --print-abbrev cloop:

<    1><0x00000000> DW_TAG_member               DW_children_no
       <0x00000003>              DW_AT_name                  DW_FORM_strp
       <0x00000005>              DW_AT_decl_file             DW_FORM_implicit_const <4 (0x4)>
       <0x00000008>              DW_AT_decl_line             DW_FORM_data1
       <0x0000000a>              DW_AT_decl_column           DW_FORM_data1
       <0x0000000c>              DW_AT_type                  DW_FORM_ref4
       <0x0000000e>              DW_AT_data_member_location  DW_FORM_data1
<    2><0x00000012> DW_TAG_base_type            DW_children_no
       <0x00000015>              DW_AT_byte_size             DW_FORM_data1
       <0x00000017>              DW_AT_encoding              DW_FORM_data1
       <0x00000019>              DW_AT_name                  DW_FORM_strp
<    3><0x0000001d> DW_TAG_pointer_type         DW_children_no
       <0x00000020>              DW_AT_byte_size             DW_FORM_implicit_const <8 (0x8)>
       <0x00000023>              DW_AT_type                  DW_FORM_ref4
<    4><0x00000027> DW_TAG_typedef              DW_children_no
       <0x0000002a>              DW_AT_name                  DW_FORM_strp
       <0x0000002c>              DW_AT_decl_file             DW_FORM_data1
       <0x0000002e>              DW_AT_decl_line             DW_FORM_data1
       <0x00000030>              DW_AT_decl_column           DW_FORM_data1
       <0x00000032>              DW_AT_type                  DW_FORM_ref4

       etc...

We can write a parser for this with the following methodology:

Create a new abbrev table, storing our current offset (the number of bytes we've read so far through the section)
If we've read to the end of the section, we're done
If we're not done, read N decls for this table via:

Create a new decl and read its code
If the code is zero, we're done with this table (all tables end with a "null entry" whose code is zero)
Read the decl's tag enum value
Read a bool that indicates whether or not this node in the tree has children
Read all attributes for this decl

Read the attribute name enum value
Read the attribute form enum value
If the name and the form are both zero, we've read all of this decl's attributes
If the form is DW_FORM_implicit_const, read and store the implicit const value (we'll see how this is used later in this post)

Check the DWARF documentation for what each of those enum values should be, and see the appendix for more information on LEB128. Translated in to code, that looks like:

type abbrevTable struct {
    offset int
    decls  map[uint64]abbrevDecl
}

type abbrevDecl struct {
    code        uint64
    hasChildren uint8
    attrs       []abbrevAttribute
}

type abbrevAttribute struct {
    name             uint64
    form             uint64
    implicitConstVal int64
}

func parseAbbrevTables(abbrevSecion []byte) []abbrevTable {
    buf := bytes.NewBuffer(abbrevSecion)
    tables := []abbrevTable{}
    reader := NewBinaryReader(buf, binary.NativeEndian)

    for {
        table := abbrevTable{
            offset: BufferOffset(len(abbrevSecion), buf),
            decls:  map[uint64]abbrevDecl{},
        }
        if table.offset >= len(abbrevSecion) {
            break // nothing left to read in the section
        }

        for {
            decl := abbrevDecl{}
            decl.code, _ = leb128.DecodeU64(reader)
            if decl.code == 0 {
                break // we've finished with this table
            }

            decl.tag, _ = leb128.DecodeU64(reader)
            decl.hasChildren, _ = Read[uint8](reader)

            for {
                attr := abbrevAttribute{}
                attr.name, _ = leb128.DecodeU64(reader)
                attr.form, _ = leb128.DecodeU64(reader)
                if attr.name == 0 && attr.form == 0 {
                    break
                }

                if attr.form == 0x21 { // DW_FORM_implicit_const
                    attr.implicitConstVal, _ = leb128.DecodeS64(reader)
                }

                decl.attrs = append(decl.attrs, attr)
            }

            table.decls[decl.code] = decl
        }

        tables = append(tables, table)
    }

    return tables
}

That's it with abbrev tables. We'll use these quite a lot as we parse the DIE tree. You can see that we already parsed something that resembles a tree, but we're not at the point where the data is meaningful yet. Note that cloop only has one table, but there will probably be many in real-world programs.

Compilation Unit Headers

Now we're ready to start tackling the .debug_info section, which by is the section that contains most of the DIE information. We're going to parse one or more compilation units and their DIEs in a loop, starting with the compilation unit header. The header isn't itself a DIE, but it does give us some critical information in dealing with the remainder of the tree. So our outer loop looks like:

type CU struct {
    header *CUHeader
    dies   []DIE
}

type CUHeader struct {
    length            uintptr
    version           uint16
    unitType          uint8 // added in v5
    debugAbbrevOffset int
    addrSize          uint8

    // not a real field in the DWARF standard,
    // but helpful for bookkeeping
    is32Bit bool
}

cus := []*CU{}
infoReader := NewBinaryReader(bytes.NewBuffer(sections.info))
for {
    if infoReader.Offset() >= len(sections.info) {
        break
    }

    cuHeader := parseCUHeader(infoReader)

    // choose the correct abbrev table for this CU
    abbrev := abbrevTables[0]
    for _, a := range abbrevTables {
        if a.offset == cuHeader.debugAbbrevOffset {
            abbrev = a
            break
        }
    }

    cu := parseCU(infoReader, cuHeader, sections, abbrev)
    cus = append(cus, cu)
}

Then, we can implement parseCUHeader. Each compilation unit header contains these fields in order, but note that the order of debug_abbrev_offset and addr_size is reversed as of DWARF v5.

Additionally, see the appendix for more information on fields of type initial length.

length, initial length: the number of bytes taken by this CU header in the .debug_info section (this does not include the number of bytes it takes to store the length itself, so either 4 or 12 bytes)
version, uint16: DWARF version number
unit_type, uint8: type of compilation unit (this field was added in DWARF v5 and there is no such field in v4 or below, see DWARF v5, section 3.1 for more details, and for our purposes we only care about DW_UT_compile)
debug_abbrev_offset, uint32 if is_32_bit, else uint64: how far to seek in to the abbrev section in order to find this CU's abbrev table
addr_size, uint8: how large an address is in this CU

So we can parse a CU header with something like:

func parseCUHeader(reader *BinaryReader) *CUHeader {
    header := &CUHeader{}

    header.length = readInitialLength(reader)

    // we know it's a 32 bit binary if our initial length
    // was 4 bytes, not 12
    header.is32Bit = reader.Offset() == 4

    header.version, _ = Read[uint16](reader)

    if header.version >= 5 {
        // this field was added in DWARF v5
        header.unitType, _ = Read[uint8](reader)

        // DWARF v5 changes the order of fields (switches abbrev_offset and addr_size)
        header.addrSize, _ = Read[uint8](reader)
        header.debugAbbrevOffset = parseAbbrevOffset(reader, header)
    } else {
        header.debugAbbrevOffset = parseAbbrevOffset(reader, header)
        header.addrSize, _ = Read[uint8](reader)
    }

    return header
}

func parseAbbrevOffset(reader *BinaryReader, header *CUHeader) int {
    if header.is32Bit {
        offset, _ := Read[uint32](reader)
        return int(offset)
    }

    offset, _ := Read[uint64](reader)
    return int(offset)
}

You can check your work against readelf --debug-dump=info cloop:

Contents of the .debug_info section:

  Compilation Unit @ offset 0:
   Length:        0x320 (32-bit)
   Version:       5
   Unit Type:     DW_UT_compile (1)
   Abbrev Offset: 0
   Pointer Size:  8

etc...

Compilation Units

Now, we're ready to begin parsing the DIE tree in this compilation unit! What we really want at the end of this section is actually just an array of DIEs, but we do need to interpret them as a tree in order to parse the .debug_info section correctly.

First, we should pick an abbrev table by searching the list of abbrev table offsets for the one that matches with the debug_abbrev_offset field from the CU header. This is the table we'll use for the rest of parsing this CU. This code is already handled in the "outer loop" example above.

Next, we'll begin reading through the .debug_info section to determine the list of DIEs, their depth in the tree, and the form of each of their attributes. DWARF forms are basically just data types, so this is just saying that we want to figure out how to interpret the bytes of each attribute (should we read a uint8, a string, etc.). Forms are an enum with the prefix DW_FORM_.

Additionally, based on which form we're looking at, we should also be sure to store its associated attribute class for more information on how to interpret this data. We won't be using this much since our code is just a demo, but real-world, robust parsers will use this a lot. Read the spec to learn more.

To complete this, we should:

Read a ULEB128, that's the code of an item in the abbrev table that we will look up
If the code is zero, we are either done reading DIEs for the entire compilation unit, or we're done reading child DIEs for a node in the tree
- If you are at the root level of the DIE tree, you're done parsing this compile unit
- Else, pop up one level in the tree and continue
Use the code to look up the abbrev decl we care about in the abbrev table
For each attribute in the decl we're going to want to choose which type of form to use to interpret the bytes of this attribute, and read that many bytes from the .debug_info section

Note that list of form types is very long, so I'll only implement enough to parse cloop on my machine in the code sample below. Refer to the documentation for the version(s) of DWARF you care about to see the full list.

If DW_CHILDREN_yes was set on the decl, push an item on to the DIE tree

Easy to say, but somewhat tedious to actually do. First, let's write that algorithm:

func parseCU(
    reader *BinaryReader,
    header *CUHeader,
    sections *DWARFSections,
    abbrev abbrevTable,
) *CU {
    dies := []DIE{}       // list of all DIEs
    dieTree := []uint64{} // stack of abbrev codes

    for {
        dieOffset := reader.Offset()

        abbrevCode, _ := leb128.DecodeU64(reader)
        if abbrevCode == 0 {
            if len(dieTree) == 1 {
                break // we're done
            }

            // this DIE has no more children, pop the stack
            dieTree = dieTree[:len(dieTree)-1]
            continue
        }

        abbrevDecl := abbrev.decls[abbrevCode]
        die := DIE{
            offset: dieOffset,
            depth:  len(dieTree),
            tag:    abbrevDecl.tag,
        }

        for _, attr := range abbrevDecl.attrs {
            form := chooseFormAndAdvanceBySize(reader, header, sections, attr)
            die.forms = append(die.forms, form)
        }

        dies = append(dies, die)
        if abbrevDecl.hasChildren == 1 {
            dieTree = append(dieTree, abbrevCode)
        }
    }

Next, let's define a few types and "enums" (Go doesn't really have enums):

type DIE struct {
    offset int
    depth  int
    tag    uint64
    forms  []Form
}

type Form struct {
    data  any
    class Class
}

type DWARFForm int

const (
    // we're not going to use all of these today, but a real parser should
    DW_FORM_addr           DWARFForm = 0x01
    DW_FORM_block2         DWARFForm = 0x03
    DW_FORM_block4         DWARFForm = 0x04
    DW_FORM_data2          DWARFForm = 0x05
    DW_FORM_data4          DWARFForm = 0x06
    DW_FORM_data8          DWARFForm = 0x07
    DW_FORM_string         DWARFForm = 0x08
    DW_FORM_block          DWARFForm = 0x09
    DW_FORM_block1         DWARFForm = 0x0a
    DW_FORM_data1          DWARFForm = 0x0b
    DW_FORM_flag           DWARFForm = 0x0c
    DW_FORM_sdata          DWARFForm = 0x0d
    DW_FORM_strp           DWARFForm = 0x0e
    DW_FORM_udata          DWARFForm = 0x0f
    DW_FORM_ref_addr       DWARFForm = 0x10
    DW_FORM_ref1           DWARFForm = 0x11
    DW_FORM_ref2           DWARFForm = 0x12
    DW_FORM_ref4           DWARFForm = 0x13
    DW_FORM_ref8           DWARFForm = 0x14
    DW_FORM_ref_udata      DWARFForm = 0x15
    DW_FORM_indirect       DWARFForm = 0x16
    DW_FORM_sec_offset     DWARFForm = 0x17
    DW_FORM_exprloc        DWARFForm = 0x18
    DW_FORM_flag_present   DWARFForm = 0x19
    DW_FORM_strx           DWARFForm = 0x1a
    DW_FORM_addrx          DWARFForm = 0x1b
    DW_FORM_ref_sup4       DWARFForm = 0x1c
    DW_FORM_strp_sup       DWARFForm = 0x1d
    DW_FORM_data16         DWARFForm = 0x1e
    DW_FORM_line_strp      DWARFForm = 0x1f
    DW_FORM_ref_sig8       DWARFForm = 0x20
    DW_FORM_implicit_const DWARFForm = 0x21
    DW_FORM_loclistx       DWARFForm = 0x22
    DW_FORM_rnglistx       DWARFForm = 0x23
    DW_FORM_ref_sup8       DWARFForm = 0x24
    DW_FORM_strx1          DWARFForm = 0x25
    DW_FORM_strx2          DWARFForm = 0x26
    DW_FORM_strx3          DWARFForm = 0x27
    DW_FORM_strx4          DWARFForm = 0x28
    DW_FORM_addrx1         DWARFForm = 0x29
    DW_FORM_addrx2         DWARFForm = 0x2a
    DW_FORM_addrx3         DWARFForm = 0x2b
    DW_FORM_addrx4         DWARFForm = 0x2c

    // extensions, etc...
)

type Class int

const (
    address Class = iota
    addrptr
    block
    constant
    exprloc
    flag
    lineptr
    loclist
    loclistptr
    macptr
    rnglist
    rnglistptr
    reference
    str
    stroffsetpt
)

Then, we can write our chooseFormAndAdvanceBySize function and a couple helpers:

func chooseFormAndAdvanceBySize(
    reader *BinaryReader,
    header *CUHeader,
    sections *DWARFSections,
    attr abbrevAttribute,
) Form {
    form := Form{}

    dwarfForm := DWARFForm(attr.form)
    switch dwarfForm {

    // read N bytes of data as a constant value
    case DW_FORM_data1:
        form.data, _ = Read[uint8](reader)
        form.class = constant
    case DW_FORM_data2:
        form.data, _ = Read[uint16](reader)
        form.class = constant
    case DW_FORM_data4:
        form.data, _ = Read[uint32](reader)
        form.class = constant
    case DW_FORM_data8:
        form.data, _ = Read[uint64](reader)
        form.class = constant
    case DW_FORM_sdata:
        form.data, _ = leb128.DecodeS64(reader)
        form.class = constant
    case DW_FORM_udata:
        form.data, _ = leb128.DecodeU64(reader)
        form.class = constant

    // read an address
    case DW_FORM_addr:
        form.data, _ = Read[uintptr](reader)
        form.class = address

    // read a reference of N bytes
    case DW_FORM_ref_addr:
        form.data = readOffset(header, reader)
        form.class = reference
    case DW_FORM_ref1:
        form.data, _ = Read[uint8](reader)
        form.class = reference
    case DW_FORM_ref2:
        form.data, _ = Read[uint16](reader)
        form.class = reference
    case DW_FORM_ref4:
        form.data, _ = Read[uint32](reader)
        form.class = reference
    case DW_FORM_ref8:
        form.data, _ = Read[uint64](reader)
        form.class = reference

    // flags
    case DW_FORM_flag:
        form.data, _ = Read[uint8](reader)
        form.class = flag
    case DW_FORM_flag_present:
        form.data = []byte{1} // just indicates true
        form.class = flag

    // strings
    case DW_FORM_string:
        // read a string from the .debug_info section
        form.data = readNullTerminatedString(reader)
        form.class = str
    case DW_FORM_strp:
        // read a string from the .debug_str section
        offset := readOffset(header, reader)
        strSection := sections.str[offset:]
        strReader := NewBinaryReader(bytes.NewBuffer(strSection))
        form.data = readNullTerminatedString(strReader)
        form.class = str
    case DW_FORM_line_strp: // first introduced in DWARF v5
        // read a string from the .debug_line_str section
        offset := readOffset(header, reader)
        strSection := sections.line_str[offset:]
        strReader := NewBinaryReader(bytes.NewBuffer(strSection))
        form.data = readNullTerminatedString(strReader)
        form.class = str

    // read a DWARF expression as an N byte buffer
    // (much more on these in a later post!)
    case DW_FORM_exprloc:
        length, _ := leb128.DecodeU64(reader)
        buf := make([]byte, length)
        reader.Read(buf)
        form.data = buf
        form.class = exprloc

    // offset in to one of many sections based on the attribute
    case DW_FORM_sec_offset:
        form.data = readOffset(header, reader)

        // there are many more of these, and this is a real pain
        // when writing a real parser, so be sure to RTFM
        switch attr.form {
        case 0x10: // DW_AT_stmt_list
            form.class = lineptr
        }

    // this is a hard-coded constant value that comes from
    // the attribute itself in the .debug_abbrev section
    case DW_FORM_implicit_const:
        form.data = attr.implicitConstVal
        form.class = constant
    }

    return form
}

func readOffset(header *CUHeader, reader *BinaryReader) uint64 {
    if header.is32Bit {
        val, _ := Read[uint32](reader)
        return uint64(val)
    }

    val, _ := Read[uint64](reader)
    return val
}

func readNullTerminatedString(reader *BinaryReader) string {
    buf := []byte{}
    ch, _ := Read[uint8](reader)
    for ch != 0 {
        buf = append(buf, ch)
        ch, _ = Read[uint8](reader)
    }
    return string(buf)
}

Phew! That was a lot. However, now we have a full DIE tree! Be sure to check your work against dwarfdump cloop because a lot of these are subtle and it's easy to make mistakes, plus a single mistake tends to lead to cascading failures (i.e. if you read one byte instead of two, all subsequent reads are now off by one).

Summary

Today, we parsed the .debug_abbrev and .debug_info sections to ultimately construct a tree of debug information entries.

As was mentioned throughout the article, there's a lot more nuance writing a .debug_info parser that's able to handle any binary you'd encounter in the wild. Refer to the DWARF documentation for more info, but this hopefully was a helpful jumping off point for understanding the structure of the section.

Stay tuned for the next part where we'll learn how to read line number information so we can map addresses in the program text back to their source location!

Thank you for reading the series on DWARF. Please don't hesitate to reach out with comments, questions, or errata to jim at this domain dot com.

How DWARF Works: Parsing Just Enough ELF

2024-09-26T00:00:00Z

How DWARF Works: Parsing Just Enough ELF - Jim Calabro

Jim Calabro

About

RSS | Atom | JSON

---

How DWARF Works: Parsing Just Enough ELF

Sep 26, 2024

This is part of the series on DWARF.

What are ELF and DWARF?

Executable and Linkable Format (ELF) is a file format for executables, object files, shared libraries, and more that's used on various Unix-like systems. If you've ever downloaded and run a program on Linux, you're using an ELF executable. It's akin to an .exe file on Windows.

DWARF is a debugging information format that is used with ELF files. Debug information allows you to do neat things with a running program such as:

Map the compiled machine code stored inside back to the original source code
Figure out where variables are stored throughout the lifetime of a program as it executes
Unwind the callstack to generate a backtrace to find out where your program is stopped and where it came from
Much more!

In this series, we'll go in to a lot of detail on topics such as these. Let's take it from the top: parsing ELF files, which contain DWARF debug information.

Our Test Program

Let's parse some files! In order to do so, we'll need a program to play around with. Throughout the rest of this series, I'm going to use this dead-simple C program called cloop that gets its own process ID, then loops forever and prints it once per second. C is a good choice because it is simple, has no runtime, and it's well-supported by all the tools we'll be using. Here's the full program:

#include <unistd.h>
#include <stdio.h>

int main() {
    pid_t pid = getpid();
    unsigned long long ndx = 0;
    while (1) {
        printf("c looping (pid %d): %llu\n", pid, ndx);
        fflush(stdout);
        ndx++;
        sleep(1);
    }

    return 0;
}

I'll be compiling this with gcc 14.2.1 on Manjaro Linux with kernel version 6.9.12 using this build.sh script, but feel free to play around with CC and DWARF as we go (it defaults to DWARF version 5):

#!/usr/bin/env bash

${CC:-gcc} -Wall -Wextra -Werror -no-pie -O0 -g -gdwarf-${DWARF:-5} -o cloop main.c

Additionally, for this series, I'll give some short code examples of Go code to help illustrate various concepts. I chose Go because it's popular, terse, simple to read, and has a large standard library to help us out. I'll intentionally omit error handling and not worry about writing efficient code to keep the examples short. I'm using Go version 1.22.7.

Parsing The ELF File Header

There's a lot of data contained within ELF files, but for our needs it's pretty straightforward, and we can ignore most of it. We just want to grab the raw binary of each debug info section as well as a couple facts about the executable.

Each binary file starts with the ELF header, followed by various "sections", each of which is just a region of the file that has a distinct job. For instance, the program text (machine code) of your executable or object is in the .text section.

We first want to open and read the contents of the binary file. It starts with 16 bytes of "ELF Identifier" header data. The first four of those bytes are the magic number 0x7f followed by 0x45, 0x4c, 0x46, or ELF in ASCII. So using our BinaryReader, we'd do:

fileBuf, _ := os.ReadFile(filePath)
reader := NewBinaryReader(bytes.NewBuffer(fileBuf), binary.NativeEndian)

magic := []byte{0x7f, 'E', 'L', 'F'}

magicBuf := make([]byte, len(magic))
reader.Read(magicBuf)

if !slices.Equal(magic, magicBuf) {
    panic("incorrect ELF magic number")
}

Next comes a the e_ident header section, which contains several one-byte flags, each prefixed with EI_, then some padding, which you should skip over. They are, in order:

EI_CLASS: address size (1 for a 32-bit binary, 2 for a 64-bit binary)
EI_DATA: byte order (1 for 2's compliment little-endian, 2 for 2's compliment big-endian)
EI_VERSION: file format version (should always be 1 as of time of writing)
EI_OSABI: operating system and ABI (see Go's implementation for a list of values)
EI_ABIVERSION: often ignored on Linux
padding: 7 bytes (gives us 16 total bytes in the e_indent section)

Next up is the rest of the ELF file headers, again in order. Refer to the documentation or a robust implementation such as Go or Zig for more information on each field and their values.

e_type, uint16: file type
e_machine, uint16: machine type
e_version, uint32: file format version
e_entry, uintptr: virtual address at which the start of the program resides
e_phoff, uintptr: byte offset from the start of the file at which the program header table is located
e_shoff, uintptr: byte offset from the start of the file at which the section header table is located
e_flags, uint32: processor-specific flags
e_ehsize, uint16: the number of bytes in this ELF header
e_phentsize, uint16: the number of bytes in one entry in the program header table (all entries are the same size)
e_phnum, uint16: the number of entries in the program header table
e_shentsize, uint16: the number of bytes in one entry in the section header table (all entries are the same size)
e_shnum, uint16: the number of entries in the section header table
e_shstrndx, uint16: the section header table index of the entry associated with the section name string table

It's giving us a few facts about the binary, then a list of offsets from the start of the file that indicate where each secion is located (everything that starts with sh). We'll use these fields to look up the section header table, read each entry in the table, and use those entries to find the debug sections we care about.

Note that in Go, uintptr is the built-in data type for an int of your machine's address size, meaning 4 bytes on 32-bit systems, and 8 bytes on 64-bit systems.

Also, In digging through the docs, you may have noticed some values such as LOPROC = 0xff00; HIPROC = 0xffff;. Both ELF and DWARF commonly reserve large ranges of high values for each processor, programming language, OS, etc. to define their own custom values for various enums. We won't be using these too much, but be aware that GNU, Go, Zig, and others commonly make use of these. You'll be able to get more information on each by reading through various compilers.

Parsing The Section Header Table

Next up, we need to parse each section header contained within the file. The "table" is just a fancy word for "an array of section header entries". So once we're done, we'll have a list of where all sections start and end within the binary, the name of each section, and some other data.

The section header table starts at the e_shoff'th byte in the file, and is e_shentsize * e_shnum bytes long.

The fields of each section header are:

sh_name, uint32: name of the section as an index in to the string table
sh_type, uint32: section type enum
sh_flags, uintptr: flags for this section
sh_addr, uintptr: the address at which this section should reside within the address space of the process, if relevant
sh_offset, uintptr: offset from the first byte of the ELF file to where the start of this section resides
sh_size, uintptr: the number of bytes in the section
sh_link, uint32: enum indicating the linkage of this section
sh_info, uint32: enum indicating extra information about this section
sh_addralign, uintptr: contraints on the alignment of addresses on the target platform (0 and 1 mean no constraints)
sh_entsize, uintptr: if the section contains a table of fixed-size elements (i.e. a symbol table), this is the size of each element

Read e_shnum entries, which should be exactly enough bytes. To give an example of how this might look in code, consider:

type ELFSectionHeader struct {
    sh_name      uint32
    sh_type      uint32
    sh_flags     uintptr
    sh_addr      uintptr
    sh_offset    uintptr
    sh_size      uintptr
    sh_link      uint32
    sh_info      uint32
    sh_addralign uintptr
    sh_entsize   uintptr

    // this is not part of the standard, but we'll
    // look up and store the name on this struct later
    name string
}

sectionHeaderTable := fileBuf[shOff : shOff+uintptr(shentSize*shNum)]
sectionHeaderTableReader := NewBinaryReader(
    bytes.NewBuffer(sectionHeaderTable),
    binary.NativeEndian,
)

sectionHeaders := []*ELFSectionHeader{}
for ndx := 0; ndx < int(shNum); ndx++ {
    header := &ELFSectionHeader{}
    header.sh_name, _ = Read[uint32](sectionHeaderTableReader)
    header.sh_type, _ = Read[uint32](sectionHeaderTableReader)
    header.sh_flags, _ = Read[uintptr](sectionHeaderTableReader)
    header.sh_addr, _ = Read[uintptr](sectionHeaderTableReader)
    header.sh_offset, _ = Read[uintptr](sectionHeaderTableReader)
    header.sh_size, _ = Read[uintptr](sectionHeaderTableReader)
    header.sh_link, _ = Read[uint32](sectionHeaderTableReader)
    header.sh_info, _ = Read[uint32](sectionHeaderTableReader)
    header.sh_addralign, _ = Read[uintptr](sectionHeaderTableReader)
    header.sh_entsize, _ = Read[uintptr](sectionHeaderTableReader)

    sectionHeaders = append(sectionHeaders, header)
}

Once we have all this information, we're going to want to use the sh_name field to look up our section name in the section header string table. This is the ELF section with index e_shstrndx, named .shstrtab. In my case with the test C program, it's the 35th section, though yours may be different.

This table is a series of null-terminated strings all next to each other in one long array. You can read the entire table in to an array, then use the sh_name field to find the entry at that index.

I'll use the sh_size and sh_offset fields of the e_shstrndx'th entry to find our location within the binary:

sectionNames := sectionHeaders[shStrTabNdx]

start := sectionNames.sh_offset
end := start + sectionNames.sh_size
sectionNamesBuf := fileBuf[start:end]

for _, header := range sectionHeaders {
    for ndx := header.sh_name; ; ndx++ {
        ch := sectionNamesBuf[ndx]
        if ch == 0 {
            break
        }
        header.name += string(ch)
    }
}

Now we're able to look up each debug information section by name!

Debug Info Sections

There's a fair number of sections in there! You may recognize some of them, but for the most part, we care about the ones that start with .debug_, though we also care about .eh_frame. If you want to check your work, you can with readelf --headers cloop. We'll get in to what each of these sections mean over time.

There are actually a few sections that are missing from the binary on my machine that we also would want to save if they were present (there are some sections that were present in older versions of DWARF for instance, but were dropped when v5 was released). We'll want to take the content of each one of those sections and save them for parsing later:

    type DWARFSections struct {
        abbrev      []byte
        line        []byte
        info        []byte
        addr        []byte
        aranges     []byte
        frame       []byte
        eh_frame    []byte
        line_str    []byte
        loc         []byte
        loclists    []byte
        names       []byte
        macinfo     []byte
        macro       []byte
        pubnames    []byte
        pubtypes    []byte
        ranges      []byte
        rnglists    []byte
        str         []byte
        str_offsets []byte
        types       []byte
    }

    getSection := func(header *ELFSectionHeader) []byte {
        start := header.sh_offset
        end := header.sh_offset + header.sh_size
        return fileBuf[start:end]
    }

    sections := &DWARFSections{}
    for _, header := range sectionHeaders {
        switch header.name {
        case ".debug_abbrev":
            sections.abbrev = getSection(header)
        case ".debug_line":
            sections.line = getSection(header)
        case ".debug_info":
            sections.info = getSection(header)
        case ".debug_addr":
            sections.addr = getSection(header)
        case ".debug_aranges":
            sections.aranges = getSection(header)
        case ".debug_frame":
            sections.frame = getSection(header)
        case ".eh_frame":
            sections.eh_frame = getSection(header)
        case ".debug_line_str":
            sections.line_str = getSection(header)
        case ".debug_loc":
            sections.loc = getSection(header)
        case ".debug_loclists":
            sections.loclists = getSection(header)
        case ".debug_names":
            sections.names = getSection(header)
        case ".debug_macinfo":
            sections.macinfo = getSection(header)
        case ".debug_macro":
            sections.macro = getSection(header)
        case ".debug_pubnames":
            sections.pubnames = getSection(header)
        case ".debug_pubtypes":
            sections.pubtypes = getSection(header)
        case ".debug_ranges":
            sections.ranges = getSection(header)
        case ".debug_rnglists":
            sections.rnglists = getSection(header)
        case ".debug_str":
            sections.str = getSection(header)
        case ".debug_str_offsets":
            sections.str_offsets = getSection(header)
        case ".debug_types":
            sections.types = getSection(header)
        }
    }

Now we're almost ready to start parsing those debug info sections in to something that allows us to inspect a running program!

PIE 🥧

The last thing we'll want to do with our ELF file for now is examine it to determine if it is a position independent executable (PIE, also known as position independent code or PIC). PIE means that the code can be loaded and executed at any address in the process' memory space, and is the opposite of aboslute code, which must be loaded at a fixed address in memory. You can enable PIE with the -fPIC compiler flag in gcc and clang. It ultimately doesn't restrict our capabilities as a debugger at all, it just means that we need to look up where in the process' address space our code is loaded when we start the program (we'll do that much later).

For now, we can determine if we're PIC based on the value of the FLAGS_1 field in the .dynamic section like so:

var dynamicHeader *ELFSectionHeader
for _, header := range sectionHeaders {
    if header.name == ".dynamic" {
        dynamicHeader = header
        break
    }
}

pie := false
dynamicBuf := getSection(dynamicHeader)
dynamicReader := NewBinaryReader(bytes.NewBuffer(dynamicBuf), binary.NativeEndian)
for {
    tag, _ := Read[uintptr](dynamicReader)
    val, err := Read[uintptr](dynamicReader)

    if tag == 0x6fff_fffb { // DT_FLAGS_1
        if (val & 0x0800_0000) > 0 { // DF_1_PIE
            pie = true
            break
        }
    }

    if err == io.EOF {
        break
    }
}

You can check your work on this using readelf --dynamic cloop. You may want to try compiling cloop with -fPIE and without -no-pie and re-running your parser to make sure things are looking good.

Summary

That's it for today! We learned what ELF and DWARF are as well as how to parse just enough ELF to get the debug information sections we care about. ELF is probably the easiest section in this series, so strap in.

Thank you for reading the series on DWARF. Please don't hesitate to reach out with comments, questions, or errata to jim at this domain dot com.

How DWARF Works: Table of Contents and Introduction

2024-09-25T00:00:00Z

How DWARF Works: Table of Contents and Introduction - Jim Calabro

Jim Calabro

About

RSS | Atom | JSON

---

How DWARF Works: Table of Contents and Introduction

Sep 25, 2024

Part One: Parsing Debug Info

Parsing ELF Files
Debug Information Entries
Line Number Information
Address Ranges
Frame Tables

Part Two: Using Debug Info at Runtime

Inspecting a Child Process
Stack Unwinding
Finding Variable Values

Appendix

Go Code Reference

Introduction

Welcome to the series on parsing and using DWARF debug info!

Its purpose is to provide a user-friendly starting point for learning about how debug information and debuggers work on Linux. It's written from the perspective of a debugger author, not a compiler author.

Non-goals include being a 100% comprehensive guide, providing details on specific languages or compilers, or giving details on other platforms.

If you want to learn more beyond what's contained in these posts, I highly recommend reading all relevant versions of the source documentation as well as other reference implementations that are pretty decent (links below).

If you have questions, comments, or corrections, please don't hesitate to reach out via email to jim at this domain dot com. I love hearing from you.

Thanks for reading!

Why Are You Writing This?

I feel strongly that the state of the art in debuggers needs to be improved on Linux. We have gdb and lldb (plus about a million graphical front ends, none of which are good). All of these tools take a long time to learn, are slow to use, and they don't make the data you need readily apparent.

There's also rr and Pernosco, which are astounding technical achievements, but suffer the same issues of taking a long time to get from "I don't understand what my program is doing" to "I understand what my program is doing" (especially for simple issues). They do a great job of helping solve some of the hardest bugs much faster, but they're not tools I use every day.

Comparatively, Windows has Visual Studio, WinDbg, x64dbg, RemedyBG, and the RAD Debugger. Each of these tools do the job of making the information I need about my program available at my fingertips as fast as possible, but unfortunately, they all are Windows-only. Despite its incredible bloat, it's awesome to be able to just press F5 in Visual Studio and have a ton of helpful information at your disposal.

Graphical debuggers are hands-down better than terminal based ones because debugging is fundamentally a problem of visualization. I want to see lots of relevant information all at once every time I step, and the terminal doesn't facilitate that nearly as well as a purpose-built GUI.

Additionally, terminal based debuggers are inherently worse than graphical ones because they dont allow me to get at the information I need quickly. Starting a debugging session by writing a .gdbinit file and using the REPL is much slower than using Visual Studio, where you just press F5.

I find that most people simply don't use gdb and instead reach for printf-debugging since it's quicker and easier for the vast majority of use-cases. As a practitioner, that's the correct choice, because what is the point of a debugger that doesn't save you time? However, it leaves a ton of power on the table and is a local maximum that we must push through.

I am certain this situation can be improved. The tools to write a robust grapical debugger for Linux already exist; it's just a long, tedious road to actually get it done.

Given that Linux is the most widely deployed operating system on the planet^{1, 2}, further investment in tooling in this area is a no-brainer. Hopefully this series can contribute towards a bright future of Linux development tools in some small way.

Useful Tools

How DWARF Works: Go Code Reference

2024-09-25T00:00:00Z

How DWARF Works: Go Code Reference - Jim Calabro

Jim Calabro

About

RSS | Atom | JSON

---

How DWARF Works: Go Code Reference

Sep 25, 2024

This is part of the series on DWARF.

This page contains Go helper snippets that illustrate how parsing works, but are used across many sections. These snippets often intentionally ignore error handling in favor of brevity.

LEB128

A key integer type that is used all over the place in DWARF is signed/unsigned LEB128. It's a variable-width signed/unsigned integer type, though in practice, I've never seen anything use more than 64 bits, but it's perfectly reasonable to assume that's not always the case.

I wrote a simple library to parse these a couple years ago, and I'm going to be using it throughout the series. You can import it just like you would any other Go library, or read its implementation and write your own. Zig also has a good implementation for reference.

Initial Length

An initial length field in DWARF is a special integer encoding that takes either 4 or 12 bytes and allows you to encode either a 4 or 8 byte integer. To parse one, you read a 4-byte integer, and if it's any value other than 0xffffffff, you use it as-is. If it's that special value, you read another eight bytes, discard the initial four bytes you read, and just use the final eight bytes.

func readInitialLength(reader *BinaryReader) uintptr {
    val32, _ := Read[uint32](reader)
    if val32 != 0xffffffff {
        return uintptr(val32)
    }

    val64, _ := Read[uint64](reader)
    return uintptr(val64)
}

Binary Reader

Go has an encoding/binary package, but it doesn't allow for reading arbitrary integers, nor does it allow us to know how far along the buffer we've read (our offset). We can wrap it a bit more nicely and ensure we're always reading in the correcty byte order using our own BinaryReader:

type Signed interface {
	~int | ~int8 | ~int16 | ~int32 | ~int64
}

type Unsigned interface {
	~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 | ~uintptr
}

type Integer interface {
	Signed | Unsigned
}

type BinaryReader struct {
    *bytes.Buffer

    startLen int
}

// Returns how many bytes we've read since the start of the buffer
func (br *BinaryReader) Offset() int {
    return br.startLen - br.Len()
}

func NewBinaryReader(r *bytes.Buffer) *BinaryReader {
    return &BinaryReader{Buffer: r, startLen: r.Len()}
}

func Read[T Integer](br *BinaryReader) (T, error) {
    empty := *new(T)
    size := unsafe.Sizeof(empty)
    enc := binary.NativeEndian

    // read N bytes br the reader
    buf := make([]byte, size)
    err := binary.Read(br, enc, buf)
    if err != nil {
        return empty, err
    }

    // convert to the appropriate type
    val := *new(T)
    switch any(val).(type) {
    case int8:
        val = T(int8(buf[0]))
    case uint8:
        val = T(uint8(buf[0]))

    case int16:
        val = T(int16(enc.Uint16(buf)))
    case uint16:
        val = T(enc.Uint16(buf))

    case int32:
        val = T(int32(enc.Uint32(buf)))
    case uint32:
        val = T(enc.Uint32(buf))

    case int, int64:
        val = T(int64(enc.Uint64(buf)))
    case uint, uint64:
        val = T(enc.Uint64(buf))
    case uintptr:
        if size == 4 {
            val = T(uintptr(enc.Uint32(buf)))
        } else if size == 8 {
            val = T(uintptr(enc.Uint64(buf)))
        } else {
            return empty, fmt.Errorf("word size %d not supported", size)
        }

    default:
        return empty, fmt.Errorf("unknown data type during load: %T", val)
    }

    return val, nil
}

Hello!

2024-07-11T00:00:00Z

Hello! - Jim Calabro

Jim Calabro

About

RSS | Atom | JSON

---

Hey! I'm Jim, and this is my site.

Most of the content here will be about computing, technology, and business. There may also be some occassional stuff on cycling, games, music, or something else.

The cadence of writing will be relateively lax; I woudln't expect more than a post every few months. I'll try my best to keep you appraised of the stuff I'm working on.

Thanks for reading!

Jim Calabro

Building Statically Linked Go Executables with CGO and Zig

Building Statically Linked Go Executables with CGO and Zig

uscope Update #2

uscope Update #2

Updates

What's Next?

Final Words

uscope Update #1

uscope Update #1

uscope: A New Debugger and Introspection Toolchain

uscope: A New Debugger and Introspection Toolchain

Why Are You Doing This?

What's the Plan?

How do You (yes you!) Get Involved?

How DWARF Works: Debug Information Entries

How DWARF Works: Debug Information Entries

Debug Information Entries

Abbrev Tables

Compilation Unit Headers

Compilation Units

Summary

How DWARF Works: Parsing Just Enough ELF

How DWARF Works: Parsing Just Enough ELF

What are ELF and DWARF?

Our Test Program

Parsing The ELF File Header

Parsing The Section Header Table

Debug Info Sections

PIE 🥧

Summary

How DWARF Works: Table of Contents and Introduction

How DWARF Works: Table of Contents and Introduction

Table of Contents

Part One: Parsing Debug Info

Part Two: Using Debug Info at Runtime

Appendix

Introduction

Why Are You Writing This?

Further Reading

Useful Tools

How DWARF Works: Go Code Reference

How DWARF Works: Go Code Reference

LEB128

Initial Length

Binary Reader

Hello!