Embedding Binary Files in C/C++

04 May 2023

Since we have a barebone approach, we rarely use Visual Studio’s resource system. We thought we would give away the embedding tool we made so you can embed binary files into your code by converting them into arrays in headers. It’s nothing fancy or novel but it has a couple of neat features and it works well.

So the general idea is that we generate arrays that contain the binary content of binary files, this is not a new technique and I’m sure that a lot of you are aware of it already, especially if you did some work on consoles that had cartridges or any coding prior to the mid 90’s for that matter.

We had a few added criterias:

• Since we use Perforce, we wanted the embedder to checkout or add files automatically
• We wanted this to be automated in the compilation process in Visual Studio
• Some projects involve a lot of files to embed, so we wanted to be able to batch process them
• We only wanted to generate new C arrays of our embedding content only when the source files changed

We won’t go through the implementation details since we are giving the code away and you can check it out or modify it at your leisure.

Here are the options it has:

Usage: embedder [-h|--help] [-b|--bigendian] [-c|--count] [-d|--decimal] [-f|--force] [-i|--identifier value]
                [-l|--list value] [-p|--perforce] [-r|--recent] [-s|--size value] [-t|--type value] [-u|--unsigned]
                [-v|--variable] [-w|--width value] [-z|--zero] <input file> <output file>
Options:
 -h, --help                             print this help
 -b, --bigendian                        Use big endian format for values (default littleEndian)
 -c, --count                            Add array count inside[]
 -d, --decimal                          Output decimal values
 -f, --force                            Force re-embedding
 -i, --identifier value                 Specify identifier name for array
 -l, --list value                       Export all the files listed in the file specified
 -p, --perforce                         Automate perforce checkout
 -r, --recent                           Embed only if source is more recent
 -s, --size value                       Specify entry size in bits (8/16/32/64, default 8)
 -t, --type value                       Specify entry type (defaults to the unsigned type of the entry size)
 -u, --unsigned                         force decimal output to be unsigned
 -v, --variable                         Generate count variable
 -w, --width value                      width, in columns, of the printed lines
 -z, --zero                             Generate leading zeros

All of these are self explanatory, I’ll spend a bit of time on the --list value option though. This option expects a filename that contains all of the files you want to embed. Here is a sample of a list file.

TTF_DejaVuSB,       .\data\fonts\DejaVuSans-Bold.woff2,               .\code\embedded\TTF_DejaVuSB.h		
TTF_DejaVuS,        .\data\fonts\DejaVuSans.woff2,                    .\code\embedded\TTF_DejaVuS.h
TTF_LibeMono,       .\data\fonts\LiberationMono.woff2,                .\code\embedded\TTF_LibeMono.h
TTF_TermL,          .\data\fonts\Iosevka-Term-Light.woff2,            .\code\embedded\TTF_TermL.h
TTF_TermO,          .\data\fonts\Iosevka-Term-Oblique.woff2,          .\code\embedded\TTF_TermO.h
TTF_TermBI,         .\data\fonts\Iosevka-Term-Semibold-Italic.woff2,  .\code\embedded\TTF_TermBI.h
TTF_TermB,          .\data\fonts\Iosevka-Term-Semibold.woff2,         .\code\embedded\TTF_TermB.h
TTF_UiB,            .\data\fonts\WeblySleekUISemibold.woff2,          .\code\embedded\TTF_UiB.h
SVG_IconClose,      .\data\icons\close_icon.svg,                      .\code\embedded\SVG_IconClose.h
SVG_IconMaximize,   .\data\icons\maximize_icon.svg,                   .\code\embedded\SVG_IconMaximize.h
SVG_IconMinimize,   .\data\icons\minimize_icon.svg,                   .\code\embedded\SVG_IconMinimize.h
SVG_IconToolClose,  .\data\icons\ios-close-circle-outline.svg,        .\code\embedded\SVG_IconToolClose.h
SVG_IconPanelMenu,  .\data\icons\md-menu.svg,                         .\code\embedded\SVG_IconPanelMenu.h
SVG_IconIoFolder,   .\data\icons\ios-folder-outline.svg,              .\code\embedded\SVG_IconIoFolder.h

[...]

Every line corresponds to a file to embed, they each have 3 comma-separated entries. The 1st entry is the identifier you want your array to have in your code once this is embedded, the 2nd entry is a path to the binary source file and the last one is the header file you want it to generate. So if you save this in a file, say list.txt and you pass -l list.txt to the tool, it will go through all the files and embed them.

Using this system, you don’t have granular control over the options for each entry, they will all use the options you pass in parameters to the tool. For example, if you add -z to generate leading zeros, all of your files listed will have leading zeros and you couldn’t, say, make the third file in the list not have leading zeros. If you want selective control, you could still split them in multiple list files and call the tool with different options using their respective list file. In practice, we noticed that we always use the same settings so we didn’t bother giving granular control.

Organizing these header files is a bit of a necessary hassle and you’ll have to update them once you add new embedding headers. The cleanest way we found is to add all the headers in a cpp file called embedded.cpp and add externs in embedded.h and include that one throughout our code. This could be automated and we probably will but we didn’t want to share code that enforced a specific organization in your own code base so you could go about it as you pleased. we’ll give you the code so it’s trivial to implement.

We’re using Visual Studio 2017 so it should be easy convert the project to later versions. Knowing where to add the custom build step was a bit annoying but I think we found the best place for the embedding to happen, just add a call to a batch file just before executing PrepareForBuild

Just make you convert.bat and call the embedder in there, we strongly recommend using -l list_filename so you can batch-embed files instead of calling the tool a gazillion times in your batch file. Using -r is also super useful since it will cache the file dates and only re-embed the files that changed. In our case, -p is also invaluable since it will either checkout or add files to our repo when data source files are added or modified. For perforce to work, the destination should be under the base directory of your depo.

Obviously, you need to add and maintain the generated headers in your project but once set up, it’s all automated.

We revamped the code a lot to make this presentable (lol) there could still be a few bugs here and there but we did a summary testing pass and it seemed to work well.

A few caveats

• if you force a type name using the -t option and it doesn’t match the size specified by -s this could cause compilation errors.
• If you select a size, say 32 bits and the content of your file is not a multiple of 32 bits, it’ll pad the missing data at the end with zeros.
• The bigendian mode is rarely useful since endianness is data-type dependent and these may not be aligned. Use it only if you know that the binary source is a continuous array of the specified size.

That’s all folks, we hope you enjoy this little tool.

Click here to Download the embedder

Comments