It’s been a long time since I posted anything here. Sorry for my inconsistency.

But today is a special day, because with this post, you’ll be able to do magical things. Yes, I really mean it.
Previously, in an other post I showed you how you could have a « hot reload » feature in your project (specifically using the raylib framework, but that’s the same principle for whatever C/C++ project anyway) in C.

Today, I’m going to show you something much more simpler, with potentially the same capabilities: make a program which can read C code and compile & run it on the fly. So to be shorter: how to use C as a « scripting » language for your project.

After that, you’ll be able to do something pretty similar as below. It’s pure C, no hot reload via dynamic library, everything is recompiled (at high speed, 9x faster than gcc) & rerun at runtime whenever the file is modified. Pretty cool no?

Warning

This post targets the windows OS with the mingw64 toolchain, but actually, it can easily be adapted to work on linux too. I managed to have the same result pretty easily on my laptop which runs PopOs, so a Ubuntu 16 based linux (just be sure to link with -ldl), by using GCC 8.1.

To be able to do that, we will use the amazing tcc compiler. It’s a really small C compiler (C99 mostly, there are just few of the C11 features supported), which has been made by Fabrice Bellard, an amazing french (cocorico, I’m french too 🙂 ) programmer.

Actually, you can compile this tiny C compiler in a library. So… You’ve got the point. You can embed this library and compile and run code 9x faster than GCC, on the fly, during runtime.

To do so, first, we are going to download the source code of TCC, which are available here, on the official website: https://download.savannah.gnu.org/releases/tinycc/

Then we need to extract this tar.bz2 file. You can use a tool like winrar for that, or the tar command line, which is included with a bash terminal (like gitbash).

Inside the folder where you extracted the TCC sources, run from the same bash terminal:

sh ./configure

Then type mingw32-make libtcc.a to create the libtcc static library which we will link against in our program.

mingw32-make libtcc.a

Now, we are going to create a new C project. Create a new directory for your project, and create 3 new directories inside it: src for the source code, include for the headers, and lib for the static libraries. Or do whatever you want, but classicly, any C project has something similar to that folder organization, and I will refer to that for the rest of this article.

Copy the libtcc.a static library from the TCC directory, that you created with the previous command line, inside your fresh created lib directory.
You can copy as well the libtcc.h, and the tcclib.h files, which are at the root of this directory, and every files which are inside the include directory (at least stdarg.h and stddef.h) inside your include directory.

libtcc.h is the header we will include in our main program to be able to call the libtcc functions, which will compile and run C code at runtime.
tcclib.h and the other headers or necessary to give the minimal C standard to the C code we will compile at runtime.

Now everything is setup, we need to do one more thing. We need to compile the whole TCC to create the libtcc1.a static library, which will not be linked staticly to our program, but to the C code we want to compile and run at runtime. Actually this library contains every definitions of the tcclibh, stdarg.h, and stddef.h include files.
So go to your TCC directory, and simply run:

mingw32-make libtcc1.a

Copy this new libtcc1.a static library you just created with this cmd to your lib folder.

Now every thing is set. We are going to create a basic C program, which will create a tcc context, and compile & execute some code at runtime. That’s the exciting part!

Go to your project directory, inside src create a C file. I will simply call it main.c, and add these lines to it:

#include <stdio.h>
#include <libtcc.h>

const char* program = \
    "int foo(const int in_value){"
    "   printf(\"this is a test: %d\n\", in_value);"
    "}";


int main(int argc, char **argv)
{
    TCCState* s = tcc_new();
    if(!s){
        printf("Can’t create a TCC context\n");
        return 1;
    }
    tcc_set_output_type(s, TCC_OUTPUT_MEMORY);
   
    if (tcc_compile_string(s, program) > 0) {
        printf("Compilation error !\n");
        return 2;
    }

    tcc_relocate(s, TCC_RELOCATE_AUTO);

    int (*const foo)(const int in_value) = tcc_get_symbol(s, "foo");
    foo(32);
   
    tcc_delete(s);
    return 0;
}

You can compile this main.c file with:

gcc main.c -I./include -L./lib -ltcc -o main.exe

And if you execute this main.exe file, you should get this output inside your terminal:

./main.exe
<string>:1: warning: implicit declaration of function ‘printf’
this is a test: 32

So we have a warning, but we have our result! We compiled successfully and run C code, which is contained inside the program variable, at runtime!
Now, I’m going to explain to you what each line actually does, and what we could do to remove this anoying warning we get from the libtcc compilation step (for those of you who are experimented in C or C++, you actually might already know why we have this warning).

We got it because the program we compiled with libtcc doesn’t know what is the declaration of the printf function (even if that’s worked). To get rid of this warning, we simply need to include tcclib.h.

const char* program = \
    "#include <tcclib.h>"
    "int foo(const int in_value){"
    "   printf(\"this is a test: %d\n\" , in_value);"
    "}";

After compiling again, no more warning!

So now, it’s time to get a little bit of an explaination about what’s happening here.
The first thing we do in our program is creating a TCC context, which set everythings up for the compilation at runtime. Of course, you know if you are familiar in C/C++ code that every time a function gives you a pointer to something in return, that means it allocate some memory and you need to test if this memory is correclty allocated by testing the adress of the pointer.

    TCCState* s = tcc_new();
    if(!s){
        printf("Can’t create a TCC context\n");
        return 1;
    }

Because we allocated some memory, we need to free it when we don’t need it anymore, in our case, after the compilation step. This is done by the tcc_delete function call, at the end.

    tcc_delete(s);

Now we have our tcc context, we are going to choose which « ouput » our program should compile for with the tcc_set_output_type.
If you take a look at the libtcc.h header file, you’ll see that there are several ouputs forms, and that gives you a hint for when you should call this function:

/* set output type. MUST BE CALLED before any compilation */
LIBTCCAPI int tcc_set_output_type(TCCState *s, int output_type);
#define TCC_OUTPUT_MEMORY   1 /* output will be run in memory (default) */
#define TCC_OUTPUT_EXE      2 /* executable file */
#define TCC_OUTPUT_DLL      3 /* dynamic library */
#define TCC_OUTPUT_OBJ      4 /* object file */
#define TCC_OUTPUT_PREPROCESS 5 /* only preprocess (used internally) */

So you see that if you want to be able to compile & run C code direclty inside your program, the output needs to be in memory.
But you could also compile code at runtime, and create a static library, or an executable file from it.

The next step is the most important one. It’s the compilation step. It’s at this step the magic happens.

    if (tcc_compile_string(s, program) > 0) {
        printf("Compilation error !\n");
        return 2;
    }

tcc_compile_string is quite explicit… It takes a string, try to compile it, and return 0 if something was wrong during the compilation step. If everything correclty compiled, the machine code is loaded into memory.

Now we move to the next step. We need to call the tcc_relocate function to be able to call the tcc_get_symbol function, which gives us back a symbol from the compiled code that we can use later (in our case, the foo function).
Again, it’s well explained in the libtcc.h header file:

* do all relocations (needed before using tcc_get_symbol()) */
LIBTCCAPI int tcc_relocate(TCCState *s1, void *ptr);
/* possible values for ‘ptr’:
   – TCC_RELOCATE_AUTO : Allocate and manage memory internally
   – NULL              : return required memory size for the step below
   – memory address    : copy code to memory passed by the caller
   returns -1 if error. */
#define TCC_RELOCATE_AUTO (void*)1

So with that information, we know we need this line at this step, and let libtcc manage the memory internally for us:

tcc_relocate(s, TCC_RELOCATE_AUTO);

Then, and only then, we can get the function back with the tcc_get_symbol call, which actually works exactly like symbols from a dll file. We can then store it inside a pointer to a const function, which I simply called with the exact same name foo here.
Of course, you need to define this function pointer with the exact same parameters you defined inside the program variable.


You could add an additional step after assigning this function pointer, to check if it’s null or not of course.
After getting this function back inside our foo variable, we can call it like any regular function.

// get the "foo" function address from the code compiled, and assign it to this new foo function pointer
    int (*const foo)(const int in_value) = tcc_get_symbol(s, "foo");
   
    foo(32);  // execute this function

And… That’s all actually!
You see, nothing complicated, the libtcc API is well designed and super easy to use. Now you are able to compile C code and run it at runtime.
You can do whatever you want with that, like hot reloading (see the gif from above for example), by watching when a file is modified, and recompile it (just look inside the libtcc.h header file to get the right function, in replacement of tcc_compile_string, to be able to compile an entire C file, and not just a char*).
Just a hint: on linux, you could use the stat system call function to get file informations 🙂

Now it’s your turn! Don’t hesitate to open the libtcc.h header file to see the functions you can use from the API. Everything is clearly commented/documented inside. For example, you could call a special function you created when an error occured during compilation (I let you find how to do that, just look for how to pass your function callback to libtcc).

I hope this tutorial will help you create awesome stuff. I’m looking forward to see how you implement this in your project, and for what purpose. Don’t hesitate to tell me in the comments below!

Victor Gallet

Victor Gallet

Programmeur jeu vidéo. J'aime par dessus tout apprendre, et je suis un éternel curieux de tout. Mon principal but dans la vie est d’être une meilleure personne, et de partager mes (faibles) connaissances avec les autres.

2 Comments

  • Marlon dit :

    Bravo! The post was very well written, I appreciate your explanation and energy. This was more or less of what I was looking for. If I understood correctly, this libtcc is only for C, right? I wanted to find something that worked with C++. Any clues?

    Merci beaucoup en tt cas !

    • Victor Gallet dit :

      Hey thanks for your answer.
      Actually, libtcc is the backend of the tcc compiler which only compiles C, not C++. But in C++ you can hot reload your code by encapsulate your code inside modules, and compiles them in DLL (on windows, or SO on linux). What you want to do is: save the state of the program, release the dll, recompile the dll with your change, reload the dll, repopulate the main program data with your new ones from the dll 😉

Leave a Reply

Ce site utilise Akismet pour réduire les indésirables. En savoir plus sur comment les données de vos commentaires sont utilisées.