Saturday, 10 January 2015

C++ Member Function Variables And Different Abi's

In Gcc, and all C++ compilers they use a structure to store information about member functions. A member function can be virtual or non-virtual. So doing something like...

auto TheFunctionX = &MyClass::Function;

Will yield a variable of a hidden structure type with two fields. These fields are later used to compute the real address of the function in memory. The structure variable might already contain the address if the function is not virtual but knowing that just from the structure requires knowing how the fields in the structure work. Unfortunately the details of how compilers implement this structure varies wildly, but for Gcc at least it is stable in that for a given ABI(Binary Interface), they will not change the details.

Assuming you can rely on the ABI you can then use your own code to interpret the real address of virtual functions and non-virtual member functions, even in interfaces. To do so you need the structure, which you can compute once knowing the type of the class, or just compute it each time you need it, and you also need the instance of the class for which you want the address. The reason you need the instance to get a virtual functions address is because the instance has the vTable. Technically for any given class instance(which is not using multiple inheritance), you can compute the address of any member function and save it.

The point is to understand that because you can compute the address, you can then go on to make other structures of your own to implement something which functions exactly like the events mechanism of Delphi.

This small class here shows an example of such a member function structure as used in the standard Itanium Abi for Gcc. Gcc supports different Abi's so it is not always the same. That requires more understanding of Abi in general between different Abi.

struct TGccMemberFunctionDetails {
   union {
     intptr_t non_virtual_address; // always even, the actual address of a normal non-virtual instance method
     intptr_t virtual_offset; // always odd, the offset into the vtable of the adjusted this pointer
   intptr_t delta;// The this pointer adjustment. Used for interfaces and multiple inheritance cases

   // Given an instance pointer aka "this", compute 2 things.
   // 1. The adjusted this pointer.
   // 2. The real address of the proc
   TComputedPmf Compute(void* cthis) {
      TComputedPmf result;
      result.inst = cthis + delta;
      if (non_virtual_address & 1) result.proc = (void*)(* (intptr_t**)( *(intptr_t**)result.inst + (virtual_offset+1)/2) + sizeof(void*));
      else result.proc = (void*)non_virtual_address;
      return result;

The Compute method returns another structure containing the address of the function and the corrected instance pointer(which for normal classes is the same as the input cthis).

Once you have the address and the corrected this you can the save these variables and use them like Delphi events by making a template class to wrap them in. Unfortunately it is not that type safe and you will have to watch your calling conventions but the general idea is to pass the corrected this argument first and then the rest of the parameters after the this. The call will go to a thunk if it was an interface or multiple inheritance method  requiring a thunk in either case.

If you are still reading then this is probably new to you, and you probably want Delphi or C# style events. So all you need to take from this brief introduction is faith in the concept and that it does work and that the execution speed is very fast to cal the function(the same as Delphi).

The problem then is how to make the event. It is very simple... This simple C++11 class here can serve you as a basis for all your events. If you need an event that returns void just define another class but omit the TResult type parameter and return nothing from the Call procedure. You can also overload the call operator to make the code look like its calling a function!

template<class TResult, typename... TArgs>
class TFunctionOfObject {
  typedef TResult (*TProc)(void*, TArgs...);
  TProc proc;
  void* inst;
 TFunctionOfObject () { }
  TFunctionOfObject (const TComputedPmf& pmf) {
    proc = pmf.proc;
    inst = pmf.inst;
  inline __attribute((force_inline))__  TResult Call(TArgs... args) {
    return proc(inst, args...);

// Declare an event proc that returns bool and takes one parameter which is an integer
using TMyEvent = TFunctionOfObject<bool,int>;

I leave the rest up to you as to how to derive the TComputedPMF structure that you can pass to the constructor. Or maybe I will edit this some more later.

Hint! I like to use a macro called _PMFX and the decltype variable to compute this on the fly, so I can say...

Font.OnChanged = _PMFX(this, OnFontChanged) ;

Then inside the macro I use the decltype of the this variable to get at the compilers structure for the member function, and then I use the Abi specific structure(like the Gcc example above), to get at the computed PMF and then it just goes into the constructor of the OnChanged field of Font in this example.

Debugging Vst Plugins Using Codeblocks...

Vst plugins are .dll files and so they need a host application to debug. The best application I have seen for doing this with using C++ is Codeblocks as the editor, with an application like Reaper. When you hit a breakpoint, control is transferred back to Codeblocks and Reapers threads and audio goes off while you are single stepping the code.

The steps to do this debugging of Vst and Vsti plugins are...

1) Get your Vst plugin going as a C++ or better C++11 project in Codeblocks.
2) In Codeblocks, there is a top level menu called "Projects". In this menu is an item called "Set Programs Arguments".
3) Go to the programs arguments panel and enter the full path to the Vst host such as  Reaper.exe(Or any other Vst host as well but Reaper works well and is free to try for this purpose).
4) Now build the program and then hit the debug button on the toolbar. The Vst host will run. Load your plugin into the host. In Reaper this means add it as an instrument or effect in the track.
5) Any breakpoints you have set in your Vst source code will now be hit when the plugin runs and you can debug the plugin like normal program debugging and do things like evaluating variables etc...

I do not know if the default compiler with Codeblocks will work well for this, because I downloaded a different newer MingW(Gcc 4.9), and I changed the Gcc path to the new MingW.

GCC C++11 Alternative To Try/Finally

In C++11 and C++ there is no try/finally that you can use. So often you have to make classes to do cleanup for you. But with the new C++11 there are now lambdas and you can use these as bound functions that can be executed later at anytime. In the case of finally we want a lambda that does the cleanup and is executed later on at the end of the scope, just like manual cleanup of some memory or files etc...

  This class defers a single void function call by wrapping it
  in a function and calling it later. Because this is C++11 you
  can pass it a lambda function.
class deferred_function_call {
  std::function<void(void)> the_function;
  inline deferred_function_call(const std::function<void(void)> &the_function) : the_function(the_function) { }
  inline ~deferred_function_call() { the_function(); }

#define _DO_TOKEN_COMBINE__(X,Y) X##Y  // helper macro

  What name you use to remember something to do at the end of the scope is your choice. I like
#define _FINALLY(...)    deferred_function_call _TOKEN_COMBINE__(_x_temp__,__LINE__) ([&]{ __VA_ARGS__ ; })
#define REMEMBER_TO(...) deferred_function_call _TOKEN_COMBINE__(_x_temp__,__LINE__) ([&]{ __VA_ARGS__ ; })
#define remember_to(...) deferred_function_call _TOKEN_COMBINE__(_x_temp__,__LINE__) ([&]{ __VA_ARGS__ ; })
#define RememberTo(...) deferred_function_call _TOKEN_COMBINE__(_x_temp__,__LINE__) ([&]{ __VA_ARGS__ ; })
#define defer(...) deferred_function_call _TOKEN_COMBINE__(_x_temp__,__LINE__) ([&]{ __VA_ARGS__ ; })

#define later(...) deferred_function_call _TOKEN_COMBINE__(_x_temp__,__LINE__) ([&]{ __VA_ARGS__ ; })

  This macro is for calling a function pair where the second call must execute.

    For example:
#define GuardedExpression(expression_start,expression_finally) expression_start; deferred_function_call _TOKEN_COMBINE__(_x_temp__,__LINE__) ([&]{ expression_finally;  })


// Here is some example code that shows how to use such a macro to make your code cleanup for you
int main {

  Example function calls that are made in order and must execute
class TCanvas {
  void MoveTo() {}
  void LineTo() {}
  void BeginDrawing() {}
  void EndDrawing() {}
  void MoveTo(int x, int y) {}
  void LineTo(int x, int y) {}
void BeginDrawing(TCanvas* canvas) {
  // Do stuff that happens before drawing, for example.
void EndDrawing(TCanvas* canvas) {
  // Do stuff that happens after drawing, for example.

int main()
    FILE *file = fopen("test_Cxx11_Finally","w");
    remember_to( fclose(file); );
    printf("Do stuff with the file...\n");

    // read something
    // write something

    printf("All went well, now the file will be closed by the remember_to...\n");
    /* This example assumes the existence of some TCanvas object. */
    TCanvas MyCanvas;

    printf("Drawing some stuff!\n");
    /* Or you can make it call the class methods. remember_to will take any code even a block of code... */
    TCanvas MyCanvas;
    printf("Drawing some stuff!\n");
    /* You could guard a new/delete sequence */
    TCanvas* MyCanvas = new TCanvas;
    RememberTo( delete MyCanvas );
    printf("Executing in a guarded section!\n");
    /* Or like this */
    GuardedExpression(TCanvas* MyCanvas = new TCanvas, delete MyCanvas);
    printf("Drawing inside a guarded section!\n");
      This naive example shows that you can use a block of code as
      well in the remember_to clause.
    int MyCounter = 0;
       // Do stuff here like a normal block, this will be executed
       // at the end of the enclosing scope.
       if (MyCounter != 10)
         printf("The code did not execute all the loop!\n");
    // Here it tries to loop and the remember clause will detect if the
    // loop did every iteration it was supposed to.
    while (MyCounter < 10) {
      if (MyCounter == (rand() % 30))
        throw "Something happened!";
    printf("Everything went fine!\n");


Saturday, 20 July 2013

initrd problem = Kernel panic - not syncing: No init found. Try passing init= option to kernel.

About Initrd
An initrd file, often zipped as initrd.gz is a file containing an "Initial Ram Disk" that Linux loads when it boots. The initrd can contain a very small set of programs needed to just work the computer, like checking files or formatting drives, or it can contain much more functionality, such as display managers and window managers and even GUI programs.

If you have ever tried to make an initrd yourself, you might have come across this problem...

1) The kernel boots
2) It finds the initrd file
3) Then this error here...
Kernel panic - not syncing: No init found. Try passing init= option to kernel.

And if you are like me, your init file is probably in the root directory as /init and it is probably a script linked to /bin/sh.

So I tried to copy over busybox and then make all the links, and I also copied all the needed libraries for busybox, which is just libc. Not matter what I tried I get "Kernel panic - not syncing: No init found. Try passing init= option to kernel." from the Linux kernel. This went on for like 8 hours of trying every single combination of things I could think of.

What does "Kernel panic - not syncing: No init found. Try passing init= option to kernel." mean?
A) It most normally means that the kernel can see your "init" script, but when it tries to load it in /bin/sh the sh program is throwing an exception, so the kernel error message can be misleading.
B) It can also mean that the initrd.gz file is corrupt(unlikely), or that the /init file is not there either, also unlikely. The error is most commonly generated from option A, so see solution 2 below for the proper fix to this annoying and misleading error message.

Solution 1 - Use Static Linking
I unzipped the initrd file from Puppy Linux 5.61 using cpio and gzip and I noted that the busybox inside the initrd.gz file is not the same as the busybox found once Puppy is booted. So I tried using this busybox inside the initrd instead and then it boots fine. The reason it works is because it is statically linked, it does not depend on any .so library files, but the real busybox when Puppy is running is linked to But this solution is really bad because now you have two copies of busybox in your distribution, one in the initrd.gz file and another somewhere else that gets loaded at the switch_root! I mean two copies of busybox is just not acceptable!

Solution 2 - Put the libraries!
So I was hunting around on the internet and went through five hundred forums where every guy was making all kind of guesses as to what the problem is but to no avail! So finally I simply copy the Puppy is like and, you need both the files and they are in /lib and are statically linked) files from /lib in Puppy 5.61 into the initrd, and it worked! In other words the kernel error message was caused because when it was trying to execute the /bin/sh program, the program tried to load, but in order to load it must also load first, which then actually loads on behalf of the program.

ldd is a program which tells you what libraries a program uses, but you need to be aware that any program which uses libraries also uses! Keep that in mind when making your initrd.gz file and you can forget about the init not found error message and get on with putting all your files from one distribution into your custom distribution.

Decreasing search popularity for Linux vs Windows

I was exploring some search terms on Google trends to see what is really happening in the world, because the news websites seem to be all advertisements. So I compared some common terms such as Windows, Java and C++ for example. As it turns out, Java is more than double everybody else in the programming department. Then I added Linux, which started out in 2005 with more searches than Java, but has since then till 2013 dropped off by a huge percentage and seems to be fading into obscurity, in direct contradiction to what you would think. But then I added in windows for comparison and the result was as expected, Windows is searched for more than all the other big keywords, and keywords like OsX an iOs will not even register on this graph.

Windows is much more popular than Linux and Osx and all the big programming languages
Click to enlarge
If you look at the picture you will see the purple line on the top is Windows, showing how much interest there actually is in the operating system. The green line is Linux and you can clearly see that Linux as a word seems to be around a lot less, which is surprising.

Friday, 19 July 2013

Run bash style shell scripts using UnxUtils

In Linux, all the scripts have the following line at the top(known as a shebang)...


So when you run the script, the shell knows to use the program /bin/sh to execute the script.

But this will not work for UnxUtils since there is no /bin directory. You can make one, but then you would have to do that for every partition on your computer, and then put sh.exe in every folder!

The solution is to change the "shebang" so that it looks like this instead...


I find that this allows you to do things like executing the script from the Unix shell in Windows.

Alternatively you can also do this...


But that seems a little non-portable and not so wise.

Unfortunately, however you get it to run, the scripts  then will not work in Linux. If you want your scripts to work in Linux then you must sadly create a bin directory with sh.exe or a bat file pointing to sh.exe in every partition on your hard disk. Then you can indeed use /bin/sh as the path of the shebang.

Monday, 15 July 2013

Processing Integers On 32 Bit vs 64 Bit Cpu

A number of questions that need to be answered when upgrading from 32 bit to 64 bit Cpu processing.
Q: Does a 64 bit integer on a 64 bit Cpu work slower than a 32 bit integer on a 32 bit Cpu?
A: Not for most operators. Operators like add and subtract and shift seem to be about the same. The exceptions are operators involving division. A 64 bit integer divide on a 64 bit Cpu is way slower than a 32 bit divide on a 32 bit Cpu.
Q: Is processing a big array of 64 bit integers on 64 bit Cpu slower than processing an array of 32 bit integers on a 32 bit Cpu?
A: Yes. 64 bit integers use more memory, so to get from one end of an array to the other will read in more memory from the RAM, and if you are writing back to the values then it will transfer more memory back to the RAM since a 64 bit integer is twice as large as a 32 bit integer. However processing an array of 32 bit integers on a 64 bit Cpu transfers just the same amount of memory as processing a 32 bit integer array on a 32 bit Cpu.
Q: Why did all the C++ compilers choose LLP64 and LP64 as the default model, which keeps int as 32 bits on 64 bit Cpu?
A: Mostly to keep compatibility with existing code but also because it requires less space in memory. Naive programs that just use int for everything in the ILP64 model will have to deal with more bytes being used up in the Cpu cache as a consequence of using 64 bit integers for everything. In most existing programs almost everything to do with lists and arrays and chunks of memory is in a context where the addressable indexes into the memory can all be contained within 32 bits. You will not find any 32 bit programs that use strings that are longer than 32 bits can store for example.

Q: Which integer type should be used for list indexing in C++ on 64 bit Cpu? 
A: For lists it is unlikely that you will ever need more than 2.4 billion items, at least with current systems, even 64 bit ones, so 32 bit is in 99.99% of cases ideal, since the extra 32 bits of a 64 bit are not needed. However progress is always going on so you might want to consider making a type called "ix_t" and using that instead for your list length fields and to index the lists. The problem is not just the indexing but if you had hundreds and thousands of objects in memory all of type "TList", then using a 64 bit integer would in all normal cases simply be unused 4 bytes on each list object.

Q: Which integer type should be used for array indexing on 64 bit Cpu?
A: This is a tricky question, since with array memory you can easily imagine people allocating large chunks of memory that cannot be indexed by a 32 bit int, for example, a 5Gb chunk of memory. However the practice of allocating very large chunks of memory is not yet common, and would be very rare in the overall usage of arrays in a program. Using a 64 bit integer for indexing does not make so much sense either. So if you are hyper optimizing some code, then for most arrays use a 32 bit integer and then a 64 bit integer only where you need it.

Otherwise the best thing to do is to benchmark your machine yourself. I have made a benchmark tool that builds in Codeblocks, and the download comes with a decent set of results for a Core 2 Duo Cpu.