Since the final function wasn't named by IDA I just named it 'ntdll_RtlAddVectoredExceptionHandler_Impl'. After reversing the structures I found out that Windows keeps a string of doubly linked list nodes that link a circle (last node points to the first node). The structure for a single node looks like this:
Windows uses the EncodePointer function to encode the handler that you pass through AddVectoredExceptionHandler and stores that inside the last member, EncodedHandler. The IsAllocated member is set to 1 once the function is sure that a node can be added and decremented when the node is removed (after calling RemoveVectoredExceptionHandler).
To get to the list of structures you have to access the exception list using this structure:
I don't know what this function is for but I know the exception list is stored. This address of this function is static and sadly I can only share the address for my version of ntdll (6.1.7601.19018), which is at address 0x7DF74744.
So to access the nodes you would do something like this:
That accesses the first node. In that case I added a top-most handler manually thus it should print the address of my handler.
But anyways without further a due here's the C++ equivalent of AddVectoredExceptionHandler:
I didn't change variable names but you can figure it out (hopefully).