r/rust•
Hiroyu

Problem with calling functions from a vtable

So I was trying to write a small wrapper lib for a dll and linking to the two exported functions works perfectly fine. So my current problem is that I get a vtable from this dll and I am also able to call the given functions of it, but this is only the case when the functions I am calling do not take any parameters. If I try to call a function of the vtable that takes any amount of parameters with the needed parameters the program ends up crashing due to an EXCEPTION_ACCESS_VIOLATION. The library is written in c++ so I was wondering if this could be the reason why I have problems, but afaik it shouldnt be. I cannot figure out what the problem is so I am asking here in hope that someone here know what my problem could be.

EDIT: My problem has been solved, my mistake was that I forgot to add the object itself as the first parameter to the function calls!

test.rs

extern crate gfxfilemanager;
use std::ptr::null_mut;

use std::os::raw::{c_char, c_int};
use std::ffi::{CStr, CString};

use gfxfilemanager::{IFileManager};

fn main() {
    let mut object = IFileManager::new(1);//create the object that holds the vtable
    unsafe {
        use std::mem::transmute;

        //the function signature of function0 is int Mode(void)
        println!("{}", transmute::<*const (), extern "thiscall" fn() -> c_int>((*object).get_func(0) as *const ())());//this return 1, the value of IFileManager::new(1)

        let name = CString::new("Test").unwrap();
        let pw = CString::new("pw").unwrap();
        //the function signature of function3 is int OpenContainer(const char *filename, const char* password, int mode)
        let fnc = transmute::<*const (), extern "thiscall" fn(*const c_char, *const c_char) -> c_int>((*object).get_func(3) as *const ());
        println!("{}", fnc(name.as_ptr(), pw.as_ptr()));//this crashes
    }
}

lib.rs

use std::os::raw::{c_char, c_int};

#[repr(C)]
pub struct Vtable(::std::os::raw::c_void);

impl Vtable {
    fn get_func(&self, index: isize) -> usize {
        unsafe {
            ::std::ptr::read((self as *const _ as *const usize).offset(index))
        }
    }
}
#[repr(C)]
#[derive(Debug)]
pub struct IFileManager {
    pub vtable: *const Vtable,
}

impl IFileManager {
    pub fn new(mode: c_int) -> *mut IFileManager {
        let mut obj = ::std::ptr::null_mut();
        unsafe { GFXDllCreateObject(mode, &mut obj, 0x1007); }
        obj
    }

    pub fn get_func(&self, index: isize) -> usize {
        unsafe {
            (*self.vtable).get_func(index)
        }
    }
}

impl Drop for IFileManager {
    fn drop(&mut self) {
        unsafe {
            GFXDllReleaseObject(self);
        }
    }
}

#[link(name = "GFXFileManager")]
extern "stdcall" {
    pub fn GFXDllCreateObject(mode: c_int, object: *mut *mut IFileManager, version: c_int) -> c_int;
    pub fn GFXDllReleaseObject(object: *mut IFileManager) -> c_int;
}    

Posted Jul 21, 2017, 9:49 PM UTC · edited

57
Loading comments…