I want to execute an untrusted .lua file in its own environment by calling lua_setfenv() so that it cannot affect any of my code.
The documentation for that function tho
luaL_loadfile() will load the chunk, then call lua_setfenv() to set the environment table, then call lua_pcall() to execute the chunk. See recent answer given by Judge Maygarden at Calling lua functions from .lua's using handles?
luaL_loadfile()
lua_setfenv()
lua_pcall()