问题
What is the internal operation that allows nvidia-smi fetch the hardware level details? The tool executes even when some process is already running on the GPU device and gets the utilization details, name and id of the process etc. Is it possible to develop such a tool at the user level? How is NVML related?
回答1:
Nvidia-smi is a thin wrapper around NVML. You can code with NVML with help of SDK contained in Tesla Deployment Kit.
Everything that can be done with nvidia-smi can be queried straight from the C library NVML. If you need to query this data in an app of some sort, it's better (and probably easier) to write against NVML instead of parse nvidia-smi stdout (which format changed in the past).
There are also python and perl bindings for NVML library. So you can use higher level programming langugage than C.
来源：https://stackoverflow.com/questions/14906675/how-does-nvidia-smi-work