CUDA disable L1 cache only for one variable

Question

Is there any way on CUDA 2.0 devices to disable L1 cache only for one specific variable? I know that one can disable L1 cache at compile time adding the flag -Xptxas -dlcm=cg to nvcc for all memory operations. However, I want to disable cache only for memory reads upon a specific global variable so that all of the rest of the memory reads to go through the L1 cache.

Based on a search I have done in the web, a possible solution is through PTX assembly code.

Answer 1

As mentioned above you can use inline PTX, here is an example:

__device__ __inline__ double ld_gbl_cg(const double *addr) {
  double return_value;
  asm("ld.global.cg.f64 %0, [%1];" : "=d"(return_value) : "l"(addr));
  return return_value;
}

You can easily vary this by swapping .f64 for .f32 (float) or .s32 (int) etc., the constraint of return_value "=d" for "=f" (float) or "=r" (int) etc. Note that the last constraint before (addr) - "l" - denotes 64 bit addressing, if you are using 32 bit addressing, it should be "r".

Answer 2

Inline PTX can be used to load and store the variable. ld.cg and st.cg instructions only cache data in L2. The cache operators are described in section 8.7.8.1 Cache Operators of the PTX ISA 2.3 document. The instructions or interest are ld and st. Inline PTX is described in Using Inline PTX Assembly in CUDA.

Answer 3

If you declare the variable to be volatile, then it will only be cached in the L2 cache on Fermi GPUs. Note that some compiler optimizations, such as removing repeated loads, are not performed on volatile variables because the compiler assumes they may be written by another thread.