This (very basic) question is the result of an exchange here.
The documentation for setkey() states:
setkey()
setkey() sorts a data.table
The rows are sorted. "Changed by reference" here means there is no copying of the entire table and rows are just swapped.
setkey(DT, NULL) is equivalent to setattr(DT, "sorted", NULL). It simply unsets the "sorted" attribute.
setkey(DT, NULL)
setattr(DT, "sorted", NULL)