Truncating a file while it's being used (Linux)

Question

I have a process that\'s writing a lot of data to stdout, which I\'m redirecting to a log file. I\'d like to limit the size of the file by occasionally copying the current

Answer 1

As of coreutils 7.0, there is a truncate command.

Answer 2

Redirect the output using >> instead of >. This will allow you to truncate the file without the file growing back to its original size. Also, don't forget to redirect STDERR (2>&1).

So the end result would be: myprogram >> myprogram.log 2>&1 &

Answer 3

I downloaded and compiled the latest coreutils so I could have truncate available.

Ran ./configure and make, but did not run make install.

All the compiled utilities appear in the "src" folder.

I ran

[path]/src/truncate -s 1024000 textfileineedtotruncate.log

on a 1.7 GB log file.

It did not change the size listed when using ls -l, but it did free up all the disk space - which is what I really needed to do before /var filled up and killed the process.

Thanks for the tip on "truncate"!

Answer 4

Take a look at the utility split(1), part of GNU Coreutils.

Answer 5

@Hobo use freopen(), it reuses stream to either open the file specified by filename or to change its access mode. If a new filename is specified, the function first attempts to close any file already associated with stream (third parameter) and disassociates it. Then, independently of whether that stream was successfully closed or not, freopen opens the file specified by filename and associates it with the stream just as fopen would do using the specified mode.

if a thirdparty binary is generating logs we need to write a wrapper which will rotate the logs, and thirdparty will run in proxyrun thread as below.

#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <unistd.h>
#include <string.h>

using namespace std;

extern "C" void * proxyrun(void * pArg){
   static int lsiLineNum = 0;
   while(1) 
   {
     printf("\nLOGGER: %d",++lsiLineNum);
     fflush(stdout);
   }
  return NULL;
}


int main(int argc, char **argv)
{
  pthread_t lThdId;
  if(0 != pthread_create(&lThdId, NULL, proxyrun, NULL))
  {
    return 1;
  }

  char lpcFileName[256] = {0,};

  static int x = 0;

  while(1)
  {
    printf("\n<<<MAIN SLEEP>>>");
    fflush(stdout);
    sprintf(lpcFileName, "/home/yogesh/C++TestPrograms/std.txt%d",++x);
    freopen(lpcFileName,"w",stdout);
    sleep(10);
  }

  return 0;
}

Answer 6

as the file is being used, if you try to nullify it or something like that, sometimes it might "confuse" the app that's writing into the log file and it might not log anything after that.

What I'd try ot do is to set up a kind of proxy/filter for that log, instead of redirecting to file, redirect to a process or something that would get input and write to a rolling file.

Maybe it can be done by script otherwise you could write a simple app for that ( java or something else ). The impact on app performance should be quite small, but you'll have to run some tests.

Btw, your app, is it a stand-alone, web app, ... ? Maybe there are other options to be investigated.

Edit: there's also an Append Redirection Operator >> that I've personally never used, but it might not lock the file.