Using Powershell to search for specific strings in the last x lines of very large log files

Question

I have a requirement that I have not found a solution or assistance for on the forums so far...

The issue: I monitor real time systems and these generate a number of

Answer 1

If you use PowerShell 3.0 or newer, you can use a combination of the get-content and select-string commandlets to get the same functionality. Since version 3.0, get-content support a -tail option that only returns the last n lines of a file in an efficient way. Using this, you can reimplement the Perl script above with the following (searching in the last 1000 lines):

# Returns the number of occurrences
(get-content logfile.txt -Tail 1000 | select-string -pattern "No packet received from EISEC Client").Length

Answer 2

To seek backwards, this contribution to the Poshcode repository appears to be a good match for your requirements:

# Name: Get-Tail.ps1
# Author: William Stacey
# Created: 02/22/2007
# Description: Gets the last N lines of a file. Does scan from end-of-file so works on large files. Also has a loop flag that prompts for refresh.

function Get-Tail([string]$path = $(throw "Path name must be specified."), [int]$count = 10, [bool]$loop = $false)
{
    if ( $count -lt 1 ) {$(throw "Count must be greater than 1.")}
    function get-last
    {
        $lineCount = 0
        $reader = new-object -typename System.IO.StreamReader -argumentlist $path, $true
        [long]$pos = $reader.BaseStream.Length - 1

        while($pos -gt 0)
        {
            $reader.BaseStream.Position = $pos
            if ($reader.BaseStream.ReadByte() -eq 10)
            {
                if($pos -eq $reader.BaseStream.Length - 1)
                {
                    $count++
                }
                $lineCount++
                if ($lineCount -ge $count) { break }
            }
            $pos--
        } 

        if ($lineCount -lt $count)
        {
            $reader.BaseStream.Position = 0
        }

        while($line = $reader.ReadLine())
        {
            $lines += ,$line
        }

        $reader.Close()
        $lines
    }

    while(1)
    {
        get-last
        if ( ! $loop ) { break }
        $in = read-host -prompt "Hit [Enter] to tail again or Ctrl-C to exit"
    }
}

Answer 3

PSCX has a cmdlet named Get-FileTail which efficiently gets the last lines of a file and thus avoiding reading the complete file beforehand.