Converting Python win32evtlog objects to xml

Question

I have a app that uses win32evtlog to get and display different events and I would like to limit the display to events of a specific level but win32evtlog doesn\'t return this.

Answer 1

ReadEventLog returns PyEventLogRecords (wrapper over [MS.Docs]: _EVENTLOGRECORD structure), while EvtRender expects (you need to work with) PyHANDLEs (PyEVT_HANDLEs (wrapper over EVT_HANDLE ([MS.Docs]: Windows Event Log Data Types) to be more precise)).
So, for getting XML data, you need to use the functions family that works with this type: e.g. EvtQuery, EvtNext.

code.py:

#!/usr/bin/env python3

import sys
import pywintypes
import win32evtlog

INFINITE = 0xFFFFFFFF
EVTLOG_READ_BUF_LEN_MAX = 0x7FFFF


def get_record_data(eventlog_record):
    ret = dict()
    for key in dir(eventlog_record):
        if 'A' < key[0] < 'Z':
            ret[key] = getattr(eventlog_record, key)
    return ret


def get_eventlogs(source_name="Application", buf_size=EVTLOG_READ_BUF_LEN_MAX, backwards=True):
    ret = list()
    evt_log = win32evtlog.OpenEventLog(None, source_name)
    read_flags = win32evtlog.EVENTLOG_SEQUENTIAL_READ
    if backwards:
        read_flags |= win32evtlog.EVENTLOG_BACKWARDS_READ
    else:
        read_flags |= win32evtlog.EVENTLOG_FORWARDS_READ
    offset = 0
    eventlog_records = win32evtlog.ReadEventLog(evt_log, read_flags, offset, buf_size)
    while eventlog_records:
        ret.extend(eventlog_records)
        offset += len(eventlog_records)
        eventlog_records = win32evtlog.ReadEventLog(evt_log, read_flags, offset, buf_size)
    win32evtlog.CloseEventLog(evt_log)
    return ret


def get_events_xmls(channel_name="Application", events_batch_num=100, backwards=True):
    ret = list()
    flags = win32evtlog.EvtQueryChannelPath
    if backwards:
        flags |= win32evtlog.EvtQueryReverseDirection
    try:
        query_results = win32evtlog.EvtQuery(channel_name, flags, None, None)
    except pywintypes.error as e:
        print(e)
        return ret
    events = win32evtlog.EvtNext(query_results, events_batch_num, INFINITE, 0)
    while events:
        for event in events:
            ret.append(win32evtlog.EvtRender(event, win32evtlog.EvtRenderEventXml))
        events = win32evtlog.EvtNext(query_results, events_batch_num, INFINITE, 0)
    return ret


def main():
    import sys, os
    from collections import OrderedDict
    standard_log_names = ["Application", "System", "Security"]
    source_channel_dict = OrderedDict()

    for item in standard_log_names:
        source_channel_dict[item] = item

    for item in ["Windows Powershell"]: # !!! This works on my machine (96 events)
        source_channel_dict[item] = item

    for source, channel in source_channel_dict.items():
        print(source, channel)
        logs = get_eventlogs(source_name=source)
        xmls = get_events_xmls(channel_name=channel)
        #print("\n", get_record_data(logs[0]))
        #print(xmls[0])
        #print("\n", get_record_data(logs[-1]))
        #print(xmls[-1])
        print(len(logs))
        print(len(xmls))

if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main()

Notes:

• The 2 lists should have the same length. The n^th entry in each of them should reference the same event (as long as both functions are called with same value for backwards argument (read below))
• get_events_xmls:
  • Returns a list of XML blobs associated to the events
  • The error handling is not the best, you could wrap all API calls in try / except clauses (I didn't run into errors, so I'm not sure what are the situations where exception could be raised)
  • You can play a little bit with [MS.Docs]: EvtNext function's arguments (Timeout and EventsSize for performance fine tuning; for me, ~20k events were processed in a matter of <10 seconds - out of which text printing and conversions took the most)
  • In Python 3, the XMLs are bytes ([Python 3.Docs]: Built-in Types - class bytes([source[, encoding[, errors]]])) rather than normal strings (I had to encode them because some contain some non-ASCII chars, and attempting to print them would raise UnicodeEncodeError)
  • Event filtering is possible, check [MS.Docs]: EvtQuery function's args (Flags and Query)
  • Note the backwards argument which allows traversing the events in reversed (chronological) order (default set to True).
• get_record_data:
  • It's just a convenience function, it converts a PyEventLogRecord object into a Python dictionary
  • The conversion is based on the fact that fields that we care about start with a capital letter (EventID, ComputerName, TimeGenerated, ...), that's why it shouldn't be used in production
  • It doesn't convert the actual values (TimeGenerated's value is pywintypes.datetime(2017, 3, 11, 3, 46, 47))
• get_eventlogs:
  • Returns a list of PyEventLogRecords
  • As in get_events_xmls's case note the backwards argument
  • I must insist on buf_size. As [MS.Docs]: ReadEventLogW function states, when getting the events, a buffer of max 512K can be used. Now (starting with PyWin32 version 220), it's possible to pass it as an argument (the last one) to win32evtlog.ReadEventLog. Check [SourceForge.hg]: mhammond/pywin32 - Add buffer size parameter for ReadEventLog (patch #143 from cristi fati) for more details. By default, there was a limitation so that the buffer size was hardcoded to 1K. Since every ReadEventLog was accessing the disk, with the new buffer size I got a 10X speed improvement (for ~180K events)
• Since I'm storing all the data in the 2 lists (instead of inplace data processing), I am choosing speed over memory consumption. For ~20K events, the 2 lists are taking ~30MB of RAM (which nowadays I think it's decent enough)

@EDIT0: I couldn't find a way to get all the required info with the Evt* functions family, so I'm getting it from both sources (I enhanced the script that I've previously posted):

@EDIT1: According to [MS.Docs]: OpenEventLogW function:

If you specify a custom log and it cannot be found, the event logging service opens the Application log; however, there will be no associated message or category string file.

[MS.Docs]: Eventlog Key lists the 3 standard ones. So, that's why it opens the Application log. I've done some small changes to the script to test the sources. I don't know where mmc gets the Setup events from.