Algorithmic issue: determining “user sessions”

Question

I\'ve got a real little interesting (at least to me) problem to solve (and, no, it is not homework). It is equivalent to this: you need to determine \"sessions\" and \"sessions

Answer 1

You are asking for an online algorithm, i.e. one that can calculate a new set of sessions incrementally for each new input time.

Concerning the choice of data structure for the current set of sessions, you can use a balanced binary search tree. Each sessions is represented by a pair (start,end) of start time and end time. The nodes of the search tree are ordered by their start time. Since your sessions are separated by at least max_inactivity, i.e. no two sessions overlap, this will ensure that the end times are ordered as well. In other words, ordering by start times will already order the sessions consecutively.

Here some pseudo-code for insertion. For notational convenience, we pretend that sessions is an array, though it's actually a binary search tree.

insert(time,sessions) = do
    i <- find index such that
         sessions[i].start <= time && time < session[i+1].start

    if (sessions[i].start + max_inactivity >= time)
        merge  time  into  session[i]
    else if (time >= sessions[i+1].start - max_inactivity)
        merge  time  into  sessions[i+1]
    else
        insert  (time,time)  into  sessions

    if (session[i] and session[i+1] overlap)
        merge  session[i] and session[i+1]

The merge operation can be implemented by deleting and inserting elements into the binary search tree.

This algorithm will take time O(n log m) where m is the maximum number of sessions, which you said is rather small.

Granted, implementing a balanced binary search tree is no easy task, depending on the programming language. The key here is that you have to split the tree according to a key and not every ready-made library supports that operation. For Java, I would use the TreeSet<E> class; as said, the element type E is a single session given by start and end time. Its floor() and ceiling() methods will retrieve the sessions I've denoted with sessions[i] and sessions[i+1] in my pseudo-code.

Answer 2

Maximum Delay
If the log entries have a "maximum delay" (e.g. with a maximum delay of 2 hours, an 8:12 event will never be listed after a 10:12 event), you could look ahead and sort.

Do Sort
Alternatively, I'd first try sorting - at least to make sure it doesnt work. A timestamp can be reasonably stored in 8 bytes (4 even for your purposes, you could put 250 Millions of then into a gigabyte). Quicksort might not be the best choice here as it has low locality, insertion sort is almost-perfect for almost-sorted data (though it has bad locality, too), alternatively, quick-sorting chunk-wise, then merging chunks with a merge sort should do, even though it increases memory requirements.

Squash and conquer
Alternatively, you can use the following strategy:

1. transform each event into a "session of duration 0"
2. Split your list of sessions into chunks (e.g. 1K values / chunk)
3. Within each chunk, sort by session start
4. Merge all sessions than can be merged (having sorted before allows you to reduce your look ahead).
5. Compact the list of remaining sessions into a large single list
6. repeat with step 2 until the list doesn't get any shorter.
7. sort-and-merge over all

If your log files have the kind of "temporal locality" your question suggests, already a single pass should reduce the data to allow a "full" sort.

[edit] [This site]1 demonstrates an "optimized quicksort with insertion sort finish" that's quite good on almost-sorted data. As has this guys std::sort

Answer 3

I am not aware of a name for your problem or a name for the solution that you found. But your solution is (more or less) the solution I would propose. I think it's the best solution for that kind of problem.

If your data is at least somewhat ordered, you might find a slightly better solution by taking this ordering into account. E.g. your data could be ordered by date but not by time. Then, you would separate individual dates.

Answer 4

Your solution using an interval search tree sounds like it would be efficient enough.

You don't say whether the data you have provided (consisting solely of timestamps without date) is the actual data that you are processing. If so, consider that there are only 24 * 60 = 1440 minutes in a day. As this is a relatively small value, creating a bit-vector (packed or not---doesn't really matter) feels like it would provide both an efficient and easy solution.

The bit-vector (once filled) would be capable of either:

• answering the query "Has the user been sighted at time T?" in O(1), if you decide to set a field of the vector to true only when the corresponding time has shown up on your input data (we can call this method "conservative add") or

• answering the query "Was a session active at time T?" in O(1) as well, but with a larger constant, if you decide to set a field of the vector to true if a session was active at that time---by this I mean that when you add time T, you also set the following 29 fields to true.

I'd like to note that by using a conservative add, you are not limiting yourself to session-intervals of 30 minutes: indeed, you can change this value online at any time, since the structure does not extrapolate any information but is just a practical way of storing/viewing presence records.