Azure Stream Analytics - Error with customized “timestamp by” while applying window tumbling

Question

I have a json file as below:

{"imei": {"imei": "358174069248418F", "imeiBinary": "NYF0BpJIQY8=","imeiNotEncoded": "358174069248418","valid": 1},"dataPackets": [["msy.mxp.datapacket.AlarmNotification",{"version": 1, "id": 21, "op": 2,"sizeDynamic": 0, "alarmStatus": 4}],["msy.mxp.datapacket.IOStatus",{"version": 1,"id": 15, "op": 2,"sizeDynamic": 0,"ioStatus": 135,"ioDirections": 120}], ["msy.mxp.datapacket.LogicalStatus",{"version": 1,"id": 16, "op": 2,"sizeDynamic": 0,"logicalStatus": 5} ],[ "msy.mxp.datapacket.Position", {"version": 1,"id": 19,"op": 2,"latitude": 40.835243,"longitude": 14.246057,"altitude": 40,"speed": 0, "course": 68, "gpsNumSatellite": 5,"glonassNumSatellite": 1,"fixValid": 1,"timeValid": 1,"wgs84degMinFormat": 1, "glonass": 1,"fixMode": 3,"timestamp": {"timeSecFrom1Gen2000": 925560202,"time": 1490648755000 }, "sizeDynamic": 0} ] ]}

and I am reading by the following query:

WITH Datapackets AS
(
SELECT imei.imei as imei,
        persistent as persistent,
        [timestamp].[time] as input_time,
        compressed as compressed,
        GetArrayElement(dataPackets, 3) as position
FROM h24
), one as(
SELECT *,
GetRecordPropertyValue (GetArrayElement(position,1), 'timestamp') as position_timestamp --1st
from Datapackets 
), two as (
select
    imei,
    GetRecordPropertyValue (GetArrayElement(position,1), 'op') as position_OP,
    [position_timestamp].[time] as position_time,
    dateadd(S, [position_timestamp].[timeSecFrom1Gen2000], '1970-01-01') as timing,
GetRecordPropertyValue (GetArrayElement(position,1), 'latitude') as position_latitude,
GetRecordPropertyValue (GetArrayElement(position,1), 'longitude') as position_longitude,
GetRecordPropertyValue (GetArrayElement(position,1), 'altitude') as position_altitude,
GetRecordPropertyValue (GetArrayElement(position,1), 'speed') as position_speed
    from one) SELECT * from two

Now I want to make window tumbling group by 30 seconds as following but I have a problem which tells me that timestamp property is not allowed for the input file"two" ,, here the query that I use

    WITH Datapackets AS
    (
    SELECT imei.imei as imei,
            persistent as persistent,
            [timestamp].[time] as input_time,
            compressed as compressed,
            GetArrayElement(dataPackets, 3) as position
    FROM h24
    ), one as(
    SELECT *,
    GetRecordPropertyValue (GetArrayElement(position,1), 'timestamp') as position_timestamp --1st
    from Datapackets 
    ), two as (
    select
        imei,
        GetRecordPropertyValue (GetArrayElement(position,1), 'op') as position_OP,
        [position_timestamp].[time] as position_time,
        dateadd(S, [position_timestamp].[timeSecFrom1Gen2000], '1970-01-01') as timing,
GetRecordPropertyValue (GetArrayElement(position,1), 'latitude') as position_latitude,
GetRecordPropertyValue (GetArrayElement(position,1), 'longitude') as position_longitude,
GetRecordPropertyValue (GetArrayElement(position,1), 'altitude') as position_altitude,
GetRecordPropertyValue (GetArrayElement(position,1), 'speed') as position_speed
        from one) SELECT imei, System.TimeStamp AS 'start', Avg(position_speed), max(position_latitude)  
FROM two TIMESTAMP BY TIMING GROUP BY imei, TumblingWindow(duration(second, 30))

The error appears in the last two lines (FROM two TIMESTAMP BY TIMING),

************** Update,, After investigating more, I found that I can only use the option timestamp by in the input, and I have to use it only if I will make customized timestamping for the events. Normally they are timestamped by the arrival time as default (https://msdn.microsoft.com/en-us/library/mt573293.aspx)

Now my problem is how to timestamp my events with a time field that is recorded in a 3rd level array in the Json file to be able to do my aggregation.

Any suggestions about how I deal with this problem, Thanks

Answer 1

According to https://msdn.microsoft.com/en-us/library/mt598501.aspx TIMESTAMP BY can only be used on input sources, so you would want to put that as part of your very first step:

WITH Datapackets AS
...
FROM h24 TIMESTAMP BY (expression)
...

Furthermore, from the same source, referring to System.TimeStamp when used with a GROUP BY over a window:

The timestamp of the result of the aggregate is the end of the time window to which this result corresponds.

So when you write System.TimeStamp in the final SELECT statement, it refers to the end of the current window.