FIWARE Cygnus -> cartodb sinks.NGSISink: Persistence error, 400 Bad request

Question

I\'m trying to connect cygnus (1.4.0_SNAPSHOT) to cartodb. I run it locally, and I use a script to send a notification to cygnus. The script runs Ok, but cygnus says:

Answer 1

The problem is geo:json type is not currently supported by NGSICartoDBSink. This sink understands ceratain ways of notifying geolocated attributes, according to Orion Context Broker specification; these ones:

• Using the geo:point type, and sending the coordinates in the value field with format "latitude, longitude".
• Using the location metadata, of type string and value WGS84, and sending the coordinates in the value field with format "latitude, longitude".

Please observe:

• The above options are exclussive, i.e. cannot be used at the same time.
• The location metadata is deprecated in Orion, nevertheless it can be still used.

While geo:json is supported (I'll start working on that, it could be ready during this sprint/month), I'll recommend you to use the geo:point type.

EDIT 1

I'm adding here an example of Cygnus execution, when receiving a notification involving a geolocated attribute (geo:point type).

Cygnus version:

1.6.0

Cygnus configuration:

cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = raw-sink
cygnus-ngsi.channels = raw-channel

cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = raw-channel
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp

cygnus-ngsi.sinks.raw-sink.type = com.telefonica.iot.cygnus.sinks.NGSICartoDBSink
cygnus-ngsi.sinks.raw-sink.channel = raw-channel
cygnus-ngsi.sinks.raw-sink.enable_grouping = false
cygnus-ngsi.sinks.raw-sink.keys_conf_file = /usr/cygnus/conf/cartodb_keys.conf
cygnus-ngsi.sinks.raw-sink.swap_coordinates = false
cygnus-ngsi.sinks.raw-sink.enable_raw = true
cygnus-ngsi.sinks.raw-sink.enable_distance = false
cygnus-ngsi.sinks.raw-sink.enable_raw_snapshot = false
cygnus-ngsi.sinks.raw-sink.data_model = dm-by-entity
cygnus-ngsi.sinks.raw-sink.batch_size = 50
cygnus-ngsi.sinks.raw-sink.batch_timeout = 10
cygnus-ngsi.sinks.raw-sink.batch_ttl = 0
cygnus-ngsi.sinks.raw-sink.batch_retries = 5000

cygnus-ngsi.channels.raw-channel.type = com.telefonica.iot.cygnus.channels.CygnusMemoryChannel
cygnus-ngsi.channels.raw-channel.capacity = 1000
cygnus-ngsi.channels.raw-channel.transactionCapacity = 100

Create the table:

$ curl -X GET -G "https://.cartodb.com/api/v2/sql?api_key=" --data-urlencode "q=CREATE TABLE x002ftestxffffx0043ar1xffffx0043ar (recvTime text, fiwareServicePath text, entityId text, entityType text, speed float, speed_md text, the_geom geometry(POINT,4326))"
{"rows":[],"time":0.005,"fields":{},"total_rows":0}

Script simulating a notification:

$ cat notification.sh
#!/bin/sh

URL=$1

if [ "$2" != "" ]
then
   SERVICE=$2
else
   SERVICE=default
fi

if [ "$3" != "" ]
then
   SERVICE_PATH=$3
else
   SERVICE_PATH=/
fi

curl $URL -v -s -S --header 'Content-Type: application/json; charset=utf-8' --header 'Accept: application/json' --header 'User-Agent: orion/0.10.0' --header "Fiware-Service: $SERVICE" --header "Fiware-ServicePath: $SERVICE_PATH" -d @- 
<

Script execution:

$ ./notification.sh http://localhost:5050/notify  /test 40.40 -3.4 120
*   Trying ::1...
* Connected to localhost (::1) port 5050 (#0)
> POST /notify HTTP/1.1
> Host: localhost:5050
> Content-Type: application/json; charset=utf-8
> Accept: application/json
> User-Agent: orion/0.10.0
> Fiware-Service: 
> Fiware-ServicePath: /test
> Content-Length: 569
> 
* upload completely sent off: 569 out of 569 bytes
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Server: Jetty(6.1.26)
< 
* Connection #0 to host localhost left intact

Cygnus logs upon notification reception:

time=2016-12-02T13:48:27.310UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv= | subsrv=/test | comp=cygnus-ngsi | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[282] : [NGSIRestHandler] Starting internal transaction (eb73b8d5-af9b-48ea-8ce7-ff21edc957f3)
time=2016-12-02T13:48:27.312UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv= | subsrv=/test | comp=cygnus-ngsi | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[299] : [NGSIRestHandler] Received data ({  "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",  "originator" : "localhost",  "contextResponses" : [    {      "contextElement" : {        "attributes" : [          {            "name" : "speed",            "type" : "float",            "value" : "120"          },          {            "name" : "the_geom",            "type" : "geo:point",            "value" : "40.40, -3.4"          }        ],        "type" : "Car",        "isPattern" : "false",        "id" : "Car1"      },      "statusCode" : {        "code" : "200",        "reasonPhrase" : "OK"      }    }  ]})
time=2016-12-02T13:48:36.404UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv= | subsrv=/test | comp=cygnus-ngsi | op=persistRawAggregation | msg=com.telefonica.iot.cygnus.sinks.NGSICartoDBSink[553] : [raw-sink] Persisting data at NGSICartoDBSink. Schema (), Table (x002ftestxffffx0043ar1xffffx0043ar), Data (('2016-12-02T13:48:27.381Z','/test','Car1','Car',ST_SetSRID(ST_MakePoint(40.40,-3.4), 4326),'120','[]'))
time=2016-12-02T13:48:38.237UTC | lvl=INFO | corr=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | trans=eb73b8d5-af9b-48ea-8ce7-ff21edc957f3 | srv= | subsrv=/test | comp=cygnus-ngsi | op=processNewBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[514] : Finishing internal transaction (eb73b8d5-af9b-48ea-8ce7-ff21edc957f3)

Getting the data:

$ curl -X GET -G "https://.cartodb.com/api/v2/sql?api_key=" --data-urlencode "q=select * from x002ftestxffffx0043ar1xffffx0043ar"
{"rows":[{"recvtime":"2016-12-02T13:48:27.381Z","fiwareservicepath":"/test","entityid":"Car1","entitytype":"Car","speed":120,"speed_md":"[]","the_geom":"0101000020E610000033333333333344403333333333330BC0"}],"time":0.001,"fields":{"recvtime":{"type":"string"},"fiwareservicepath":{"type":"string"},"entityid":{"type":"string"},"entitytype":{"type":"string"},"speed":{"type":"number"},"speed_md":{"type":"string"},"the_geom":{"type":"geometry"}},"total_rows":1}

EDIT 2

This answer is only valid if you own an "enterprise" Carto account. Please, see my other answer to this question.