Upload new line JSON to Elasticsearch bulk API

Question

I\'m trying to upload a new line JSON to Elasticsearch using the Bulk API. The bulk JSON I\'m uploading looks like this, with each JSON on a new line:

{\"ip\": \         


        

Answer 1

curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/index-name/doc-type/_bulk?pretty' --data-binary @my_newline_json.json

Change your bulk JSON, to the following format. Your my_newline_json.json should look like this:

{"index":{}}
{"ip": "x.x.x.x", "seen": true, "classification": "malicious", "spoofable": false, "first_seen": "2020-03-31", "last_seen": "2020-04-15", "actor": "unknown", "tags": ["ADB Worm", "HTTP Alt Scanner", "Mirai", "Web Scanner"], "cve": [], "metadata": {"country": "United Kingdom", "country_code": "GB", "city": "redacted", "organization": "redacted", "rdns": "", "asn": "ASxxx", "tor": false, "os": "Linux 2.2-3.x", "category": "isp"}, "raw_data": {"scan": [{"port": 80, "protocol": "TCP"}, {"port": 81, "protocol": "TCP"}, {"port": 88, "protocol": "TCP"}, {"port": 5555, "protocol": "TCP"}, {"port": 8080, "protocol": "TCP"}], "web": {}, "ja3": []}}
{"index":{}}
{"ip": "x.x.x.x", "seen": true, "classification": "malicious", "spoofable": true, "first_seen": "2020-04-09", "last_seen": "2020-04-11", "actor": "unknown", "tags": ["Eternalblue", "SMB Scanner"], "cve": ["CVE-2017-0144"], "metadata": {"country": "United Kingdom", "country_code": "GB", "city": "redacted", "organization": "redacted", "rdns": "host.somehost.com", "asn": "ASxxx", "tor": false, "os": "Windows 7/8", "category": "isp"}, "raw_data": {"scan": [{"port": 445, "protocol": "TCP"}], "web": {}, "ja3": []}}
{"index":{}}
{"ip": "x.x.x.x", "seen": true, "classification": "malicious", "spoofable": true, "first_seen": "2019-09-05", "last_seen": "2020-04-06", "actor": "unknown", "tags": ["Mirai"], "cve": [], "metadata": {"country": "United Kingdom", "country_code": "GB", "city": "redacted", "organization": "redacted", "rdns": "redacted", "asn": "ASxxx", "tor": false, "os": "Linux 2.2.x-3.x (Embedded)", "category": "isp"}, "raw_data": {"scan": [{"port": 23, "protocol": "TCP"}, {"port": 2323, "protocol": "TCP"}], "web": {}, "ja3": []}}

Dont forget to add a new line at the end of your content.

Format of bulk JSON:

Output Result: