detect duplicate in JSON String Golang

Question

I have JSON string like

\"{\\\"a\\\": \\\"b\\\", \\\"a\\\":true,\\\"c\\\":[\\\"field_3 string 1\\\",\\\"field3 string2\\\"]}\"

how to dete

Answer 1

Use the json.Decoder to walk through the JSON. When an object is found, walk through keys and values checking for duplicate keys.

func check(d *json.Decoder, path []string) error {
    // Get next token from JSON
    t, err := d.Token()
    if err != nil {
        return err
    }

    delim, ok := t.(json.Delim)

    // There's nothing to do for simple values (strings, numbers, bool, nil)
    if !ok {
        return nil
    }

    switch delim {
    case '{':
        keys := make(map[string]bool)
        for d.More() {
            // Get field key
            t, err := d.Token()
            if err != nil {
                return err
            }
            key := t.(string)

            // Check for duplicates
            if keys[key] {
                fmt.Printf("Duplicate %s\n", strings.Join(append(path, key), "/"))
            }
            keys[key] = true

            // Check value
            if err := check(d, append(path, key)); err != nil {
                return err
            }
        }
        // Consume trailing }
        if _, err := d.Token(); err != nil {
            return err
        }

    case '[':
        i := 0
        for d.More() {
            if err := check(d, append(path, strconv.Itoa(i))); err != nil {
                return err
            }
            i++
        }
        // Consume trailing ]
        if _, err := d.Token(); err != nil {
            return err
        }

    }
    return nil
}

Here's how to call it:

data := `{"a": "b", "a":true,"c":["field_3 string 1","field3 string2"], "d": {"e": 1, "e": 2}}`
if err := check(json.NewDecoder(strings.NewReader(data)), nil); err != nil {
    log.Fatal(err)
}

The output is:

Duplicate a
Duplicate d/e

Run it on the Playground

Answer 2

One that would probably work well would be to simply decode, reencode, then check the length of the new json against the old json:

https://play.golang.org/p/50P-x1fxCzp

package main

import (
  "encoding/json"
  "fmt"
)

func main() {
  jsn := []byte("{\"a\": \"b\", \"a\":true,\"c\":[\"field_3 string 1\",\"field3 string2\"]}")
  var m map[string]interface{}
  err := json.Unmarshal(jsn, &m)
  if err != nil {
      panic(err)
  }
        l := len(jsn)
        jsn, err = json.Marshal(m)
  if err != nil {
      panic(err)
  }
  if l != len(jsn) {
      panic(fmt.Sprintf("%s: %d (%d)", "duplicate key", l, len(jsn)))
  }
}

The right way to do it would be to re-implement the json.Decode function, and store a map of keys found, but the above should work (especially if you first stripped any spaces from the json using jsn = bytes.Replace(jsn, []byte(" "), []byte(""), -1) to guard against false positives.