问题
I want to use json.Encoder
to encode a large stream of data without loading all of it into memory at once.
// I want to marshal this
t := struct {
Foo string
// Bar is a stream of objects
// I don't want it all to be in memory at the same time.
Bar chan string
}{
Foo: "Hello World",
Bar: make(chan string),
}
// long stream of data
go func() {
for _, x := range []string{"one", "two", "three"} {
t.Bar <- x
}
close(t.Bar)
}()
I thought maybe the json package had this functionality build in, but that's not the case.
playground
// error: json: unsupported type: chan string
if err := json.NewEncoder(os.Stdout).Encode(&t); err != nil {
log.Fatal(err)
}
I'm currently just building the json string myself.
playground
w := os.Stdout
w.WriteString(`{ "Foo": "` + t.Foo + `", "Bar": [`)
for x := range t.Bar {
_ = json.NewEncoder(w).Encode(x)
w.WriteString(`,`)
}
w.WriteString(`]}`)
Is there a better way to do this?
If the json.Marshaler was like this is would be trivial.
type Marshaler interface {
MarshalJSON(io.Writer) error
}
回答1:
Unfortunately the encoding/json
package doesn't have a way to do this yet. What you're doing now (manually) is the best way to do it, without modifying the built-in package.
If you were to patch encoding/json
, you could modify the reflectValueQuoted
function in encoding/json/encode.go
You would want to focus on the Array case (Slice has a fallthrough
):
// Inside switch:
case reflect.Array:
e.WriteByte('[')
n := v.Len()
for i := 0; i < n; i++ {
if i > 0 {
e.WriteByte(',')
}
e.reflectValue(v.Index(i))
}
e.WriteByte(']')
I'm assuming you'd want to treat channel the same way. It would look something like this:
// Inside switch:
case reflect.Chan:
e.WriteByte('[')
i := 0
for {
x, ok := v.Recv()
if !ok {
break
}
if i > 0 {
e.WriteByte(',')
}
e.reflectValue(x)
i++
}
e.WriteByte(']')
I haven't done much with channels in reflect
, so the above may need other checks.
If you do end up going this route, you could always submit a patch.
回答2:
You can unpack the channel in the MarshalJSON
method in your struct like this:
type S struct {
Foo string
Bar chan string
}
func (s *S) MarshalJSON() (b []byte, err error) {
b, err := json.Marshal(s.Foo)
if err != nil { return nil, err }
for x := range s.Bar {
tmp, err := json.Marshal(x)
if err != nil { return nil, err }
b = append(b, tmp...)
}
return
}
来源:https://stackoverflow.com/questions/18133950/marshaljson-without-having-all-objects-in-memory-at-once