问题
I recently saw that the go yaml lib has new version (V3)
with the nodes capabilities (which in my opinion is a killer feature :) ) which can helps a lots with modifying yamls without changing the structure of the file
But since it is fairly new (from last week ) I didn't find some helpful docs and example for the context which I need (add new object/node and to keep the file structure the same without removing the comments etc)
what I need is to manipulate yaml file
for example
lets say I’ve this yaml file
version: 1
type: verbose
kind : bfr
# my list of applications
applications:
- name: app1
kind: nodejs
path: app1
exec:
platforms: k8s
builder: test
Now I got an json object (e.g. with app2) which I need to insert to the existing file
[
{
"comment: "Second app",
"name": "app2",
"kind": "golang",
"path": "app2",
"exec": {
"platforms": "dockerh",
"builder": "test"
}
}
]
and I need to add it to the yml file after the first application, (applications is array of application)
version: 1
type: verbose
kind : bfr
# my list of applications
applications:
# First app
- name: app1
kind: nodejs
path: app1
exec:
platforms: k8s
builder: test
# Second app
- name: app2
kind: golang
path: app2
exec:
platforms: dockerh
builder: test
is it possible to add from the yaml file the new json object ? also remove existing
I also found this blog https://blog.ubuntu.com/2019/04/05/api-v3-of-the-yaml-package-for-go-is-available
This is the types which represent the object
type VTS struct {
version string `yaml:"version"`
types string `yaml:"type"`
kind string `yaml:"kind,omitempty"`
apps Applications `yaml:"applications,omitempty"`
}
type Applications []struct {
Name string `yaml:"name,omitempty"`
Kind string `yaml:"kind,omitempty"`
Path string `yaml:"path,omitempty"`
Exec struct {
Platforms string `yaml:"platforms,omitempty"`
Builder string `yaml:"builder,omitempty"`
} `yaml:"exec,omitempty"`
}
update
after testing the solution which is provided by wiil7200 I found 2 issues
I use at the end write it to file
err = ioutil.WriteFile("output.yaml", b, 0644)
And the yaml output have 2 issue.
The array of the application is starting from the comments, it should start from the name
After the
nameentry thekindproperty and all others after are not aligned to thename
any idea how to solve those issue ? regard the comments issue, lets say I got it from other property
and not from the json (if it make it more simpler)
version: 1
type: verbose
kind: bfr
# my list of applications
applications:
- # First app
name: app1
kind: nodejs
path: app1
exec:
platforms: k8s
builder: test
- # test 1
name: app2
kind: golang
path: app2
exec:
platform: dockerh
builder: test
回答1:
First, let Me Start off by saying using yaml.Node does not produce valid yaml when marshalled from a valid yaml, given by the following example. Probably should file an issue.
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v3"
)
var (
sourceYaml = `version: 1
type: verbose
kind : bfr
# my list of applications
applications:
# First app
- name: app1
kind: nodejs
path: app1
exec:
platforms: k8s
builder: test
`
)
func main() {
t := yaml.Node{}
err := yaml.Unmarshal([]byte(sourceYaml), &t)
if err != nil {
log.Fatalf("error: %v", err)
}
b, err := yaml.Marshal(&t)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(b))
}
Produces the following invalid yaml in go version go1.12.3 windows/amd64
version: 1
type: verbose
kind: bfr
# my list of applications
applications:
- # First app
name: app1
kind: nodejs
path: app1
exec:
platforms: k8s
builder: test
Secondly, using a struct such as
type VTS struct {
Version string `yaml:"version" json:"version"`
Types string `yaml:"type" json:"type"`
Kind string `yaml:"kind,omitempty" json:"kind,omitempty"`
Apps yaml.Node `yaml:"applications,omitempty" json:"applications,omitempty"`
}
From ubuntu's blog and the source documentation it made it seem that it would correctly identify fields within the struct that are nodes and build that tree separately, but that is not the case. When unmarshalled, it will give a correct node tree, but when remarshalled it will produce the following yaml with all of the fields that yaml.Node exposes. Sadly we cannot go this route, must find another way.
version: "1"
type: verbose
kind: bfr
applications:
kind: 2
style: 0
tag: '!!seq'
value: ""
anchor: ""
alias: null
content:
- # First app
name: app1
kind: nodejs
path: app1
exec:
platforms: k8s
builder: test
headcomment: ""
linecomment: ""
footcomment: ""
line: 9
column: 3
Overlooking the first issue and the marshal bug for yaml.Nodes in a struct (on gopkg.in/yaml.v3 v3.0.0-20190409140830-cdc409dda467) we can now go about manipulating the Nodes that the package exposes. Unfortunately, there is no abstraction that will add Nodes with ease, so uses might vary and identifying nodes can be a pain. Reflection might help here a bit, so I leave that as an exercise for you.
You will find comment spew.Dumps that dump the entire node Tree in a nice format, this helped with debugging when adding Nodes to the source tree.
You can certainly remove nodes as well, you will just need to identify which particular nodes that need to be removed. You just have to ensure that you remove the parent nodes if it were a map or sequence.
package main
import (
"encoding/json"
"fmt"
"log"
"gopkg.in/yaml.v3"
)
var (
sourceYaml = `version: 1
type: verbose
kind : bfr
# my list of applications
applications:
# First app
- name: app1
kind: nodejs
path: app1
exec:
platforms: k8s
builder: test
`
modifyJsonSource = `
[
{
"comment": "Second app",
"name": "app2",
"kind": "golang",
"path": "app2",
"exec": {
"platforms": "dockerh",
"builder": "test"
}
}
]
`
)
// VTS Need to Make Fields Public otherwise unmarshalling will not fill in the unexported fields.
type VTS struct {
Version string `yaml:"version" json:"version"`
Types string `yaml:"type" json:"type"`
Kind string `yaml:"kind,omitempty" json:"kind,omitempty"`
Apps Applications `yaml:"applications,omitempty" json:"applications,omitempty"`
}
type Applications []struct {
Name string `yaml:"name,omitempty" json:"name,omitempty"`
Kind string `yaml:"kind,omitempty" json:"kind,omitempty"`
Path string `yaml:"path,omitempty" json:"path,omitempty"`
Exec struct {
Platforms string `yaml:"platforms,omitempty" json:"platforms,omitempty"`
Builder string `yaml:"builder,omitempty" json:"builder,omitempty"`
} `yaml:"exec,omitempty" json:"exec,omitempty"`
Comment string `yaml:"comment,omitempty" json:"comment,omitempty"`
}
func main() {
t := yaml.Node{}
err := yaml.Unmarshal([]byte(sourceYaml), &t)
if err != nil {
log.Fatalf("error: %v", err)
}
// Look for the Map Node with the seq array of items
applicationNode := iterateNode(&t, "applications")
// spew.Dump(iterateNode(&t, "applications"))
var addFromJson Applications
err = json.Unmarshal([]byte(modifyJsonSource), &addFromJson)
if err != nil {
log.Fatalf("error: %v", err)
}
// Delete the Original Applications the following options:
// applicationNode.Content = []*yaml.Node{}
// deleteAllContents(applicationNode)
deleteApplication(applicationNode, "name", "app1")
for _, app := range addFromJson {
// Build New Map Node for new sequences coming in from json
mapNode := &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map"}
// Build Name, Kind, and Path Nodes
mapNode.Content = append(mapNode.Content, buildStringNodes("name", app.Name, app.Comment)...)
mapNode.Content = append(mapNode.Content, buildStringNodes("kind", app.Kind, "")...)
mapNode.Content = append(mapNode.Content, buildStringNodes("path", app.Path, "")...)
// Build the Exec Nodes and the Platform and Builder Nodes within it
keyMapNode, keyMapValuesNode := buildMapNodes("exec")
keyMapValuesNode.Content = append(keyMapValuesNode.Content, buildStringNodes("platform", app.Exec.Platforms, "")...)
keyMapValuesNode.Content = append(keyMapValuesNode.Content, buildStringNodes("builder", app.Exec.Builder, "")...)
// Add to parent map Node
mapNode.Content = append(mapNode.Content, keyMapNode, keyMapValuesNode)
// Add to applications Node
applicationNode.Content = append(applicationNode.Content, mapNode)
}
// spew.Dump(t)
b, err := yaml.Marshal(&t)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(b))
}
// iterateNode will recursive look for the node following the identifier Node,
// as go-yaml has a node for the key and the value itself
// we want to manipulate the value Node
func iterateNode(node *yaml.Node, identifier string) *yaml.Node {
returnNode := false
for _, n := range node.Content {
if n.Value == identifier {
returnNode = true
continue
}
if returnNode {
return n
}
if len(n.Content) > 0 {
ac_node := iterateNode(n, identifier)
if ac_node != nil {
return ac_node
}
}
}
return nil
}
// deleteAllContents will remove all the contents of a node
// Mark sure to pass the correct node in otherwise bad things will happen
func deleteAllContents(node *yaml.Node) {
node.Content = []*yaml.Node{}
}
// deleteApplication expects that a sequence Node with all the applications are present
// if the key value are not found it will not log any errors, and return silently
// this is expecting a map like structure for the applications
func deleteApplication(node *yaml.Node, key, value string) {
state := -1
indexRemove := -1
for index, parentNode := range node.Content {
for _, childNode := range parentNode.Content {
if key == childNode.Value && state == -1 {
state += 1
continue // found expected move onto next
}
if value == childNode.Value && state == 0 {
state += 1
indexRemove = index
break // found the target exit out of the loop
} else if state == 0 {
state = -1
}
}
}
if state == 1 {
// Remove node from contents
// node.Content = append(node.Content[:indexRemove], node.Content[indexRemove+1:]...)
// Don't Do this you might have a potential memory leak source: https://github.com/golang/go/wiki/SliceTricks
// Since the underlying nodes are pointers
length := len(node.Content)
copy(node.Content[indexRemove:], node.Content[indexRemove+1:])
node.Content[length-1] = nil
node.Content = node.Content[:length-1]
}
}
// buildStringNodes builds Nodes for a single key: value instance
func buildStringNodes(key, value, comment string) []*yaml.Node {
keyNode := &yaml.Node{
Kind: yaml.ScalarNode,
Tag: "!!str",
Value: key,
HeadComment: comment,
}
valueNode := &yaml.Node{
Kind: yaml.ScalarNode,
Tag: "!!str",
Value: value,
}
return []*yaml.Node{keyNode, valueNode}
}
// buildMapNodes builds Nodes for a key: map instance
func buildMapNodes(key string) (*yaml.Node, *yaml.Node) {
n1, n2 := &yaml.Node{
Kind: yaml.ScalarNode,
Tag: "!!str",
Value: key,
}, &yaml.Node{Kind: yaml.MappingNode,
Tag: "!!map",
}
return n1, n2
}
Produces yaml
version: 1
type: verbose
kind: bfr
# my list of applications
applications:
- # First app
name: app1
kind: nodejs
path: app1
exec:
platforms: k8s
builder: test
- # Second app
name: app2
kind: golang
path: app2
exec:
platform: dockerh
builder: test
回答2:
You could create a new node and directly append to the contents, without deleting the previous node. The following example illustrates this point:
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v3"
)
var (
sourceYaml = `version: 1
type: verbose
kind : bfr
# my list of applications
applications:
# First app
- name: app1
kind: nodejs
path: app1
exec:
platforms: k8s
builder: test
`
)
type Application struct {
Name string `yaml:"name,omitempty" json:"name,omitempty"`
Kind string `yaml:"kind,omitempty" json:"kind,omitempty"`
Path string `yaml:"path,omitempty" json:"path,omitempty"`
Exec struct {
Platforms string `yaml:"platforms,omitempty" json:"platforms,omitempty"`
Builder string `yaml:"builder,omitempty" json:"builder,omitempty"`
} `yaml:"exec,omitempty" json:"exec,omitempty"`
}
func newApplicationNode(
name string,
kind string,
path string,
platforms string,
builder string,
comment string) (*yaml.Node, error) {
app := Application{
Name: name,
Kind: kind,
Path: path,
Exec: struct {
Platforms string `yaml:"platforms,omitempty" json:"platforms,omitempty"`
Builder string `yaml:"builder,omitempty" json:"builder,omitempty"`
}{platforms, builder},
}
marshalledApp, err := yaml.Marshal(&app)
if err != nil {
return nil, err
}
node := yaml.Node{}
if err := yaml.Unmarshal(marshalledApp, &node); err != nil {
return nil, err
}
node.Content[0].HeadComment = comment
return &node, nil
}
func main() {
yamlNode := yaml.Node{}
err := yaml.Unmarshal([]byte(sourceYaml), &yamlNode)
if err != nil {
log.Fatalf("error: %v", err)
}
newApp, err := newApplicationNode("app2", "golang", "app2", "dockerh",
"test", "Second app")
if err != nil {
log.Fatalf("error: %v", err)
}
appIdx := -1
for i, k := range yamlNode.Content[0].Content {
if k.Value == "applications" {
appIdx = i + 1
break
}
}
yamlNode.Content[0].Content[appIdx].Content = append(
yamlNode.Content[0].Content[appIdx].Content, newApp.Content[0])
out, err := yaml.Marshal(&yamlNode)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(out))
}
Clearly instead of going the hacky way as I did in newApplicationNode you can unmarshal properly from your JSON. However, as stated in previous answers, it is important to notice that the key and actual value are in subsequent indexes inside Content, therefore you need to take this into account when modifying the document. (e.g., look up applications key but then consider the next index (appIdx = i + 1 in my example) for its contents.
Hope that helps!
来源：https://stackoverflow.com/questions/55674853/modify-existing-yaml-file-and-add-new-data-and-comments