11

I want to merge two yaml documents with the result containing

  • all mapped values (with the last one taking precedence)
  • concatenated arrays

e.g. given this file

# file1.yml
animals:
  - elephant
    donkey
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
strange: true

and this file

#file2.yml
animals:
  - monkey
    pelican
flavours:
  sweet:
    - vanilla
strange: false

the result should contain all the nodes, with merged arrays and values that are not arrays from the last file

#result.yml
animals:
  - elephant
    donkey
    monkey
    pelican
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
      vanilla
strange: false

Can yq do this, maybe?

Joseph Tura
  • 6,290
  • 8
  • 47
  • 73
  • My experience with yq is that it is quite quick to abort with strange errors once you do something complex. [This answer](https://stackoverflow.com/a/65784135/347964) has usable code that solves the problem in Go if that helps you. – flyx Mar 18 '21 at 16:52
  • Which version of yq are you using? The Go or the Python version? – Inian Mar 18 '21 at 19:38
  • I tried using the latest Go version. I have now got it running using v3.4.1, which may be less powerful, but also a lot less complex. Any pointers as to how to get it to work with 4.x are appreciated. – Joseph Tura Mar 19 '21 at 06:53

1 Answers1

24

Merging YAML files with yq, concatenating arrays.

Assumption

In both file1.yml and file2.yml, the animals array contains a single index with a multiline string.

elephant
donkey

for file1.yml, and

monkey
pelican

for file2.yml.

Since you asked about concatenated array, I am assuming file1.yml, file2.yml and results.yml should be like this:

# file1.yml
animals:
  - elephant
  - donkey
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
strange: true

# file2.yml
animals:
  - monkey
  - pelican
flavours:
  sweet:
    - vanilla
strange: false

# result.yml
animals:
  - elephant
  - donkey
  - monkey
  - pelican
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
    - vanilla
strange: false

Example

With yq 4.x, you can use the ireduce operator to merge two or more files:

https://mikefarah.gitbook.io/yq/operators/reduce#merge-all-yaml-files-together

$ yq4 eval-all '. as $item ireduce ({}; . * $item)' file1.yml file2.yml
animals:
  - monkey
  - pelican
flavours:
  sour:
    - lemon
  sweet:
    - vanilla
strange: false

Both files have been merged, but duplicated keys and arrays have been overwritten by the latest file.

To appends arrays instead of overriding them, simply add a + after the merge operator (*).

https://mikefarah.gitbook.io/yq/operators/multiply-merge#merge-appending-arrays

$ yq4 eval-all '. as $item ireduce ({}; . *+ $item)' file1.yml file2.yml
animals:
  - elephant
  - donkey
  - monkey
  - pelican
flavours:
  sour:
    - lemon
  sweet:
    - chocolate
    - vanilla
strange: false
jpmorin
  • 583
  • 3
  • 11
  • 1
    I am having the similar requirement, how to merge and save into another file ? is it possible to write this content into another file ? – Dnyaneshwar Jadhav Sep 22 '21 at 09:35
  • 3
    @DnyaneshwarJadhav. With yq ou can modify a file `--inplace`. When multiple files are used, only the first one will be modified. Note that you can always redirect the output to another file: `yq4 eval-all '. as $item ireduce ({}; . *+ $item)' file1.yml file2.yml > anotherFile.yml` – jpmorin Sep 23 '21 at 17:16
  • 1
    In terms of duplicate variable entries (like strange: false vs strange: true) the right most file wins. `yq4 eval-all '. as $item ireduce ({}; . *+ $item)' file1.yml file2.yml > anotherFile.yml` --> anotherFile.yml having strange: false, but `yq4 eval-all '. as $item ireduce ({}; . *+ $item)' file2.yml file1.yml > anotherFile.yml` --> anotherFile.yml would have strange: true – neoakris Dec 06 '21 at 22:35
  • 1
    Thank you, @jpmorin! `yq eval-all '. as $item ireduce ({}; . *+ $item)'` merges nicely ~/.kube K8s configs. – Evgeny Goldin Apr 07 '22 at 16:53