Asserts

You can assert what will be the behavior of executions:

Assert	Value	Description
assertStdout	Boolean	Is output produced?
assertStdoutEqual	String	Is the output equal to the String?
assertStdoutNotEqual	String	Is the output different than the String?
assertStdoutContains	String	Does the output contain the String?
assertStdoutNotContains	String	Does the output not contain the String?
assertStdoutSHA256	SHA256Checksum	Is the output equal to this SHA256 hash?
assertStdoutRegex	Regex	Does the output match your regular expression?
assertStdoutNotRegex	Regex	Does the output not match your regular expression?
assertStderr	Boolean	Are errors produced?
assertStderrEqual	String	Is the error equal to the String?
assertStderrNotEqual	String	Is the error different than the String?
assertStderrContains	String	Does the error contain the String?
assertStderrNotContains	String	Does the error not contain the String?
assertStderrSHA256	SHA256Checksum	Is the error equal to this SHA256 hash?
assertStderrRegex	Regex	Does the error match your regular expression?
assertStderrNotRegex	Regex	Does the error not match your regular expression?
assertReturnCode	Integer	Is the return code equal to a certain value?
assertReturnCodeNot	Integer	Is the return code not equal to a certain value?
assertDifferent	Boolean	Does the execution behave differently when using different inputs?
assertKilled	Boolean	Did the software time out?

---

General usage

Asserts are defined within tests:

HelloWorld:
  assertStdoutContains: "Hello World"
  echo:
  - echo Hello World

And they can have a severity associated, especially useful when working with ticketing systems:

HelloWorld:
  assertStdoutContains: "Hello World"
  setSeverity: 1
  echo:
  - echo Hello World

You can also define multiple strings that should much, such as the following case:

install:
  - apt update
  - apt install -qy curl

test:
  assertStdoutContains: 
  - HTTP/2 200
  - Satori CI
  - info@satori-ci.com
  - nonexistentstring
  satori:
    - curl -si https://satori.ci

---

assertStdout

Input	Description
Boolean	Asserts if an output has been produced

• Example Pass Test: the program should deliver output, and it does:

test:
    assertStdout: True
    run:
    - echo Hello World

• Example Fail Test: the program should deliver output, but no output is produced:

test:
    assertStdout: True
    run:
    - ./broken_executable

---

assertStdoutEqual

Input	Description
String	Asserts that the output is equal to the String

• Example Pass Test: the program should output "Hello World" and a newline, and it does:

test:
    assertStdoutEqual: "Hello World\n"
    run:
    - echo Hello World

• Example Fail Test: the program should output "Hello World" and a newline, but it doesn't:

test:
    assertStdoutEqual: "Hello World\n"
    run:
    - echo hello world

---

assertStdoutNotEqual

Input	Description
String	Is the output different than String?

• Example Pass Test: the program output should not be equal to "Hello World", and is not:

test:
    assertStdoutNotEqual: "Hello World\n"
    input:
    - - value: "Hello World"
        mutate: radamsa
        mutate_qty: 1
    run:
    - echo ${{input}}

Mutations output different inputs than the one originally provided. Read more about it on the Inputs section for Playbooks.

---

assertStdoutContains

Input	Description
String	Does the output contain the String?

• Example Pass Test: the program output should contain the string "Hello World", and it does:

test:
    assertStdoutContains: "Hello World"
    run:
    - echo Hello World 2023

---

assertStdoutNotContains

Input	Description
String	Does the output not contain the String?

• Example Pass Test: the program output should not contain the string "Error", and it does not:

test:
    assertStdoutNotContains: "Error"
    run:
    - echo Hello World

---

assertStdoutSHA256

Input	Description
SHA256Checksum	Is the output equal to this SHA256 hash?

• Example Pass Test: Google's root webpage hash has not changed:

If an output should be equal to a certain hash, to confirm that its original value has not changed, you can verify that with an assert. Consider for example how Google shows consistently the same SHA 256 hash:

$ curl -s https://google.com | shasum -a 256
5b61b0c2032b4aa9519d65cc98c6416c12415e02c7fbbaa1be5121dc75162edb  -

At the time of writing this, the hash 5b61b0c2032b4aa9519d65cc98c6416c12415e02c7fbbaa1be5121dc75162edb can be asserted programmatically:

You could assert that the output is consistent with that hash:

settings:
  image: curlimages/curl:7.83.1

test:
  assertStdoutSHA256: 5b61b0c2032b4aa9519d65cc98c6416c12415e02c7fbbaa1be5121dc75162edb
  google:
    - curl -s https://google.com

---

assertStdoutRegex

Input	Description
Regex	Does the output match your regular expression?

• Example Pass Test: the program output should contain the string "Hello " and additional characters, and it does:

test:
    assertStdoutRegex: "Hello .*"
    run:
    - echo Hello World

---

assertStdoutNotRegex

Input	Description
Regex	Does the output not match your regular expression?

• Example Unknown Test: the program output should not contain the string "Hello World" anywhere on the output, but the input could be mutated to "somethingHello World" and the result depends on the mutation:

test:
    assertStdoutNotRegex: ".*Hello World.*"
    input:
    - - value: "Hello World"
        mutate: radamsa
        mutate_qty: 1
    run:
    - echo Hello ${{input}}

---

assertStderr

Input	Description
Boolean	Are errors produced?

• Example Pass Test: the program output should not output errors, and it does not:

test:
    assertStderr: False
    run:
    - echo Hello World

---

assertStderrEqual

Input	Description
String*	Is the error equal to the String?

• Example Pass Test: the programs should verify that the error is a certain string:

settings:
  image: python

test:
  assertStderrEqual: Verify this error
  error:
  - python3 -c "import sys; sys.stderr.write('Verify this error')"

---

assertStderrNotEqual

Input	Description
String	Is the error different than the String?

• Example Fail Test: the programs should verify that the error is not a certain string:

settings:
  image: python

test:
  assertStderrNotEqual: Verify this error
  error:
  - python3 -c "import sys; sys.stderr.write('Verify this error')"

---

assertStderrContains

Input	Description
String	Does the error contains the String?

• Example Pass Test: the program's errors should contain the string Traceback, and it does:

settings:
  image: python

install:
   - echo import nonexistent > test.py
test:
    assertStderrContains: "Traceback"
    run:
    - python3 test.py

---

assertStderrNotContains

Input	Description
String	Does the error not contain the String?

• Example Fail Test: the program's errors should not contain the string Traceback, but it does:

settings:
  image: python

install:
   - echo import nonexistent > test.py
test:
    assertStderrNotContains: "Traceback"
    run:
    - python3 test.py

---

assertStderrSHA256

Input	Description
SHA256Checksum	Is the error equal to this SHA256 hash?

• Example Fail Test: the program's errors should not contain the string Traceback, but it does:

settings:
  image: python

install:
   - echo import nonexistent > test.py
test:
    assertStderrSHA256: "69827a4c85154b891cae9c35d99887375d815ec676bb7ce86e1f7601f6fec3ad"
    run:
    - python3 test.py

---

assertStderrRegex

Input	Description
Regex	Does the error match your regular expression?

• Example Pass Test: the Python script referencing a non-existent object will throw a NameError:

settings:
  image: python

test:
  assertStderrRegex: "(?i)error|warning|traceback|exception"
  python:
  - python3 -c "non-existent"

---

assertStderrNotRegex

Input	Description
Regex	Does the error not match your regular expression?

• Example Pass Test: the program's errors should not throw a Traceback, and it doesn't:

settings:
  image: python

install:
   - echo import os > test.py
test:
    assertStderrNotRegex: ".*Traceback.*"
    run:
    - python3 test.py

---

assertReturnCode

Input	Description
Integer	Is the return code equal to the Integer?

• Example Pass Test: the program should return the code 0, and it does:

test:
    assertReturnCode: 0
    run:
    - echo This program is executed correctly

---

assertReturnCodeNot

Input	Description
Integer	Is the return code not equal to the Integer?

• Example Positive Pass Test: the program should not return the code 0, and it doesn't:

test:
    assertReturnCodeNot: 0
    run:
    - return 2

• Example Negative Fail Test: the program should return a code because it can be executed:

test:
  assertReturnCodeNot: null
  run:
  - non_existing_software

TBC

---

assertDifferent

Input	Description
Boolean	Does the execution behave differently when using different inputs?

• Example Fail Test: the production and staging environment should look the same, and it does not:

API:
- - "www.example.com"
  - "staging.example.com"

test:
    assertDifferent: False
    run:
    - curl $API

---

assertKilled

Input	Description
Boolean	Did the software time out?

• Example Fail Test: the software should finish execution within 10 seconds, and it does not:

settings:
  timeout: 10

test:
  assertKilled: False
  run:
    - sleep 20

TBC

---

If you need any help, please reach out to us on Discord or via Email