non-zero exit code if checking failed when using --input

[stuebinm]

It would be nice if the validator returned a non-zero exit code when used on the cli with the --input option, either optionally behind a cli flag (e.g. --exit-code) or simply by default if --input is used. Perhaps it would even be possible to use different exit codes depending on severity of warnings or errors found (e.g. code 1 for fatal errors, 2 if there were no fatal errors but some warnings, etc.).

My motivation for this is that it'd make it much easier to use the tool in scripts or CI pipelines; currently the easiest way to answer "did validation fail?" i can find is to parse the json output with jq and check the validations (or issues_count) keys directly.

[antoine-de]

yep, seems a good use case 👍

Maybe we can add flags like a --fail-on-errors/--fail-on-warnings? wouldn't it be easier for your CI use case?

[stuebinm]

hm, that would of course also be an option — though it'd also make it harder to have a three-way split in e.g. a shell script (e.g. "do $x when everything's fine, $y if there were warnings, $z if there were errors).

but really i don't have a strong opinion either way, I can work with both :)

[ttezer]

I checked the current implementation in src/bin/main.rs and the discussion above.

When the validator is used with --input, validation issues are printed as JSON, but the process still exits with code 0. The reason is that main() always returns Ok(()) after printing the serialized response.

This makes it difficult to use the validator in CI pipelines or shell scripts.

To keep the current behavior backward-compatible, I suggest adding two opt-in flags, as proposed above:

• --fail-on-errors
• --fail-on-warnings

The concrete changes would be in src/bin/main.rs.

First, update the imports:

-use validator::{custom_rules, validate};
+use validator::{custom_rules, validate, Severity};

Then add the flags to Opt:

#[arg(
    long = "fail-on-errors",
    help = "Exit with a non-zero code when fatal or error issues are found",
    conflicts_with = "fail_on_warnings"
)]
fail_on_errors: bool,

#[arg(
    long = "fail-on-warnings",
    help = "Exit with a non-zero code when warnings, errors or fatal issues are found",
    conflicts_with = "fail_on_errors"
)]
fail_on_warnings: bool,

Add a helper function:

fn should_fail(
    response: &validate::Response,
    fail_on_errors: bool,
    fail_on_warnings: bool,
) -> bool {
    response
        .validations
        .values()
        .flatten()
        .any(|issue| {
            if fail_on_warnings {
                matches!(
                    issue.severity,
                    Severity::Warning | Severity::Error | Severity::Fatal
                )
            } else if fail_on_errors {
                matches!(issue.severity, Severity::Error | Severity::Fatal)
            } else {
                false
            }
        })
}

After printing the serialized validation result, set the exit code:

println!("{}", serialized);

if should_fail(validations, opt.fail_on_errors, opt.fail_on_warnings) {
    std::process::exit(1);
}

The expected behavior would be:

• without either flag: keep the current exit code behavior;
• --fail-on-errors: exit with code 1 for Error or Fatal issues;
• --fail-on-warnings: exit with code 1 for Warning, Error or Fatal issues;
• Information issues should not cause a failure;
• using both flags together should be rejected by clap;
• the JSON output should still be printed before the process exits.

Regression tests should cover a clean feed, a feed containing only warnings and a feed containing errors, and should verify both the output and the process exit code.