How do I debug a failing test?

You can use debug flags like --skip-delete and --pause-on-failure to keep resources alive and inspect their state after a test fails.

What is Chainsaw testing in the context of Ark?

Chainsaw is an E2E testing tool used within the Ark framework to validate the behavior and orchestration of agentic resources like models and agents.

What cloud providers are supported for real LLM tests?

The Ark framework is provider-agnostic, though it includes built-in support for Azure OpenAI for E2E validation.

How is an Ark test directory structured?

A standard test includes a chainsaw-test.yaml definition, a manifests folder for resource YAMLs, and a README for documentation.

Can I run tests without spending LLM credits?

Yes, this skill supports mock-llm configurations, allowing you to run comprehensive test suites locally without calling external AI providers.

Ark Chainsaw Testing

Name: Ark Chainsaw Testing
Author: mckinsey

bymckinsey

•

305

セキュリティとテスト

Executes and authors end-to-end tests for agentic resources using the Chainsaw framework and mock LLM configurations.

概要

The Ark Chainsaw Testing skill enables developers to validate complex agentic application patterns by running automated end-to-end tests within the McKinsey Ark ecosystem. It facilitates the creation of robust test suites for models, agents, and queries, supporting both mock LLM environments for cost-effective local development and real LLM providers like Azure OpenAI for production-grade validation. With specialized debugging modes that allow resource persistence on failure, this skill streamlines the process of identifying and fixing orchestration issues in AI-driven workflows.

主な機能

Mock LLM support for fast and cost-effective local test execution.
Seamless integration with Azure OpenAI for real-world provider validation.
End-to-end testing for agentic resources including models, agents, and queries.
Advanced debug mode with resource persistence and failure pausing.
Standardized test structure using YAML manifests and Chainsaw definitions.
305 GitHub stars

ユースケース

Validating agentic workflow orchestration before production deployment.
Automating regression testing for complex AI-driven application logic.
Debugging failed resource deployments in isolated, reproducible environments.

概要

主な機能

Mock LLM support for fast and cost-effective local test execution.
Seamless integration with Azure OpenAI for real-world provider validation.
End-to-end testing for agentic resources including models, agents, and queries.
Advanced debug mode with resource persistence and failure pausing.
Standardized test structure using YAML manifests and Chainsaw definitions.
305 GitHub stars

ユースケース

Validating agentic workflow orchestration before production deployment.
Automating regression testing for complex AI-driven application logic.
Debugging failed resource deployments in isolated, reproducible environments.