AI Testing Demystified: Companion Page

Welcome, and thank you for reading AI Testing Demystified: A Practical Guide to Testing Machine Learning Models, LLMs and AI Agents. This page holds the free code and templates that go with the book.

Download the companion files

Download the code and templates (ZIP)

The zip contains:

  • code: the runnable examples from Chapters 3, 4, 5, 6 and 8, plus a requirements file.
  • templates: the first-hour spreadsheet, golden-set record, risk table, test plan outline, defect report, one-page test report, go/no-go checklist and the quick-reference card.

How to run the code

  1. Install Python 3.
  2. Unzip the folder, open a terminal inside the code folder and run: python3 -m pip install -r requirements.txt
  3. Run an example, for example: python3 ch04_model_tests.py
  4. For the chatbot and agent tests in Chapter 5, run: pytest

The output printed in the book comes from these files. Library versions change, so the last digit of a number may differ slightly.

Start here if you are new

Do the one-hour test at the start of the book. Use first_hour_spreadsheet.csv to set up your sheet, then work through the Five Questions: Is the data right? Is it right where it matters? Is it fair? Can it be attacked? Will it hold up?

Who this book is for

AI Testing Demystified is written for software testers and QA leads who are being asked to test machine learning models, LLM chatbots and AI agents for the first time. It also helps test managers, product owners and developers who need a plain-English way to decide whether an AI feature is ready to release. You do not need a data science background: the examples use Python, but each step is explained.

The Five Questions at a glance

The book organises AI testing around five questions. Use them as a checklist on any AI feature:

  1. Is the data right? Check that the data used to train and evaluate the system is accurate, complete and representative of real use.
  2. Is it right where it matters? Overall accuracy can hide failures. Test the specific groups, cases and scenarios where a wrong answer is most costly.
  3. Is it fair? Compare results across different groups of users so the system does not treat some people worse than others.
  4. Can it be attacked? Try misleading inputs, prompt injection and other misuse to see whether the system can be pushed into unsafe or wrong behaviour.
  5. Will it hold up? Check that quality stays stable over time, across repeated runs and as real-world data changes.

What each template is for

  • First-hour spreadsheet: a quick starting sheet for your first hour of testing a new AI feature.
  • Golden-set record: a place to keep your trusted test cases and their expected answers, so you can re-run them after every change.
  • Risk table: list what could go wrong, how likely it is and how serious it would be, to decide what to test first.
  • Test plan outline: a ready structure for writing an AI test plan your team and stakeholders can review.
  • Defect report: a format for logging AI issues clearly, including the input, the output and why it is wrong.
  • One-page test report: a short summary of results for managers and decision makers.
  • Go/no-go checklist: the final checks before you recommend releasing an AI feature.
  • Quick-reference card: the key ideas on a single page to keep beside you while you test.

Troubleshooting

  • “pip” or “python3” is not recognised: on Windows, try py -m pip install -r requirements.txt and py ch04_model_tests.py, or reinstall Python with “Add Python to PATH” ticked.
  • A package fails to install: upgrade pip first with python3 -m pip install --upgrade pip, then run the install again.
  • Your numbers are slightly different from the book: this is expected when library versions change. The pattern of results is what matters.

Frequently asked questions

Are the companion files free?

Yes. The code and templates are free for readers. You may copy and adapt the templates for your own work.

Do I need the book to use the templates?

The templates are usable on their own, but the book explains when and how to use each one, so you will get much more from them alongside it.

Where can I learn more about software testing?

Start with the beginner’s guide to software testing, then read agile testing explained and how to write test cases with ChatGPT.

Keep learning

Enjoyed the book?

A short, honest review on the page where you bought it helps other testers find the book. Thank you.

The code and templates are for learning. They are not a clinical or production tool, and nothing here is legal, medical or financial advice. You may copy and adapt the templates for your own work.