More and more of the software organizations already buy now comes with AI built in. The sales call covers features. The questions below cover what happens to your data, your people, and your liability once the contract is signed.
Start with the job, not the product
Before you talk to a vendor, write one sentence describing the job you want done. "Summarize intake notes for caseworkers." "Draft first replies to routine customer email." If you cannot write that sentence, you are not ready to buy yet, and the demo will fill the gap with whatever the vendor does best.
Then ask the vendor to show that task, using material that looks like yours. Not their prepared example. A demo built around their best case tells you what the tool does on a good day. You need to know what it does on an ordinary one.
The data questions come first
Most of the risk in an AI purchase sits in the data. Get written answers to these:
- Does our data train or improve your models? If the answer is "not by default," ask which setting controls it, who can change it, and whether it covers every user on the account or only the administrator.
- Where is our data stored and processed? Include subprocessors — the other companies the vendor relies on to run the service.
- How long do you keep prompts, uploaded files, and outputs? Can we set that period ourselves?
- What happens to our data when we leave? Ask for the deletion process and whether you get confirmation it happened.
- Who at your company can see our content, and under what circumstances?
Ask how it fails
Every AI tool produces wrong output sometimes. A vendor who says theirs does not has told you something useful — about the vendor.
- What kinds of mistakes does this tool make on the task we described?
- What does it do when it does not know — say so, or give a confident answer anyway?
- Can users see where an answer came from, such as the source document or passage?
- How do we report a bad output, and what happens after we do?
A good answer names specific weaknesses and explains what the product does about them. A vague answer about "continuous improvement" means nobody on the other side has thought about your failure case.
Ask what can change without notice
AI products change faster than most software. The model underneath can be swapped or updated, and the same request can give different results next month.
- Will you tell us before you change the underlying model?
- Can we stay on the current version while we test a new one?
- Can features we rely on be moved to a different plan?
This matters most when the tool sits inside a process you have documented or trained staff on. A quiet model change can make your training material wrong overnight.
Security and access, in plain terms
You do not need to be a security specialist to ask the basic questions. Your IT lead or managed service provider can judge the answers.
- Does it support single sign-on, so access ends when someone leaves?
- Can we control which users get which features?
- Is there an admin log showing who used the tool and when?
- Which independent security assessments or audit reports can you share, and how current are they?
Ask for the report itself, not a badge on a web page. And check the date on it.
Contract terms worth a second read
Have counsel review the agreement. These are the sections to point them to:
| Term | What to look for |
|---|---|
| Ownership of outputs | Who owns what the tool produces from your content |
| Use of your data | Training, product improvement, aggregation, and whether you can opt out |
| Indemnification | Who is responsible if output infringes someone else's rights |
| Liability cap | Whether the cap makes sense for how you plan to use the tool |
| Changes to terms | Whether the vendor can change the agreement by updating a web page |
| Termination and export | How you get your data out, in what format, and how quickly |
Clickthrough terms deserve the same attention. A staff member who signs up for a free account with a work email may be agreeing to terms nobody in your organization has read.
Ask who you will actually work with
For a small organization, support can matter more than features. Ask who answers when something goes wrong, how, and how fast. Ask whether onboarding and training are included or billed separately. And ask what the vendor expects you to have in place — an administrator, a policy, trained users — for the product to work the way it was sold.
A short scoring sheet
When you compare vendors, score each one on the same five lines, so the decision is not made by whoever gave the best demo:
- Data. Written, enforceable answers on training, retention, location, and deletion.
- Failure. A clear account of how it gets things wrong and what your users can see.
- Change control. Notice before model or term changes.
- Access. Sign-on, permissions, and logs your IT lead is satisfied with.
- Fit. It did your task, with your kind of material, in front of you.
A vendor that scores well on fit and poorly on data is not a bargain. It is a risk you are choosing to carry.
After you sign
The purchase is where the work starts. Add the tool to your record of where AI is used, name the person who owns it, and set the data rules for staff before rollout — your AI use policy is the place for them. Train the people who will use it on what it gets wrong, not only on what it does well. Then put a date on the calendar to re-read the vendor's terms, because they can change.
Questions people ask
What questions should I ask an AI vendor before buying?
Start with data: whether your data trains their models, where it is stored, how long it is kept, and how it is deleted when you leave. Then ask how the tool fails, whether you get notice before the model changes, and what access controls and logs you get.
How do I know if an AI tool uses my data for training?
Read the terms and privacy documentation for the exact plan you are buying, then ask the vendor in writing. If training is off by default, find out which setting controls it and who can change it. Get the answer into the contract or a document the contract references.
Is a free AI tool safe to use for work?
It depends on the terms. Free and paid plans from the same vendor can carry different data terms, so check the terms for the plan staff are actually using. Until someone has read them, keep client, confidential, and personal information out of it.
What should be in an AI vendor contract?
Ownership of outputs, how the vendor may use your data, indemnification, liability caps, whether the vendor can change terms by updating a web page, and how you get your data back when the contract ends. Have counsel review the agreement; this article is not legal advice.
Does a small organization need a formal AI vendor review?
Not a formal program, but the same questions. A one-page scoring sheet covering data, failure, change control, access, and fit keeps the decision from being made on the strength of a good demo.