Advanced Domain Testing
Schema for Domain Testing
Simple examples make Domain Testing (aka EP & BVA) look easy. Nevertheless, even seemingly simple examples can show what kind of deep, structured analysis is possible.
In The Domain Testing Workbook1, Kaner et al. offer a schema (set of steps) to carry out advanced domain testing. Below is a simplified, distilled version:
A. Identify explicit and implicit variables.
Not all variables are conveniently laid out in requirements for testers and developers to consume and convert into tests. See (B) below for where to search for implicit variables.
B. Determine the dimension, type, and scale of each variable
Amount? Length? Size? Time? Speed? Frequency? Hardware configuration?
Minimum increment? 1? 0.1? 0.0005? Does the increment change past certain limits?
Can it be ordered?
C. Determine how this variable is used
How it interacts with other variables immediately or later
How it is stored or propagated
D. Partition and create EP & BVA Table with best representatives
Best representatives mean "highest chance of revealing a fault".
Usually border values, but consider 0 and other special technical or business values
Include invalid values
Add risk
E. Imagine and document risks beyond the obvious or stated requirements
ATM Example
Requirements: An ATM allows withdrawals of cash in amounts of $20 increments from $20 to $200
A. Identify explicit and implicit variables
Explicit variables: amount requested
Implicit variables: amount received, current balance, updated balance
Other potential variables: daily / other temporal withdrawal limit that applies to explicit or implicit variables
B. Determine the dimension, type, scale of each variable
Money, from $20 to $200, an input variable, not an expected output
Decimals should not be valid (20.1 or 20.01) but double-check
0 is probably not valid
Max input length is 3 ("200" is 3 digits long - important for future test cases)
C. Determine how this variable is used
Amount Requested interacts with balance
If accepted, updates the balance
If accepted, corresponds to the amount given out
It is probably logged and stored somewhere
It should show up on statements. Where else? Internal reports? Research and test.
D. Partition and create EP & BVA Table with best representatives
One way to fill up a EP & BVA table is to add comments and questions for some inputs that must be clarified. See the Comment column.
Primary dimension table: input number
Valid amount
20
Check amount received and updated balance
Smallest allowed value
200
Check amount received and updated balance
Largest allowed value
40 to 180, all valid increments of 20
Check amount received and updated balance
Exhaustive testing of valid values is possible due to limited set size
20, with balance of $20.
Check amount received and updated balance
Focus on driving a dependent variable (balance) to an allowed extreme. Any analogous amount + balance pair may be used.
Invalid amount
0
Rejected, balance unchanged
What should happen after? Release card or keep it, notify of user error and offer to try again?
199, 199.9
As above
As above. Precise input subject to interface limitations (not possible to specify decimals).
201, 200.1
As above
As above
As above
Focus: length of input. Attempt 256-long (28 + 1 boundary) or longer input.
Secondary dimensions table: length, time, sequence, other
Length (of input)
2000000000
Rejected, balance unchanged
Focus: length of input. Attempt 256-long (28 + 1 boundary) or MAXINT + 1 or longer input.
Time
Input amount Under Max Time (boundary)
Check amount received and updated balance
How long before the waiting for input times out? Between each key stroke or for the entire input?
Input amount Over Max Time
?
As above.
Withdraw exactly up to the daily / weekly / other temporal limit
Focus on the border value of a related variable of another dimension: time.
Withdraw beyond the daily / weekly / other temporal limit
As above
E. Imagine and document risks beyond the obvious or stated requirements
This is the most interesting part for further iterative exploration and testing. Strictly speaking, this may simply be called "deep, critical analysis" that goes far beyond explicitly stated requirements.
The necessity of such analysis is reflected in the reality that the unknown is vastly greater than the known for any individual or a team. When simplified:
Known: stated requirements + our knowledge
Known unknown: "It's not written down yet", but you thought about it, and you'll find out
Unknown unknown: "No one even knew or thought this was necessary to consider."
Risk analysis doesn't really belong to a particular technique, and it may result in using other test techniques, such as Decision Table or State Based Testing. Below is a non-exhaustive list.
Balance
Invalid input or insufficient balance: how many failed attempts should the ATM be willing to tolerate before doing something else?
Overdrafts allowed? Fee applied, if any? Fee applied to what balance? Fee included in the total when calculating whether the withdrawal is within the allowed limit? Overdraft rules?
Currency conversion: if there is a second account (e.g., savings) in another currency, can the withdrawal use that money? What currency conversion rules apply? This is a real feature with some financial institutions.
Temporal limit variations: change the daily/weekly/ etc. limit. See if it takes effect with a withdrawal. Reset the limit, withdraw again.
Async/Concurrent behavior
Assuming two cards for one account, withdraw "at the same time".
Buy something online and withdraw "at the same time".
Card
What if the card is blocked? Other states invalidating its use? How should the ATM react to withdrawal or other requests?
If the ATM swallows the card, what message should it display? How helpful is it? How long should the message be displayed for?
Is the card's withdrawal limit at odds with the ATM rules? (Card max. $100, ATM max. $200).
Card from another bank that is allowed to withdraw? Service fee?
Card from another bank NOT allowed to withdraw? Behavior?
Service charge for a withdrawal?
ATM
What if there are not enough banknotes in the ATM for the requested amount? Best, non-confusing way to inform the user?
What if the ATM fails to communicate with the back-end?
Withdrawal confirmation receipt? Check the correct data for successful and failed withdrawals.
What if the user never retrieves the cash after it was given out?
References
The Domain Testing Workbook: Cem Kaner, Sowmya Padmanabhan, Douglas Hoffman
Last updated