An API is a collection of functions or methods, called API calls, that can be executed by other applications. Application developers code with help of existing APIs to make use of their functionality. This link is seamless and end-users of the application are generally unaware of using a separately developed API.
During testing, a test harness, an application that links the API and methodically exercises its functionality is constructed to simulate the use of the API by end-user applications. The interesting problems where testing should be concentrated are:
1. Ensuring that the test harness varies parameters of the API calls in ways that verify functionality and expose failures. This includes assigning common parameter values as well as exploring boundary conditions.
2. Generating interesting parameter value combinations for calls with two or more parameters.
3. Determining the content under which an API call is made. This might include setting external environment conditions (files, peripheral devices, and so forth) and also internal stored data that affect the API.
4. Sequencing API calls to vary the order in which the functionality is exercised and to make the API produce useful results from successive calls.
Test case Development Modes
Test case development will be done broadly in three modes:
Isolation Mode: In this method each Api is put to test individually. Test cases are developed around it. Might be few api’s require some initial configuration to be done.
State based Mode: In this mode, each api is invoked with different configuration settings, so that api is invoked in different states.
Sequence Mode: In this mode, set of api’s are put to test and executed in some specific sequence.
Isolation Mode Example
The “Send” method in WTT Messaging is taken as an example and it is as follows:
virtual HRESULT Send (
WTTMessageSendOptions sendOptions,
LPCWSTR header,
DWORD payloadSize,
LPBYTE payload
)
The Parameter SendOptions is another structure variable that needs to be assigned before passing to the method send, since we are testing the api in isolation mode we can assume that this parameter is passed is valid.
The parameter header takes a string. Valid data is an XML string. The various categories for this parameter is { TOO SHORT, NORMAL, TOOLENGTHY, NONPRINTABLEASCII , NOTHING}
Likewise for
payloadSize, the categories are {TOOSHORT, NORMAL, TOOLENGTHY, NEGATIVE, NOTHING}
payload, the categories are {TOOSHORT, NORMAL, TOOLENGTHY, NEGATIVE, NOTHING}
Each category that we have defined above needs to be mapped with range of or set of actual values that we will pass to test this api.
These categories are based on the specification. We will see that there are other categories, unspecified. Someone must decide whether these additional categories are bugs or undocumented features.
Parameter values might look like below,
| payloadSize [DWORD] | ||
| Valid1 | Send Exact payload Size | Send Exact payload Size |
| InValid1 | Send other than the exact size | Send other than the exact size, But it actually works fine with this also |
| Invalid Default | 0 | it fails to send message |
| payload [LPBYTE] | ||
| Valid1 | NULL | |
| Valid2 | MAXIMUM LENGTH | "this should be a long message … verry verry long message" |
| InValid | with terminator chars | "this sting invalid because it ends with \0 asdfadfsads" |
| Valid3 | NULL | no message |
| Header [LPCWSTR] | ||
| Valid1 | <?xml version=\"1.0\"?><WTTMessageHeader><Message Target=\"Test2\" RetryInterval=\"30\" Timeout=\"300\"/><WTTCommHeader><Connection><WTTCommProtocol DLL=\"WTTProtocolTCPIP\"><ProviderData ServerName=\"%s\" AlternateServerNames=\"%s\" Port=\"1780\" IPVersion=\"4\" /></WTTCommProtocol><WTTCommSecurity DLL=\"WTTNULLSecProv\"><ProviderData /></WTTCommSecurity></Connection></WTTCommHeader></WTTMessageHeader> | Ideal Header with all valid values |
| InValid1 | <?xml version=\"1.0\"?><WTTMessageHeader><Message Target=\"TestInValid\" RetryInterval=\"30\" Timeout=\"300\"/><WTTCommHeader><Connection><WTTCommProtocol DLL=\"WTTProtocolTCPIPInValid\"><ProviderData ServerName=\"%s\" AlternateServerNames=\"%s\" Port=\"1780\" IPVersion=\"4\" /></WTTCommProtocol><WTTCommSecurity DLL=\"WTTNULLSecProv\"><ProviderData /></WTTCommSecurity></Connection></WTTCommHeader></WTTMessageHeader> | Inavlid Header with invalid dll,taget name etc |
State based Mode Example
In this mode the example send api would be invoked with different configurations of initial settings.
The parameter WTTMessageSendOptions is the one which is more effective in these tests.
The various states that this parameter can represent are { PERSIST, PERSISTONERROR, PERSISTORDER }
Following are the sample values that this parameter can take:
| WTTMessageSendOptions.StoreAndForwardFlag [DWORD] | ||
| Valid1 | WTTMSG_FLAGS_PERSIST (0x00000001) | Any Case Copies Message locally to the file and then tries to send |
| Valid2 | WTTMSG_FLAGS_PERSISTONERROR (0x00000010) | On Error only it will persist |
| Valid3 | WTTMSG_FLAGS_PERSISTONERROR_ORDERED (0x00000100) | On Error only it will persist and sends message later in order |
| Valid4 [Default] | 0x00000000 | It works |
| Valid5 | 0x11111111 | It works |
| WTTMessageSendOptions.ResponseTimeout [DWORD] | ||
| Valid1 | 120 | Common value (In Seconds) |
| Valid2 | 2^32 - 1 | Maximum DWORD Value |
| Valid3 | 0 | Recieiver timesout after 0 seconds before receiving the entire payload |
| WTTMessageSendOptions.MessageExpiry [DWORD] | ||
| Valid1 | 1 | Any common value(Time in hours that message will be persistat most) |
| Valid2 | 2^32 - 1 | Maximum DWORD Value |
| InValid | ?? | No Invalid, It is meanigful to give any values in DWORD range |
| WTTMessageSendOptions.RetryInterval [DWORD] | ||
| Valid1 | 5 | Any common value(Time in hours that message will be persistat most) |
| Valid2 | 2^32 - 1 | Maximum DWORD Value |
| Default | 0 | NotSure |
| InValid | ?? | No Invalid, It is meanigful to give any values in DWORD range |
| WTTMessageSendOptions.ReuseConnections [BOOL] | ||
| Valid1 | TRUE | Pools Connections |
| vaLID2 | FALSE | new connection |
Sequence Mode Example
Sequence mode is same as State base mode except here the initial configuration/state for each API is configured by running a sequence of other API’s.
Hybrid Model
We have the hybrid technique which uses all of the above said modes (isolation, state, sequence) to test api. First each individual api, is applied with the different values for the parameters that are planned and each api is moved to a particular state, which is made by either executing some other api or because of the input parameter values. As it can be seen that this technique combines all the above said api testing models in to one.
Conclusion
Usage of this Hybrid Model for generating test cases for api’s will test the api and will cover all scenarios of the tests. These testcases cannot be generated manually because of their complexity of permutations and combinations. We have few tools in the market to automate these cases so that the test cases are generated automatically like Test Run, QEngine.