Currently, openapi specs are authored completely by hand by wazo developers, alongside the API code.
When a new API is added, it is up to the developer to write additional spec for it.
When an existing API changes, it is up to the developer to update the spec for it to match the API.
There is currently no mechanism to automatically verify that the openapi spec is correct and matches the API implementation. This leaves the spec vulnerable to human errors. It is sometimes found incomplete or incorrect.
It should be possible to use the openapi spec to automatically generate requests to an API and validate that the response also conforms to the spec.
Details
Spec-based Mock
MockServer is an API mocking tool that are used in integration tests to mock an API server(another wazo component or a third-party API that is used by the component under test).
MockServer supports consuming an openapi specification in order to configure the expectations, which are currently defined “ad-hoc”/”by hand” in the tests setup.
The expectations of a mock must logically be consistent with the spec, as the spec must logically be consistent with the implementation.
API fuzzing
If an openapi spec includes descriptions of all available parameters, including a complete & valid schema for the request body(for POST, PUT & PATCH), then it should be possible to generate a well-formed request.
Those requests may not be semantically valid, e.g. if some string parameters are actually references to existing entities, or subject to validation that are not expressed in the spec. Tests requiring semantic validation are currently added manually as integration tests.
Spec-based testing could at least verify that responses to autogenerated requests match one of the specified response spec, even if that response is an error(e.g. 400).
Spec-based testing can help improve the spec to be more complete and correct, as well as to find corner-cases that should be supported according to the spec, but are not in the implementation.
See for example .
Integrating spec validation in existing tests
Using , we could validate that requests & responses used in integration tests also match the spec for the API under test.
This could be done automatically by integrating this into test frameworks(wazo-test-helpers).
Medium
Currently, openapi specs are authored completely by hand by wazo developers, alongside the API code.
When a new API is added, it is up to the developer to write additional spec for it.
When an existing API changes, it is up to the developer to update the spec for it to match the API.
There is currently no mechanism to automatically verify that the openapi spec is correct and matches the API implementation. This leaves the spec vulnerable to human errors. It is sometimes found incomplete or incorrect.
It should be possible to use the openapi spec to automatically generate requests to an API and validate that the response also conforms to the spec.
Details
Spec-based Mock
MockServer is an API mocking tool that are used in integration tests to mock an API server(another wazo component or a third-party API that is used by the component under test).
MockServer supports consuming an openapi specification in order to configure the expectations, which are currently defined “ad-hoc”/”by hand” in the tests setup.
The expectations of a mock must logically be consistent with the spec, as the spec must logically be consistent with the implementation.
API fuzzing
If an openapi spec includes descriptions of all available parameters, including a complete & valid schema for the request body(for POST, PUT & PATCH), then it should be possible to generate a well-formed request.
Those requests may not be semantically valid, e.g. if some string parameters are actually references to existing entities, or subject to validation that are not expressed in the spec. Tests requiring semantic validation are currently added manually as integration tests.
Spec-based testing could at least verify that responses to autogenerated requests match one of the specified response spec, even if that response is an error(e.g. 400).
Spec-based testing can help improve the spec to be more complete and correct, as well as to find corner-cases that should be supported according to the spec, but are not in the implementation.
See for example .
Integrating spec validation in existing tests
Using , we could validate that requests & responses used in integration tests also match the spec for the API under test.
This could be done automatically by integrating this into test frameworks(wazo-test-helpers).