This post originally appeared on https://novatec-gmbh.de
The OpenAPI specification has already established itself in many project and application areas. It can be applied to describe HTTP APIs in a standardized way, understandable for humans and computers. While OpenAPI is mainly intended for synchronous interfaces, asynchronous communication places new demands on the interface definition.
Asynchronous architectures can build on different types of protocols, e.g., Kafka, AMQP, or MQTT. Additionally, communication often includes a messaging broker, which maintains several topics or channels. Another difference is the communication style: OpenAPI only allows specifying one-to-one connection links. In contrast to that, asynchronous communication often involves multiple communication partners. Therefore, common patterns like publish/subscribe require a new approach to defining APIs.
AsyncAPI was developed as an extension of OpenAPI to meet these new requirements of asynchronous communication and interfaces. This article highlights the differences between OpenAPI and AsyncAPI in more detail.
As another aspect, large projects usually include multiple teams working and developing together. Therefore, standardized development of APIs is a central aspect of cross-team application development. At the same time, the API definitions need to be available and easy to access for all teams. These demands are also addressed by Apicurio Registry. It supports registering different document formats, e.g., OpenAPI, AsyncAPI, GraphQL, Apache Avro, or Protobuf.
Since AsyncAPI is developing quite fast, we want to have a look at what the specification and corresponding tools currently cover. Additionally, we will shortly examine how Apicurio and AsyncAPI can work together at the moment. Nevertheless, there are many new tools and features for AsyncAPI on the roadmap. For this reason, this article can only be a snapshot. So stay tuned until the end of this post to get an outlook on what is next to come.
Elements of an AsyncAPI Document
Coming to the basics of AsyncAPI: How does an AsyncAPI definition look? AsyncAPI allows using YAML or JSON for document definition. A document consists of the following elements:
In practice, the definition can become quite long. Consider the below example from AsyncAPI’s GitHub to get an impression of how such a definition can look:
1asyncapi: '2.2.0'
2info:
3 title: Streetlights Kafka API
4 version: '1.0.0'
5 description: |
6 The Smartylighting Streetlights API allows you to remotely manage the city lights.
7 license:
8 name: Apache 2.0
9 url: https://www.apache.org/licenses/LICENSE-2.0
10
11servers:
12 test:
13 url: test.mykafkacluster.org:8092
14 protocol: kafka-secure
15 description: Test broker
16 security:
17 - saslScram: []
18
19defaultContentType: application/json
20
21channels:
22 smartylighting.streetlights.1.0.event.{streetlightId}.lighting.measured:
23 description: The topic on which measured values may be produced and consumed.
24 parameters:
25 streetlightId:
26 $ref: '#/components/parameters/streetlightId'
27 publish:
28 summary: Inform about environmental lighting conditions of a particular streetlight.
29 operationId: receiveLightMeasurement
30 message:
31 $ref: '#/components/messages/lightMeasured'
32
33components:
34 messages:
35 lightMeasured:
36 name: lightMeasured
37 title: Light measured
38 summary: Inform about environmental lighting conditions of a particular streetlight.
39 contentType: application/json
40 payload:
41 $ref: "#/components/schemas/lightMeasuredPayload"
42
43 schemas:
44 lightMeasuredPayload:
45 type: object
46 properties:
47 lumens:
48 type: integer
49 minimum: 0
50 description: Light intensity measured in lumens.
51 sentAt:
52 $ref: "#/components/schemas/sentAt"
53 sentAt:
54 type: string
55 format: date-time
56 description: Date and time when the message was sent.
57
58 securitySchemes:
59 saslScram:
60 type: scramSha256
61 description: Provide your username and password for SASL/SCRAM authentication
62
63 parameters:
64 streetlightId:
65 description: The ID of the streetlight.
66 schema:
67 type: stringIf you want to look at it in a more readable way, you can copy and paste the definition to the AsyncAPI Studio or the AsyncAPI playground(Remark: AsyncAPI Playground will soon be archived and replaced by AsyncAPI Studio). More examples are available on GitHub.
I do not want to dive too deep into the exact details of the specification because the documentation is already helpful enough for this. But I want to highlight some parts of the specification that could be the icing on the cake when considering it.
Besides some general information about the API, like the title and the description, the "Info" object can contain a "Contact" object. Especially when thinking about multiple groups working together, the responsible team and how to reach them can be linked at this place. In the example from above, this would look like this:
1asyncapi: '2.2.0'
2info:
3 title: Streetlights Kafka API
4 version: '1.0.0'
5 description: |
6 The Smartylighting Streetlights API allows you to remotely manage the city lights.
7 contact:
8 name: API Support
9 url: https://www.asyncapi.org/support
10 email: support@asyncapi.org
11 license:
12...The fixed structure defined by the specification might not be sufficient for all use cases. Therefore, it is possible to link external documentation for nearly every object in the AsyncAPI definition. Additionally, user-defined properties can be set to extend the specification. This can be done by simply prepending an "x-" at the field name. For example, we could add the internal team or project name:
1...
2 contact:
3 name: API Support
4 url: https://www.asyncapi.org/support
5 email: support@asyncapi.org
6 x-project-name: Future-Light-X10
7 x-team-name: Kafka Streetlights Team
8...For long and detailed API definitions, the document can become messy fast. As an improvement for this, it is possible to include references to internal and external resources (using the $ref field). They enable us to reuse objects, add schemas and add definitions from external sources.
As another use case, let’s assume there are already event-driven applications using an Avro schema for their messages. For example, the schema is registered at Confluent Schema Registry or Apicurio Registry. Using $ref, it is possible to reference this Avro schema within a message object in the AsyncAPI definition:
1...
2messages:
3 lightMeasured:
4 name: lightMeasured
5 title: Light measured
6 summary: Inform about environmental lighting conditions of a particular streetlight.
7 contentType: avro/binary
8 schemaFormat: application/vnd.apache.avro+json;version=1.9.0
9 payload:
10 $ref: 'http://schema-registry:8081/subjects/topic/versions/1/#LightMeasuredPayload'
11...AsyncAPI also offers an Avro schema parser which collects local Avro references or remote files from Confluent Schema Registry and inserts them into the definition.
Kafka, AMQP, MQTT or HTTP: Protocol-Specific Properties
As already mentioned at the beginning of this post, asynchronous communication can involve different types of protocols. As this is a fundamental aspect of your API definitions, AsyncAPI supports adding protocol-specific properties.
These protocol-specific properties can be necessary on different levels: server level, channel level, operation level, and message level. For these levels, AsyncAPI allows defining protocol-specific objects, so-called bindings. This repository provides a complete list of all bindings and their specification details.
For example, it is possible to add a key for Kafka messages:
1...
2channels:
3 smartylighting.streetlights.1.0.event.{streetlightId}.lighting.measured:
4 description: The topic on which measured values may be produced and consumed.
5 parameters:
6 streetlightId:
7 $ref: '#/components/parameters/streetlightId'
8 publish:
9 summary: Inform about environmental lighting conditions of a particular streetlight.
10 operationId: receiveLightMeasurement
11 message:
12 bindings:
13 kafka:
14 key:
15 type: string
16 enum: ['Germany', 'Austria', 'Switzerland']
17 bindingVersion: '0.1.0'
18 $ref: '#/components/messages/lightMeasured'
19...Unfortunately, for many protocols, there are a lot of binding objects which are not specified yet. E.g., for Kafka, there are no specifications for the Server and Channel Binding objects. In the case of Kafka, relevant but missing properties could be the number of partitions or the delivery guarantee. These properties would be a good use case for specification extensions that currently cannot be used in binding objects.
In addition, some conventions for the specification of bindings are on the roadmap. As protocol-specific bindings are still in the alpha version, this feature will become mature with continuous development.
AsyncAPI and Apicurio: A Match?
In contrast to the Confluent Schema Registry, Apicurio is not limited to message schemas but also supports API specifications. Apicurio could therefore also replace the Confluent Schema Registry.
However, this is probably only true if the architecture does not mainly base on a Confluent ecosystem. But depending on the use case, Apicurio can be a helpful addition to the Schema Registry due to the additionally supported formats.
Apicurio Registry allows registering AsyncAPI definitions, which can then be searched and browsed. At the moment, unfortunately, AsyncAPI documents are not rendered to make them more readable. External references (like in the example above) are currently not resolved automatically either. Orientation for this could be the AsyncAPI playground, where this is already possible. However, Apicurio Studio already offers this functionality for OpenAPI, and support for AsyncAPI in Apicurio Studio is already planned.
At the same time, Apicurio also offers the Data Models Library, which allows reading, writing, and modifying AsyncAPI and OpenAPI documents. The Dereferencer-class resolves external references to other documents and inserts them into the definition. It is also possible to add your implementations of the IReferenceResolver interface for custom resolution of references.
When testing the library, I noticed that it cannot resolve all references in a document yet: The AsyncAPI specification also allows references in the payload object of a message (as shown in the example above). The library only checks objects for links that also implement the IReferenceNode interface. However, the payload object does not yet implement this interface, and therefore, references in this object remain unresolved.
There already is a GitHub issue which will hopefully resolve this soon.
For Avro schemas, Apicurio Registry supports schema validation and a compatibility check. However, this seems to be still missing for AsyncAPI, even when the user interface suggests the opposite (see screenshot below). I tested this feature with syntactically invalid AsyncAPI definitions (which were classified as invalid by AsyncAPI playground), which were not recognized as wrong by Apicurio Registry. Also, definition updates, including breaking changes, were not recognized as so. Nevertheless, this GitHub issue suggests that support is planned.
The Data Models Library of Apicurio and Apicurio Registry gives a first idea of what can be done with AsyncAPI. Nevertheless, the functionalities and use cases are still limited. When I tested the Data Models Library, I hoped for more features that could be used at runtime. In fact, it is currently cumbersome to extract message schemas from the AsyncAPI definition. Therefore, I would not recommend the library for this use case.
However, one possibility would be to use Confluent Schema Registry or Apicurio Registry for Avro schemas. Messages could be serialized and deserialized with the schemas from there. The AsyncAPI definition can then reference these schemas. In addition, Apicurio Registry can be used as an API catalog. Developers could then use this catalog to work on new services or to mock applications.
Besides these aspects, you should consider that Apicurio has little to no competition. At least I could not find another established open-source registry on the market that supports both schemas and APIs and so many formats. Feel free to leave a comment if you know more about this!
As an alternative, it is also possible to build your own AsyncAPI catalog to support missing features. E.g., you could use existing AsyncAPI libraries and modules like the AsyncAPI React component to render definitions. If you want to know how this could look, have a look at this repository where I implemented this myself. In this prototype, I used the React component with Angular. In case you feel more comfortable with other frameworks, it is also possible to integrate it in Vue or NextJS.
If you want to test how Apicurio works, check out this small quick start guide. It starts an Apicurio container and deploys some AsyncAPI example files.
The Roadmap: What is Next to Come
If you take a closer look at the GitHub repositories of AsyncAPI, you will quickly notice that a lot is happening here right now. There are many exciting and promising ideas, and the community seems to grow steadily. For example, an event gateway is currently being developed that will make it possible to validate and modify messages even before they arrive at the broker. Another idea is the Glee framework, which ensures that AsyncAPI definition and code match. At first glance, an unexpected project is the AsyncAPI Chatbot. It helps to create an AsyncAPI document without having to know the specification yourself.
Besides these innovative ideas, there are also tools to enable validation and compatibility checks. The CLI tool validates AsyncAPI files. Also, the Diff tool points out breaking changes to ensure backward compatibility. A combination of these projects would be helpful, as it is already requested on GitHub. These two tools could also be integrated into Apicurio Registry to add missing features.
The AsyncAPI website also lists tools developed by the community. Many of these and several tools by the core team mainly support JavaScript - so ideally, you should not have a problem with that. Nevertheless, it is to be hoped that other programming languages will be supported in the future. This would make it easier for the community to access them.
Many other official projects of the AsyncAPI team are still in the alpha or beta phase. Therefore, most of them cannot yet be used to their full potential. But since companies like eBay and Slack already use AsyncAPI, we can expect a lot more to come.
If you want to dive a bit deeper, I recommend having a look at the recording of the AsyncAPI conference in November 2021. There were many interesting talks on how AsyncAPI can be used in projects and with other tools. For more details on how to use schemas, schema registries, and API registries with AsyncAPI, listen to my conference talk below.