diff --git a/SUMMARY.md b/SUMMARY.md index ca86068..d7b4472 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -66,5 +66,13 @@ ## Asynchronous API Guidelines -* [Introduction](asynchronous-api-guidelines/asyncapi.md) +* Preliminary concepts + * [Introduction](asynchronous-api-guidelines/01_introduction/a_introduction.md) + * [Basic concepts](asynchronous-api-guidelines/01_introduction/b_basic_concepts.md) +* [General guidelines for asynchronous APIs](asynchronous-api-guidelines/02_asynchronous_api_guidelines/main.md) +* AsyncAPI specs for Kafka + * [Introduction](asynchronous-api-guidelines/03_asyncapi_kafka_specs/a_introduction.md) + * [Guidelines](asynchronous-api-guidelines/03_asyncapi_kafka_specs/b_guidelines.md) + * [Tooling](asynchronous-api-guidelines/03_asyncapi_kafka_specs/c_tooling.md) + diff --git a/assets/eda_overview.png b/assets/eda_overview.png new file mode 100644 index 0000000..1fdfbc0 Binary files /dev/null and b/assets/eda_overview.png differ diff --git a/asynchronous-api-guidelines/01_introduction/a_introduction.md b/asynchronous-api-guidelines/01_introduction/a_introduction.md new file mode 100644 index 0000000..17947f6 --- /dev/null +++ b/asynchronous-api-guidelines/01_introduction/a_introduction.md @@ -0,0 +1,22 @@ +# adidas Asynchronous API guidelines + +## Introduction + +### About guidelines + +In the scope of a company, organization or team, the document known as guidelines is a set of best practices or hints provided by a group of subject matter experts in that particular technology. The most important aspects of it: + +- Help to create a standardized way of completing specific tasks, making outcome more predictable and alike +- Help to identify do's and dont's with regards to a specific technology or tool +- Help to avoid gotchas or problems related with company specifics +- Gather the knowledge of several Subject Matter Experts and prevent others for falling in frequent caveats or mistakes + +**Note** In any case, the content in the guidelines should be taken as recommendations, not something that needs to be done in a mandatory way. + + + + + + + + diff --git a/asynchronous-api-guidelines/01_introduction/b_basic_concepts.md b/asynchronous-api-guidelines/01_introduction/b_basic_concepts.md new file mode 100644 index 0000000..dd18c06 --- /dev/null +++ b/asynchronous-api-guidelines/01_introduction/b_basic_concepts.md @@ -0,0 +1,19 @@ +# adidas Asynchronous API guidelines + +## Basic concepts about asynchronous APIs + +### Event-driven architectures + +An Event-Driven Architecture (EDA) uses events to trigger and communicate between services and is common in modern applications built with microservices. An event is a change in state, or an update, like adding a shopping item in a cart on an e-commerce website. + +![EDA overview](../../assets/eda_overview.png) + +In most cases, EDAs are broker-centric, as seen in the diagram above. There are some new concepts in that diagram, so let's go through them now. + +In Event-Driven Architecture (EDA), an application must be a producer, a consumer, or both. Applications must also use the protocols the server supports if they wish to connect and exchange messages. + +### Messages and events + +A message carries information from one application to another, while an event is a message that provides details of something that has already occurred. One important aspect to note is that depending on the type of information a message contains, it can fall under an event, query, or command. + +Overall, events are messages but not all messages are events. \ No newline at end of file diff --git a/asynchronous-api-guidelines/02_asynchronous_api_guidelines/main.md b/asynchronous-api-guidelines/02_asynchronous_api_guidelines/main.md new file mode 100644 index 0000000..71d92b5 --- /dev/null +++ b/asynchronous-api-guidelines/02_asynchronous_api_guidelines/main.md @@ -0,0 +1,93 @@ +# adidas Asynchronous API guidelines + +## Asynchronous API guidelines + +### Contract + +Approved API Design, represented by its API Description or schema, **MUST** represent the contract between API stakeholder, implementers, producers and consumers. + +That contract **MUST** contain enough information to use the API (servers, URIs, credentials, contact information, etc). + +### API First + +Asyncrhonous APIs **SHOULD** use the API First principle : + +- The API designs **SHOULD** involve all relevant stakeholders (developers, consumers, ...) to ensure that final design fulfil requirements from different perspectives +- The resulting API specification will be the source of truth rather than the API implementation + +### Immutability + +After agreement with the stakeholders the contract **MUST** be published in order to do it immutable. Changes to the API related to the data model, **MUST** be published in a schema registry. + +Schema registry acts as a central location for storing and accessing the schemas of all published APIs. + +### Common data types + +The API types **MUST** adhere to the formats defined below: + +| Data type | Standard | Example | +| --------- | -------- | ------- | +| Date and Time | [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) | 2017-06-21T14:07:17Z (Always use UTC) | +| Date | [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) | 2017-06-21 | +| Duration | [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) | P3Y6M4DT12H30M5S | +| Time interval | [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) | 2007-03-01T13:00:00Z/2008-05-11T15:30:00Z | +| Timestamps | [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) | 2017-01-01T12:00:00Z | +| Language Codes | [ISO 639](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) | en <-> English | +| Country Code | [ISO 3166-1 alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) | DE <-> Germany | +| Currency | [ISO 4217](https://en.wikipedia.org/wiki/ISO_4217) | EUR <-> Euro | + +### Schemas and data evolution + +All asynchronous APIs **SHOULD** leverage Schema Registry to ensure consistency across consumers/producers with regards to message structure and ensuring compatibility across different versions. + +The default compatibility mode in Schema Registry is FULL_TRANSITIVE. This is the more restrictive compatibility mode, but others are also available. + +| Mode | Description | +|------|-------------| +|BACKWARD|new schema versions are backward compatible with older versions| +|BACKWARD_TRANSITIVE|backward compatibility across all schema versions, not just the latest one.| +|FORWARD|new schema versions are compatible with older consumer versions| +|FORWARD_TRANSITIVE|forward compatibility across all schema versions.| +|FULL|both backward and forward compatibility with the latest schema version| +|FULL_TRANSITIVE|both backward and forward compatibility with all schema versions| +|NONE|schema compatibility checks are disabled| + +If for any reason you need to use a less strict compatibility mode in a topic, that compatibility mode **SHOULD NOT** be modified on the same topic. Instead, a new topic **SHOULD** be used to avoid unexpected behaviors or broken integrations. + +Applications **MUST NOT** enable automatic registration of schemas because FDP's operational model for the Schema Registry relies on GitOps (every operation is done through GIT PRs + automated pipelines) + + Please refer to [Kafka_Schema_Registry-Default_Requirements](https://confluence.tools.3stripes.net/display/FDP/Kafka_Schema_Registry-Default_Requirements) for more information about Schema Registry. + + ### Key/Value message format + + Kafka messages **MAY** include a key, which needs to be properly designed to have a good partition balanceare key-value pairs. + +The message key and the payload (often called value) can be serialized independently and can have different formats. For example, the payload of the message can be sent in AVRO format, while the message key can be a primitive type (string).  + +Message keys **SHOULD** be kept as simple as possible and use a primitive type when possible. + +### Naming conventions + +As general naming conventions, asynchronous APIs **MUST** adhere to the following conventions + +- Use of english +- Avoid acronyms or explain them when used +- Use camelCase unless stated otherwise + +### Protocols + +Protocols define how clients and servers communicate in an asynchronous architecture. + +The accepted protocols for asynchronous APIs are: + +- Kafka +- HTTPs +- WebSockets +- MQTT + +This version of the guidelines focuses on Kafka protocol, but it could be extended in the future. In any case, this document will be updated to reflect the state of the art. + +### Security + +The [security guidelines](https://github.com/adidas/api-guidelines/blob/feature/asyncapi-guidelines/general-guidelines/security.md) for regular APIs **MUST** be followed strictly when applicable. + diff --git a/asynchronous-api-guidelines/03_asyncapi_kafka_specs/a_introduction.md b/asynchronous-api-guidelines/03_asyncapi_kafka_specs/a_introduction.md new file mode 100644 index 0000000..293509d --- /dev/null +++ b/asynchronous-api-guidelines/03_asyncapi_kafka_specs/a_introduction.md @@ -0,0 +1,60 @@ +# adidas Asynchronous API guidelines + +## Introduction to AsyncAPI spec definitions for Kafka protocol + +This section is specific to the definition of API specs with [AsyncAPI](https://www.asyncapi.com/) for Kafka protocol. + +Also, take into account that across the section there will be multiple references to this [AsyncAPI reference spec](https://design.api.3stripes.io/apis/adidas/asyncapi-adoption-initiative/1.0.0) which is publicly available for reference.  + +### Basic concepts about AsyncAPI + +#### Why AsyncAPI? + +Event-driven architectures are becoming increasingly popular for building scalable, responsive, and efficient applications. AsyncAPI plays a crucial role in this landscape by offering a standardized way to describe asynchronous APIs, similar to how OpenAPI does for REST APIs. AsyncAPI seeks to make the development, maintenance, and testing of asynchronous APIs easier by providing a machine-readable specification. + +It supports various messaging protocols, including MQTT, WebSocket, Kafka, AMQP, and more, making it versatile for different use cases. In adidas, we will use it mainly to document Kafka resources created in FDP but nothing prevents you from using it for a different purpose. + +The benefits of using AsyncAPI are, amongst others: + +- Standardization + - AsyncAPI defines a STANDARD format (YAML or JSON) for describing asynchronous APIs. + - By defining the structure of messages, channels, and events, you can ensure that all components adhere to the same conventions. + - Using a single standard ensures consistency in the design and documentation of all your asynchronous APIs. + - This simplifies integration, maintenance, and troubleshooting across different parts of your system. +- Improved Developer Experience + - AsyncAPI documents the messages being exchanged, their structure, and the events triggered by them. + - It provides developers with a clear picture of how to interact with the API, what data to expect, and how to interpret responses without digging into the implementation details.  +- Code scaffolding + - Using tools like asyncapi-generator allow to easily generate the skeleton of applications that can work with the resources described in the spec. + - This can be done in different programming languages (Python, Java, Node.js. ...), reducing significantly the development time and the coding errors. +- Design-first approach: It encourages designing the API first before writing code, leading to better planned and more reliable APIs. + +In addition to those benefits, Platform & Engineering is working hard to create a data catalogue built upon AsyncAPI that allows to have a good level of discoverability, allowing teams to be able to find exactly the data they need with regards to any data object in the company. + +Questions like: + +- Who is responsible for a specific data object +- Where is that data hosted +- Which kind of information is available + +Will be easy to answer once this catalogue is in place. Also, it is important to have a good discoverability and search & filtering capabilities. + +#### Kafka to AsyncAPI concept mapping + +|Kafka Concept|AsyncAPI Concept| +|-------------|----------------| +|broker|server| +|topic|channel| +|consumer|subscriber| +|producer|publisher| + +#### First level items in AsyncAPI structure + +|Element|Meaning| +|-------|-------| +|asyncapi|Specifies the AsyncAPI specification version| +|info|Provides metadata about the API such as the version, title and description| +|servers|Describes servers where the API is available| +|channels|Defines the channels through which messages are received/published| +|components|Reusable elements to be references across the spec| + diff --git a/asynchronous-api-guidelines/03_asyncapi_kafka_specs/b_guidelines.md b/asynchronous-api-guidelines/03_asyncapi_kafka_specs/b_guidelines.md new file mode 100644 index 0000000..4da1555 --- /dev/null +++ b/asynchronous-api-guidelines/03_asyncapi_kafka_specs/b_guidelines.md @@ -0,0 +1,249 @@ +# adidas Asynchronous API guidelines + +## AsyncAPI guidelines for Kafka + +### AsyncAPI version + +Any version of AsyncAPI **MAY** be used for spec definitions. + +However, to be aligned with adidas tooling, spec versions **SHOULD** be *v2.6.0*, because to the date of this document creation (April 2024) this is the highest supported version on Swaggerhub, the current API portal to render, discover and publish specs. + +```yaml +asyncapi: 2.6.0 +... +``` + +### Internal vs public specs + +AsyncAPI specs **MAY** be created both for public APIs or for internal APIs.  + +- Public APIs are those who are created to be consumed by others +- Internal APIs are only for development teams for a particular project + +There are no differences with regards to the spec definition, but internal APIs **SHOULD** have restricted access limited only to the internal development team for a particular project or product. + +This access control is handled through Role-Based Access Control (RBAC) implemented in Swaggerhub. + +### Spec granularity + +In FDP all resources are grouped by namespace. + +For that reason specs **SHOULD** be created with a relation 1:1 with namespaces. In other words, every namespace will have an AsyncAPI spec including all the assets belonging to that namespace. + +Different granularities **MAY** be chosen depending on the needs.  + +### Meaningful descriptions + +All fields included in the specs **MUST** include a proper description.  + +### Self-contained specs + +All AsyncAPI specs **SHOULD** include as much information as needed in order to make the spec self-contained and clearly documented + +### Contact information + +AsyncAPI specs **MUST** include at least one main contact under the info.contact section. + +The spec only allows to include one contact there, but it **MAY** also include additional contacts using extension fields. For example: + +```yaml +... +info: + ... + contact: + name: "Main point of contact" + email: "team_dl@adidas.com" + x-additional-responsibles: + - person2@adidas.com + - person3@adidas.com + - person4@adidas.com +``` + +### AsyncAPI ID + +According to [AsyncAPI documentation](https://v2.asyncapi.com/docs/reference/specification/v2.6.0#A2SIdString), every AsyncAPI spec **SHOULD** use a unique identifier for the application being defined, following RFC-3986. + +More concretely, ASyncAPI specs created in adidas should use the following pattern + +```yaml +... +id: urn:fdp:adidas:com:namespace:asyncapi_reference_spec +... +``` + + +### Servers + +All AsyncAPI specs **MUST** include a servers section including references to the right Kafka clusters, defined and maintained by FDP team and made available through domains in Swaggerhub. + +Those definitions are handled in Swaggerhub as reusable domains publicly available: + +https://design.api.3stripes.io/domains/adidas/asyncapi_adoption_commons/1.0.0 + +that can be referred from any spec, picking the right kafka servers as required (see example below). + +```yaml +... +servers: + pivotalDev: + $ref: https://design.api.3stripes.io/v1/domains/adidas/asyncapi_adoption_commons/1.0.0#/components/servers/pivotalDev + pivotalSit: + $ref: https://design.api.3stripes.io/v1/domains/adidas/asyncapi_adoption_commons/1.0.0#/components/servers/pivotalSit + pivotalPro: + $ref: https://design.api.3stripes.io/v1/domains/adidas/asyncapi_adoption_commons/1.0.0#/components/servers/pivotalPro +... +``` +**Important note** Don't forget to include '*/v1/*' in the URL of the domain + +### Channels + +All AsyncAPI specs **MUST** include definitions for the channels (kafka topics) including: + +- Description of the topic +- Servers in which the topic is available + - This is a reference to one of the server identifiers included in the servers section +- publish/subscribe operations + - Operation ID + - Summary or short description for the operation + - Description for the operation + - Security schemes + - Tags + - External Docs + - Message details + +In addition to those supported fields, it **MAY** be possible to use extension attributes (using the x- prefix) to specify specific configuration parameters and metadata. In so, the recommended attributes to use are : + +- x-metadata + - To include additional configuration specific to your team or project +- x-configurations + - To include Kafka configuration parameters and producers/consumers + +As the parameters can be different per environment, it is very convenient to add an additional level for the environment + +As part of the publish/subscribe operations, the spec **SHOULD** specify the different kafka clients currently consuming from the different topics for each cluster/environment. For this, the extended attributes x-producers and x-consumers will be used. + +```yaml +... +channels: + namespace.source.event.topic-name: + description: A description of the purpose of the topic and the contained information + servers: ["pivotalDev", "pivotalSit", "pivotalPro"] + x-metadata: + myField1: myValue1 + myField2: myValue2 + x-configurations: + pivotal.dev: + kafka: + partitions: 12 + replicas: 1 + topicConfiguration: + min.insync.replicas: "1" + retention.ms: "2592000000" + pivotal.sit: + kafka: + partitions: 12 + replicas: 2 + topicConfiguration: + min.insync.replicas: "1" + retention.ms: "2592000000" +    publish: + operationId: "producer" + summary: "Description for the operation" + description: "An extensive explanation about the operation" + security: + - producerAcl: [] + tags: + - name: tagA + - name: tagB + x-producers: + pivotal.dev: + - producer1 + - producer2 + pivotal.sit: + - producer1 + - producer2 + pivotal.pro: + - producer3 + - producer4 + externalDocs: + description: documentation + url: http://confluence.adidas.fdp/catalogue/myTopic + ... + subscribe: + operationId: "consumer" + ... + x-consumers: + pivotal.dev: + - consumer1 + - consumer2 + pivotal.sit: + - consumer1 + - consumer2 + pivotal.pro: + - consumer3 +... +``` + + +### Schemas + +Kafka messages **SHOULD** use schemas (AVRO, Json, Protobuf) registered in the Schema Registry to ensure compatibility between producers/consumers. + +If so, always refer to the schema definitions directly in the schema registry instead of duplicating the schema definitions inline. This is to avoid double maintenance.  + +An example directly taken from reference spec is shown below + +```yaml +... +channels: + namespace.source.event.topic-name: + ... + publish: + ... + message: + $ref: '#/components/messages/topic1Payload' +components: + ... + schemas: + ... + topic1SchemaValue: + schemaFormat: 'application/vnd.apache.avro;version=1.9.0' + payload: + $ref: https://pro-fdp-pivotal-schema-registry.api.3stripes.io/subjects/sap_retail_pricing.sap_retail.master.prices-value/versions/latest/schema + messages: + topic1Payload: + $ref: '#/components/schemas/topic1SchemaValue' +``` + +**Important note** To have a reference to a real schema, a schema from sap_retail_pricing in pivotal.pro schema registry was used in the spec reference as an example + +### Security Schemes + +Specs **MAY** use security schemas to reflect the fact that the kafka servers use mTLS. It is something quite static at the moment so the recommendation is reuse the ones specified in the reference spec. + +channels: + namespace.source.event.topic-name: + ... + publish: + ... + security: + - producerAcl: [] + ... +components: + securitySchemes: + ... + consumerAcl: + type: X509 + producerAcl: + type: X509 + +### External docs + +The external docs **SHOULD** be used to refer to LeanIX factsheet associated to the spec. + +```yaml +... +externalDocs: + description: LeanIX + url: https://adidas.leanix.net/adidasProduction/factsheet/Application/467ff391-876c-49ad-93bf-facafffc0178 +``` \ No newline at end of file diff --git a/asynchronous-api-guidelines/03_asyncapi_kafka_specs/c_tooling.md b/asynchronous-api-guidelines/03_asyncapi_kafka_specs/c_tooling.md new file mode 100644 index 0000000..0c3d701 --- /dev/null +++ b/asynchronous-api-guidelines/03_asyncapi_kafka_specs/c_tooling.md @@ -0,0 +1,44 @@ +# adidas Asynchronous API guidelines + +## AsyncAPI tools + +### API Design platform + +The current platform available in adidas to design, host, and render AsyncAPI specs is [Swaggerhub](https://design.api.3stripes.io/). + +Every AsyncAPI spec **MUST** be hosted in Swaggerhub under the *adidas* organization. + +In the future, Fast Data Platform team will provide mechanisms to auto generate your specs as part of the self-service initiative that is ongoing. + +But until then, the specs will be created manually in the platform following the API-first approach if possible. + +**Important note** Swaggerhub has limited capabilities with regards to discoverability, search and filtering of APIs. Other alternatives are being evaluated. Any upcoming decision impacting this will be reflected in this document in the future. + +### Editors + +Aside from Swaggerhub editing capabilities, other alternative editor options are available: + +- AsyncAPI Studio: A web-based editor designed specifically for creating and validating AsyncAPI documents. +- Visual Studio Code: VS Code can be extended with plugins like "AsyncAPI for VS Code" to provide AsyncAPI-specific features,  for editing AsyncAPI files. + +### Command Line Interface (CLI) tool + +Unfortunately, Swaggerhub is not offering a Command Line Interface (CLI) tool which allows including this capability as part of CICD workflows.  + +For this, there is an official AsyncAPI CLI tool which can be checked here: https://www.asyncapi.com/tools/cli. This includes a validator against the AsyncAPI spec, templated generators, version conversion, spec optimizer, bundler, etc. + +For example, to validate a yaml spec file: + +``` +asyncapi validate --file your-asyncapi-file.yaml +``` + +### Generators + +These tools are capable of generate a variety of outputs from any valid AsyncAPI spec, including: + +- API documentation in various formats like HTML, Markdown, or OpenAPI +- Code samples in various programming languages like Python, Java, and Node.js based on your API definition.  +- Functionally complete applications + +There is an official generator tool which can be checked here: https://www.asyncapi.com/docs/tools/generator. \ No newline at end of file diff --git a/asynchronous-api-guidelines/asyncapi.md b/asynchronous-api-guidelines/asyncapi.md deleted file mode 100644 index fbb29eb..0000000 --- a/asynchronous-api-guidelines/asyncapi.md +++ /dev/null @@ -1,6 +0,0 @@ -# Introduction - -## adidas Asynchronous APIs Guidelines - - -