diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/README.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/README.md index 2797772..53af7f9 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/README.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/README.md @@ -1,6 +1,6 @@ # Kafka Asynchronous Guidelines -There are several technologies to implement event-driven architectures, but this section is going to focus on the predominant technology on this subject : Apache Kafka. +There are several technologies to implement event-driven architectures, but this section is going to focus on the predominant technology on this subject: Apache Kafka. **Apache Kafka** can be considered as a Streaming Platform which relies on the several concepts: diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/a_introduction/README.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/a_introduction/README.md index 768093e..ab87116 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/a_introduction/README.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/a_introduction/README.md @@ -2,7 +2,7 @@ This section is specific to the definition of API specs with [AsyncAPI](https://www.asyncapi.com/) for Kafka protocol. -Also, take into account that across the section there will be multiple references to this [AsyncAPI reference specification](https://design.api.3stripes.io/apis/adidas/asyncapi-adoption-initiative/1.0.0) which is publicly available for reference. +Also, take into account that across the section there will be multiple references to this [AsyncAPI reference specification](https://design.api.3stripes.io/apis/adidas/asyncapi-adoption-initiative/1.0.0) which is publicly available for reference. #### Kafka to AsyncAPI concept mapping diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/a_introduction/why-asyncapi.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/a_introduction/why-asyncapi.md index 3ca169a..ed95d27 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/a_introduction/why-asyncapi.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/a_introduction/why-asyncapi.md @@ -7,24 +7,24 @@ It supports various messaging protocols, including MQTT, Web Socket, Kafka, AMQP The benefits of using AsyncAPI are, amongst others: * Standardization - * AsyncAPI defines a STANDARD format (YAML or JSON) for describing asynchronous APIs. - * By defining the structure of messages, channels, and events, you can ensure that all components adhere to the same conventions. - * Using a single standard ensures consistency in the design and documentation of all your asynchronous APIs. - * This simplifies integration, maintenance, and troubleshooting across different parts of your system. + * AsyncAPI defines a STANDARD format (YAML or JSON) for describing asynchronous APIs + * By defining the structure of messages, channels, and events, you can ensure that all components adhere to the same conventions + * Using a single standard ensures consistency in the design and documentation of all your asynchronous APIs + * This simplifies integration, maintenance, and troubleshooting across different parts of your system * Improved Developer Experience - * AsyncAPI documents the messages being exchanged, their structure, and the events triggered by them. - * It provides developers with a clear picture of how to interact with the API, what data to expect, and how to interpret responses without digging into the implementation details. + * AsyncAPI documents the messages being exchanged, their structure, and the events triggered by them + * It provides developers with a clear picture of how to interact with the API, what data to expect, and how to interpret responses without digging into the implementation details * Code scaffolding - * Using tools like asyncapi-generator allow to easily generate the skeleton of applications that can work with the resources described in the spec. - * This can be done in different programming languages (Python, Java, Node.js. ...), reducing significantly the development time and the coding errors. -* Design-first approach: It encourages designing the API first before writing code, leading to better planned and more reliable APIs. + * Using tools like asyncapi-generator allow to easily generate the skeleton of applications that can work with the resources described in the spec + * This can be done in different programming languages (Python, Java, Node.js. ...), reducing significantly the development time and the coding errors +* Design-first approach: It encourages designing the API first before writing code, leading to better planned and more reliable APIs In addition to those benefits, Platform & Engineering is working hard to create a data catalogue built upon AsyncAPI that allows to have a good level of discoverability, allowing teams to be able to find exactly the data they need with regards to any data object in the company. Questions like: -* Who is responsible for a specific data object -* Where is that data hosted -* Which kind of information is available +* Who is responsible for a specific data object? +* Where is that data hosted? +* Which kind of information is available? Will be easy to answer once this catalogue is in place. Also, it is important to have a good discoverability and search & filtering capabilities. diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/b_asyncapi_version.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/b_asyncapi_version.md index 5c9ff7b..4b23a53 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/b_asyncapi_version.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/b_asyncapi_version.md @@ -5,6 +5,6 @@ Any version of AsyncAPI **MAY** be used for spec definitions. However, to be aligned with adidas tooling, spec versions **SHOULD** be _v2.6.0_, because to the date of this document creation (April 2024) this is the highest supported version on Swaggerhub, the current API portal to render, discover and publish specs. ```yaml -asyncapi: 2.6.0 +asyncapi: '2.6.0' ... ``` diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/c_internal_public_specs.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/c_internal_public_specs.md index 442bc2e..6148270 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/c_internal_public_specs.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/c_internal_public_specs.md @@ -1,6 +1,6 @@ # Internal vs Public Specifications -AsyncAPI specs **MAY** be created both for public APIs or for internal APIs. +AsyncAPI specs **MAY** be created both for public APIs or for internal APIs: * Public APIs are those who are created to be consumed by others * Internal APIs are only for development teams for a particular project diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_schema_data_evolution/README.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_schema_data_evolution/README.md index a3c5938..859a809 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_schema_data_evolution/README.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_schema_data_evolution/README.md @@ -17,9 +17,9 @@ As a summary, the available compatibility types are listed below: | Mode | Description | | -------------------- | --------------------------------------------------------------------------- | | BACKWARD | new schema versions are backward compatible with older versions | -| BACKWARD\_TRANSITIVE | backward compatibility across all schema versions, not just the latest one. | +| BACKWARD\_TRANSITIVE | backward compatibility across all schema versions, not just the latest one | | FORWARD | new schema versions are compatible with older consumer versions | -| FORWARD\_TRANSITIVE | forward compatibility across all schema versions. | +| FORWARD\_TRANSITIVE | forward compatibility across all schema versions | | FULL | both backward and forward compatibility with the latest schema version | | FULL\_TRANSITIVE | both backward and forward compatibility with all schema versions | | NONE | schema compatibility checks are disabled | @@ -28,7 +28,7 @@ As a summary, the available compatibility types are listed below: #### Upgrading process of clients based on compatibility -Depending on the compatibility mode, the process of upgrading producers/consumers will be different based on the compatibility mode enabled. +Depending on the compatibility mode, the process of upgrading producers/consumers will be different based on the compatibility mode enabled: * NONE * As there are no compatibility checks, no order will grant a smooth transition diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_schema_data_evolution/full-compatibility.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_schema_data_evolution/full-compatibility.md index 844c792..caaa907 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_schema_data_evolution/full-compatibility.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_schema_data_evolution/full-compatibility.md @@ -2,7 +2,7 @@ This is a combination of both compatibility types (backward and forward). It also has 2 variants: -* FULL - Backward and forward compatible between schemas X and X - 1. +* FULL - Backward and forward compatible between schemas X and X - 1 * FULL\_TRANSITIVE - Backward and forward compatible between schemas X and all previous ones (X - 1, X - 2, ...) **Important** Once more, FULL\_TRANSITIVE is the default compatibility mode in adidas, it is set at cluster level and all new schemas will inherit it diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_self_contained_specs/e_meaningful_descriptions.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_self_contained_specs/e_meaningful_descriptions.md index 4ab2272..e3c3162 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_self_contained_specs/e_meaningful_descriptions.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/f_self_contained_specs/e_meaningful_descriptions.md @@ -1,3 +1,3 @@ # Meaningful Descriptions -AsyncAPI specs **MUST** make use of the _description_ fields, including there meaningful information to understand the purpose of the different elements. +AsyncAPI specs **MUST** make use of the _description_ fields, including there meaningful information to understand the purpose of the different elements. diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/g_key_value_format.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/g_key_value_format.md index fc81f83..2d82725 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/g_key_value_format.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/g_key_value_format.md @@ -2,6 +2,6 @@ Kafka messages **MAY** include a key, which needs to be properly designed to have a good balance of data across partitions. -The message key and the payload (often called value) can be serialized independently and can have different formats. For example, the value of the message can be sent in AVRO format, while the message key can be a primitive type (string). +The message key and the payload (often called value) can be serialized independently and can have different formats. For example, the value of the message can be sent in AVRO format, while the message key can be a primitive type (string). Message keys **SHOULD** be kept as simple as possible and use a primitive type when possible. diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/h_asyncapi_id.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/h_asyncapi_id.md index 1e677bd..d4ad7fb 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/h_asyncapi_id.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/h_asyncapi_id.md @@ -2,10 +2,10 @@ According to [AsyncAPI documentation](https://v2.asyncapi.com/docs/reference/specification/v2.6.0#A2SIdString), every AsyncAPI spec **SHOULD** use a unique identifier for the application being defined, following RFC-3986. -More concretely, ASyncAPI specs created in adidas should use the following pattern +More concretely, ASyncAPI specs created in adidas should use the following pattern: ```yaml ... -id: urn:fdp:adidas:com:namespace:asyncapi_reference_spec +id: 'urn:fdp:adidas:com:namespace:asyncapi_reference_spec' ... ``` diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/i_servers.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/i_servers.md index f70843d..ac7b165 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/i_servers.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/i_servers.md @@ -4,20 +4,20 @@ All AsyncAPI specs **MUST** include a _servers_ section including references to For example, in Swaggerhub it is possible to refer to those reusable definitions through: -[https://design.api.3stripes.io/domains/adidas/asyncapi\_adoption\_commons/1.0.0](https://design.api.3stripes.io/domains/adidas/asyncapi\_adoption\_commons/1.0.0) +[https://design.api.3stripes.io/domains/adidas/cluster-landscape/1.0.0](https://design.api.3stripes.io/domains/adidas/cluster-landscape/1.0.0) more specifically, it is possible to add the right Kafka servers like this: ```yaml ... servers: - pivotalDev: - $ref: https://design.api.3stripes.io/v1/domains/adidas/asyncapi_adoption_commons/1.0.0#/components/servers/pivotalDev - pivotalSit: - $ref: https://design.api.3stripes.io/v1/domains/adidas/asyncapi_adoption_commons/1.0.0#/components/servers/pivotalSit - pivotalPro: - $ref: https://design.api.3stripes.io/v1/domains/adidas/asyncapi_adoption_commons/1.0.0#/components/servers/pivotalPro + playground-dev: + $ref: 'https://design.api.3stripes.io/v1/domains/adidas/cluster-landscape/1.0.0#/components/servers/playground-dev' + playground-sit: + $ref: 'https://design.api.3stripes.io/v1/domains/adidas/cluster-landscape/1.0.0#/components/servers/playground-sit' + playground-pro: + $ref: 'https://design.api.3stripes.io/v1/domains/adidas/cluster-landscape/1.0.0#/components/servers/playground-pro' ... ``` -**Important note** Don't forget to include '_/v1/_' in the URL of the domain +**Important note** Don't forget to include '_/v1/_' in the URL of the domain. diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/j_channels.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/j_channels.md index 459dff8..9e5fc0b 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/j_channels.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/j_channels.md @@ -14,14 +14,14 @@ All AsyncAPI specs **MUST** include definitions for the channels (Kafka topics) * External Docs * Message details -In addition to those supported fields, it **MAY** be possible to use extension attributes (using the x- prefix) to specify specific configuration parameters and metadata. In so, the recommended attributes to use are : +In addition to those supported fields, it **MAY** be possible to use extension attributes (using the x- prefix) to specify specific configuration parameters and metadata. In so, the recommended attributes to use are: * x-metadata * To include additional configuration specific to your team or project * x-configurations * To include Kafka configuration parameters and producers/consumers -As the parameters can be different per environment, it is very convenient to add an additional level for the environment +As the parameters can be different per environment, it is very convenient to add an additional level for the environment. As part of the publish/subscribe operations, the spec **SHOULD** specify the different Kafka clients currently consuming from the different topics for each cluster/environment. For this, the extended attributes x-producers and x-consumers will be used. @@ -29,60 +29,60 @@ As part of the publish/subscribe operations, the spec **SHOULD** specify the dif ... channels: namespace.source.event.topic-name: - description: A description of the purpose of the topic and the contained information - servers: ["pivotalDev", "pivotalSit", "pivotalPro"] + description: 'A description of the purpose of the topic and the contained information' + servers: ['playground-dev', 'playground-sit', 'playground-pro'] x-metadata: - myField1: myValue1 - myField2: myValue2 + myField1: 'myValue1' + myField2: 'myValue2' x-configurations: - pivotal.dev: + playground.dev: kafka: - partitions: 12 - replicas: 1 + partitions: '12' + replicas: '1' topicConfiguration: - min.insync.replicas: "1" - retention.ms: "2592000000" - pivotal.sit: + min.insync.replicas: '1' + retention.ms: '2592000000' + playground.sit: kafka: - partitions: 12 - replicas: 2 + partitions: '12' + replicas: '2' topicConfiguration: - min.insync.replicas: "1" - retention.ms: "2592000000" + min.insync.replicas: '1' + retention.ms: '2592000000'     publish: - operationId: "producer" - summary: "Description for the operation" - description: "An extensive explanation about the operation" + operationId: 'producer' + summary: 'Description for the operation' + description: 'An extensive explanation about the operation' security: - producerAcl: [] tags: - - name: tagA - - name: tagB + - name: 'tagA' + - name: 'tagB' x-producers: - pivotal.dev: - - producer1 - - producer2 - pivotal.sit: - - producer1 - - producer2 - pivotal.pro: - - producer3 - - producer4 + playground.dev: + - 'producer1' + - 'producer2' + playground.sit: + - 'producer1' + - 'producer2' + playground.pro: + - 'producer3' + - 'producer4' externalDocs: - description: documentation - url: http://confluence.adidas.fdp/catalogue/myTopic - ... + description: 'documentation' + url: 'http://confluence.adidas.fdp/catalogue/myTopic' + ... subscribe: - operationId: "consumer" - ... + operationId: 'consumer' + ... x-consumers: - pivotal.dev: - - consumer1 - - consumer2 - pivotal.sit: - - consumer1 - - consumer2 - pivotal.pro: - - consumer3 + playground.dev: + - 'consumer1' + - 'consumer2' + playground.sit: + - 'consumer1' + - 'consumer2' + playground.pro: + - 'consumer3' ... ``` diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/k_schemas.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/k_schemas.md index 7348b44..3ab6e8f 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/k_schemas.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/k_schemas.md @@ -2,9 +2,9 @@ Kafka messages **SHOULD** use schemas (AVRO, JSON, Protobuf) registered in the Schema Registry to ensure compatibility between producers/consumers. -If so, always refer to the schema definitions directly in the schema registry instead of duplicating the schema definitions inline. This is to avoid double maintenance. +If so, always refer to the schema definitions directly in the schema registry instead of duplicating the schema definitions inline. This is to avoid double maintenance. -An example directly taken from reference spec is shown below +An example directly taken from reference spec is shown below. ```yaml ... @@ -14,18 +14,24 @@ channels: publish: ... message: - $ref: '#/components/messages/topic1Payload' + $ref: '#/components/messages/namespace.source.event.topic-name' + components: ... - schemas: - ... - topic1SchemaValue: - schemaFormat: 'application/vnd.apache.avro;version=1.9.0' - payload: - $ref: https://sit-fdp-pivotal-schema-registry.api.3stripes.io/subjects/pea_fd_fdp.sample.test-value/versions/latest/schema messages: - topic1Payload: - $ref: '#/components/schemas/topic1SchemaValue' + namespace.source.event.topic-name: + $ref: '#/components/schemas/namespace.source.event.topic-name' + ... + schemas: + namespace.source.event.topic-name: + description: 'Schema[s] retrieved from Schema Registry' + schemaFormat: 'application/vnd.apache.avro;version=1.9.0' + payload: + $ref: 'https://dev-fdp-playground-schema-registry.api.3stripes.io/subjects/namespace.source.event.topic-name-value/versions/latest/schema' + bindings: + kafka: + key: + $ref: 'https://dev-fdp-playground-schema-registry.api.3stripes.io/subjects/namespace.source.event.topic-name-key/versions/latest/schema' ``` -**Important note** The used schema is a very simple one, it is only used to illustrate how to refer to it. +**Important note** The used schema is a very simple one, it is only used to illustrate how to refer to it. In case the message doesn't use schema for the key, _bindings_ field can be omitted from the schema. diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/l_security_schemes.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/l_security_schemes.md index 120419f..70b6328 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/l_security_schemes.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/l_security_schemes.md @@ -1,6 +1,8 @@ # Security Schemes -Specs **MAY** use security schemas to reflect the fact that the Kafka servers use mTLS. It is something quite static at the moment so the recommendation is reuse the ones specified in the reference spec. +Specs **MAY** use security schemas to reflect the fact that the Kafka servers use mTLS or SASL. It is something quite static at the moment so the recommendation is refer to reusable definitions. + +Below example includes references to other security schemas used by the Kafka servers. ```yaml channels: @@ -15,7 +17,15 @@ components: securitySchemes: ... consumerAcl: - type: X509 + $ref: 'https://design.api.3stripes.io/v1/domains/adidas/cluster-landscape/1.0.0#/components/securitySchemes/consumerAcl' producerAcl: - type: X509 + $ref: 'https://design.api.3stripes.io/v1/domains/adidas/cluster-landscape/1.0.0#/components/securitySchemes/producerAcl' + mtlsV1Server: + $ref: 'https://design.api.3stripes.io/v1/domains/adidas/cluster-landscape/1.0.0#/components/securitySchemes/mtlsV1Server' + mtlsV2Server: + $ref: 'https://design.api.3stripes.io/v1/domains/adidas/cluster-landscape/1.0.0#/components/securitySchemes/mtlsV2Server' + saslV1Server: + $ref: 'https://design.api.3stripes.io/v1/domains/adidas/cluster-landscape/1.0.0#/components/securitySchemes/saslV1Server' + saslV2Server: + $ref: 'https://design.api.3stripes.io/v1/domains/adidas/cluster-landscape/1.0.0#/components/securitySchemes/saslV2Server' ``` diff --git a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/m_external_docs.md b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/m_external_docs.md index 000d892..abecf4b 100644 --- a/asynchronous-api-guidelines/kafka-asynchronous-guidelines/m_external_docs.md +++ b/asynchronous-api-guidelines/kafka-asynchronous-guidelines/m_external_docs.md @@ -5,6 +5,6 @@ The external docs **SHOULD** be used to refer to LeanIX fact-sheet associated to ```yaml ... externalDocs: - description: LeanIX - url: https://adidas.leanix.net/adidasProduction/factsheet/Application/467ff391-876c-49ad-93bf-facafffc0178 + description: 'LeanIX' + url: 'https://adidas.leanix.net/adidasProduction/factsheet/Application/467ff391-876c-49ad-93bf-facafffc0178' ```