Spec Repo

Note

This is documentation for apigentools major version 1, which has significant differences from the 0.X series. Refer to upgrading docs for further details.

A "spec repo" is a repository (or, more generally, a directory), that contains files necessary for apigentools to generate your client code. The recommended layout is as follows:

.
├── .gitignore
├── config                          # config directory is mandatory
│   ├── config.yaml                 # general config for apigentools, mandatory
│   └── languages                   # languages directory is mandatory when using openapi-generator
│       └── java_v1.json            # openapi-generator config for Java client for v1 API
├── downstream-templates            # optional directory for downstream templates, name can be arbitrary
│   └── java
│       └── LICENSE                 # a file/template to add to generated client
├── generated                       # generated code will end up in this directory, mandatory
│   └── .gitkeep
├── spec                            # directory with all OpenAPI spec files, mandatory
│   └── v1                          # directory with spec of v1 of the API
│       ├── accounts.yaml           # example: spec of the accounts API
│       ├── header.yaml             # header of the OpenAPI spec
│       └── shared.yaml             # entities shared between multiple parts of the OpenAPI spec
├── template-patches                # optional directory for template patches, name can be arbitrary
│   └── java-01-minor-change.patch  # a patch to apply to openapi-generator template
└── templates                       # openapi-generator templates with applied patches end up here, mandatory
    └── .gitkeep

You can create a basic scaffold of a spec repo using the init subcommand. For example:

apigentools init myspecrepo

Most of the paths in this layout can be overriden with command line arguments passed to the apigentools executable.

Overview of spec repo contents

.gitignore

This is a standard .gitignore file.

config/

This is a directory containing config.yaml, which is a configuration file for apigentools.

config/languages/

This directory contains openapi-generator configuration files. There must be one file for every language and major API version configuration in your config/config.yaml. For information on what keys can go in these files, use openapi-generator config-help -g <LANGUAGE>.

downstream-templates/

This directory contains downstream templates, if used (i.e. templates that are provided by the user, rather than openapi-generator).

generated/

This directory contains code generated by apigentools. It is usually empty when you clone this repository, as the generated clients are supposed to have their own repositories and be gitignored from the spec repo.

spec/

This directory contains per-major-API-version OpenAPI specifications of your API. For example, if your API has a v1 and v2, the spec directory would have v1 and v2 subdirectories. Each of these would contain at least header.yaml and shared.yaml "section files". Refer to section files reference for more information.

template-patches/

This directory contains your downstream template patches, if you decide to use these. These are applied to openapi-generator templates before the generation process using the apigentools templates command, thus allowing you to customize the upstream templates.

templates/

This directory contains processed upstream templates, i.e. IOW templates from openapi-generator with your template patches applied on top. These are usually gitignored from the spec repo; to process templates without running the whole code generation, use the apigentools templates command.

File Formats

This document serves as a reference of various file formats used by apigentools.

config/config.yaml

This is a configuration file for apigentools. It must be present in the spec repo for apigentools to work.

Example:

container_opts:
  image: datadog/apigentools:latest
minimum_apigentools_version: 1.0.0
languages:
  java:
    generation:
      default:
        templates:
          patches:
            - template-patches/java-0001-custom-license-header.patch
          source:
            type: openapi-jar
            jar_path: /usr/bin/openapi-generator-cli.jar
            templates_dir: Java
        commands:
          - commandline:
            - rm
            - -rf
            - src/main/java/com/datadog/api/{{spec_version}}
            description: Remove old generated files
            - commandline:
              - function: openapi_generator_generate # expands to the default generation command
              - --additional-properties
              - java8=true,dateLibrary=java8
              description: Generate code using openapi-generator
          - commandline:
            - rm
            - -rf
            - docs
            description: Remove unwanted docs folder
        tests:
        - commandline:
          - mvn
          - test
          description: "Run tests using maven"
    downstream_templates:
      downstream-templates/java/README.md: README.md
    github_org_name: my-github-org
    github_repo_name: my-java-client
    library_version: "0.0.1"
    spec_versions: ["v1", "v2"]
    version_path_template: "myapi_{{spec_version}}"
spec_sections":
  v1: ["accounts.yaml", "users.yaml"]
  v2: ["users.yaml"]
spec_versions: ["v1", "v2"]
user_agent_client_name: "MyCompany"
validation_commands:
- commandline:
    - openapi-generator
    - validate
    - -i
    - "{{full_spec_path}}"
  description: "Validate full spec using openapi-generator"

The structure of the general config file is as follows, starting with top level keys:

  • config_version (Added in config version 1.2). - Version of the configuration file.
  • container_opts - Options for containerized command execution. See container_opts section below.
  • minimum_apigentools_version - Minimum required version of the apigentools package required for this spec repository.
  • languages - Settings for individual languages; contains a mapping of language names to their settings.
    • container_opts - See container_opts section below.
    • individual-language-settings:
      • container_opts - See container_opts section below.
      • generation - Setting for code generation.
        • default - Default settings for code generations. When any of the keys defined in the default mapping are not found in the spec-version-settings, it's taken from here.
          • container_opts - See container_opts section below.
          • commands - Commands to execute to generate code. See commands.
          • templates - Instructions on how to preprocess templates to pass to code generator.
          • tests - Commands to execute to test code using apigentools test. See commands.
          • validation_commands (Added in config version 1.2) - Same as top-level and per-language validation_commands, allows overriding both of these.
        • spec-version-settings - Exactly the same as default above, but used for overriding operations specifically for the given major API version.
      • downstream_templates - Downstream templates.
      • github_org_name - Name of the Github organization of the client for this language.
      • github_repo_name - Name of the Github repository of the client for this language.
      • library_version - Version of the generated library, for now this only serves as a variable useful in command templating
      • spec_sections - Same as top-level spec_sections. Use to override the subset of spec sections to generate for each spec version of this language. For every spec version not specified as a key, the top-level list of sections for this spec version is used.
      • spec_versions - Same as top-level spec_versions. Use to override the subset of major versions to generate for this language. If not specified, the top-level spec_versions value is used.
      • validation_commands (Added in config version 1.2) - Same as top-level validation_commands, allows overriding the top-level value on per-language basis.
      • version_path_template - Mustache template for the name of the subdirectory in the Github repo where code for individual major versions of the API will end up, e.g. with myapi_{{spec_version}} as value and a github_repo_name of value my-java-client, the code for v1 of the API will end up in myapi-java-client/myapi_v1
  • spec_sections - Mapping of major spec versions (these must be in spec_versions) to lists of files with paths/tags/components definitions to merge when creating full spec. Files not explicitly listed here are ignored.
  • spec_versions - List of major versions currently known and operated on. These must be subdirectories of the spec directory.
  • user_agent_client_name - The HTTP User Agent string will be set to {user_agent_client_name}/{package_version}/{language}.
  • validation_commands - List of commands to run during validation phase. The commands have the same structure and mechanics like language commands (see above). Validation commands will be executed for each full spec that is created.

config_version

In apigentools 1.2.0, the concept of config_version was introduced. Since this version, apigentools will verify that the config it's running against is of a supported version.

config_version adopts a MAJOR.MINOR versioning scheme. Configuration will only change in minor apigentools releases, so an X.Y.Z version of apigentools always supports config_version: "X.Y" and perhaps some older versions as well.

Configs created for apigentools between 1.0.0 and 1.2.0 (as well as these which don't have config_version specified) are considered to be of version 1.0.

container_opts

Since apigentools major version 1, all commands are expected to be executed inside containers by default. In order to achieve this a container_opts structure can be provided on different levels of the configuration file to modify how containers are executed.

The container_opts defined at various levels have certain inheritance set up. To compute resulting container_opts for a command, the container_opts are taken and considered in the following order:

  • container_opts defined for the command
  • container_opts defined for the spec version that the command is being run
  • container_opts for the default generation
  • container_opts for the language
  • container_opts for the top level

If at any point an inherit: False value is reached, the inheritance process stops. The inheritance is mostly intuitive, for example if environment value FOO is defined for the command and for the language, the one for the command is used.

Following container_opts are recognized:

  • environment - Environment variables to pass to the container. These are merged while inheriting, so different environment variables from different levels will all be used.
  • image - What image to use to run the command. This is the only option that's not stopped by inheritance. If a command has inherit: False but no image defined, the image value is still inherited. There is always a default global value datadog/apigentools:latest unless specified otherwise.
  • inherit - Stop the inheritance at this point (if False; the default is True).
  • no_container - Run this command outside of container (recommended for development/debugging purposes only).

Preprocess Templates

Template preprocessing is a key feature of apigentools, as it allows for downstream modification of templates used for code generation. Preprocessing templates is a two-step operation:

  • First, upstream templates are obtained (by default, these are obtained from the container specified by container_opts for given spec version; container_opts inheritance applies).
  • Second, template patches are applied (optional).

Based on the above, the templates step contains two keys:

  • patches - List of paths to patches (inside the Spec Repo) to apply to templates
  • source - Source for the upstream templates; recognized configurations follow in subsections bellow. Additionally, following configuration keys are recognized:
    • no_container - Don't obtain the templates from container, but from local host.

openapi-jar

Extract templates from openapi-generator JAR file.

templates:
  source:
    type: openapi-jar
    jar_path: /usr/bin/openapi-generator-cli.jar # path to the openapi generator JAR file
    templates_dir: Java # directory with templates for this language

openapi-git

Extract templates from openapi-generator Github repository.

templates:
  source:
    type: openapi-git
    git_committish: "v4.3.0" # git committish to checkout before extracting the templates
    templates_dir: Java # directory with templates for this language

directory

Extract templates from a directory on filesystem.

temlates:
  source:
    type: directory
    directory_path: /path/to/templates
    templates_dir: Java # directory with templates for this language

Commands

Commands provided inside generated.default.commands and generated.<spec_version>.commands are executed one by one inside the directory with code for currently generated spec version.

Commands provided inside generated.default.tests and generated.<spec_version>.tests are executed one by one inside the directory with code for currently tested spec version.

Each command has following attributes:

  • description - A description of the command.
  • commandline - The command itself; the items in the list are:
    • templatable strings and, potentially
    • functions that represent callbacks to the Python code. Each function is represented as a map:
      • function - Name of the function (see below for list of recognized functions).
      • args - List of args to pass to the function in Python code (as in *args).
      • kwargs - Mapping of args to pass to the function in Python code (as in **kwargs).
      • The result of the function call is then used in the actual command line call (note that the functions are called by apigentools outside the container that actually executes the command).
  • container_opts - container_opts describing how to execute the command inside a container. Do note that commands are executed in non-interactive containers.

Validation commands

Validation commands have exactly the same structure as the above commands. They're run for every created full OpenAPI spec (since there might be more of these based on how different languages define their spec_sections).

New in 1.2.0: validation_commands can be specified on multiple levels, with inheritance working like this: For a specific language version, validation_commands are taken from:

  • validation_commands definition for the specific version (if present there); else
  • validation_commands definition in the default section (if present there); else
  • validation_commands definition for the specific language (if present there); else
  • validation_commands definition from the top level (if present there); else
  • empty command list is used

The only templating variable these commands can use is {{full_spec_path}}, which is a path to the spec currently being validated.

Functions in Commands

This section lists recognized functions for language phase and validation commands, as mentioned in the section above.

  • glob - runs Python's glob.glob
  • glob_re - runs Python's glob.glob and then filters its results by calling re.match using given Python regular expression

When used as this: json { "function": "glob_re", "args": [ "*.go", ".*(?<!_test.go)$" ] } It will match all files ending with .go except files that end with _test.go.

  • openapi_generator_generate - expands to

    - "openapi-generator" - "generate" - "--http-user-agent" - "{{user_agent_client_name}}/{{library_version}}/{{language_name}}" - "-g" - "{{language_name}}" - "-c" - "{{language_config}}" - "-i" - "{{full_spec_path}}" - "-o" - "{{version_output_dir}}" - "--additional-properties" - "apigentoolsStamp='{{stamp}}'"

    If the spec version that the default command is being run for is using a templates section, the commandline is further appended with:

    - -t - "{{templates_dir}}"

Templating Commands

The commandline arguments of commands can also use following templating values (Mustache templating is used):

  • {{cwd}} - Current working directory
  • {{full_spec_path}} - Path to the input full OpenAPI spec
  • {{github_repo_name}} - Value from config
  • {{github_repo_org}} - Value from config
  • {{language_config}} - Path to the proper config file under config/languages/ directory
  • {{language_name}} - Name of the language from the config
  • {{library_version}} - Value from config
  • {{spec_version}} - Version of the spec currently being processed
  • {{stamp}} - Same as apigentoolsStamp mentioned in reproducibility
  • {{templates_dir}} - Directory with templates for this language/API version combination
  • {{user_agent_client_name}} - Value from config
  • {{version_output_dir}} - Directory to which the output for this spec version will be put
  • {{top_level_dir}} - Relative path (to {{version_output_dir}}) to the top level directory of the generated language repo

Downstream Templates

downstream_templates are a mapping of additional templates (relative to the Spec Repo directory) to files they'll be rendered as (relative the top level directory of the generated library).

Downstream templates are only generated once per language and have access to a subset of templating values available for commands:

  • {{full_spec_path}}
  • {{github_repo_name}}
  • {{github_repo_org}}
  • {{language_name}}
  • {{library_version}}
  • {{stamp}}
  • {{user_agent_client_name}}

Section Files

By design, apigentools doesn't operate on full OpenAPI specification files, but on "section files". These are, essentially, an exploded OpenAPI spec—these aid the development experience when working on these individual files instead of one huge file. Note that these files have to be listed explicitly in spec_sections.

On apigentools' invocation, these are merged into a single OpenAPI spec file and used as input for further operations, e.g. for openapi-generator.

We recommend having a header.yaml file to for your openapi, info and servers attributes and a shared.yaml file for entities shared among multiple spec sections.

pre-commit Hooks

You can make sure that all contributors create syntactically valid specification that also follow all rules from config.yaml by creating a pre-commit check. Here is an example, to add in your .pre-commit-config.yaml file:

---
# Update the rev variable with the release version that you want, from the apigentools repo.
- repo: https://github.com/DataDog/apigentools.git
  rev: v0.10.0
  hooks:
    - id: validate