Spec Repo

Note

This is documentation for apigentools major version 1, which has significant differences from the 0.X series. Refer to upgrading docs for further details.

A "spec repo" is a repository (or, more generally, a directory), that contains files necessary for apigentools to generate your client code. The recommended layout is as follows:

.
├── .gitignore
├── config                          # config directory is mandatory
│   ├── config.yaml                 # general config for apigentools, mandatory
│   └── languages                   # languages directory is mandatory when using openapi-generator
│       └── java_v1.json            # openapi-generator config for Java client for v1 API
├── downstream-templates            # optional directory for downstream templates, name can be arbitrary
│   └── java
│       └── LICENSE                 # a file/template to add to generated client
├── generated                       # generated code will end up in this directory, mandatory
│   └── .gitkeep
├── spec                            # directory with all OpenAPI spec files, mandatory
│   └── v1                          # directory with spec of v1 of the API
│       ├── accounts.yaml           # example: spec of the accounts API
│       ├── header.yaml             # header of the OpenAPI spec
│       └── shared.yaml             # entities shared between multiple parts of the OpenAPI spec
├── template-patches                # optional directory for template patches, name can be arbitrary
│   └── java-01-minor-change.patch  # a patch to apply to openapi-generator template
└── templates                       # openapi-generator templates with applied patches end up here, mandatory
    └── .gitkeep

You can create a basic scaffold of a spec repo using the init subcommand. For example:

apigentools init myspecrepo

Most of the paths in this layout can be overriden with command line arguments passed to the apigentools executable.

Overview of spec repo contents

.gitignore

This is a standard .gitignore file.

config/

This is a directory containing config.yaml, which is a configuration file for apigentools.

config/languages/

This directory contains openapi-generator configuration files. There must be one file for every language and major API version configuration in your config/config.yaml. For information on what keys can go in these files, use openapi-generator config-help -g <LANGUAGE>.

downstream-templates/

This directory contains downstream templates, if used (i.e. templates that are provided by the user, rather than openapi-generator).

generated/

This directory contains code generated by apigentools. It is usually empty when you clone this repository, as the generated clients are supposed to have their own repositories and be gitignored from the spec repo.

spec/

This directory contains per-major-API-version OpenAPI specifications of your API. For example, if your API has a v1 and v2, the spec directory would have v1 and v2 subdirectories. Each of these would contain at least header.yaml and shared.yaml "section files". Refer to section files reference for more information.

template-patches/

This directory contains your downstream template patches, if you decide to use these. These are applied to openapi-generator templates before the generation process using the apigentools templates command, thus allowing you to customize the upstream templates.

templates/

This directory contains processed upstream templates, i.e. IOW templates from openapi-generator with your template patches applied on top. These are usually gitignored from the spec repo; to process templates without running the whole code generation, use the apigentools templates command.

File Formats

This document serves as a reference of various file formats used by apigentools.

config/config.yaml

This is a configuration file for apigentools. It must be present in the spec repo for apigentools to work.

Example:

container_opts:
  image: datadog/apigentools:latest
minimum_apigentools_version: 1.0.0
languages:
  java:
    generation:
      default:
        templates:
          patches:
            - template-patches/java-0001-custom-license-header.patch
          source:
            type: openapi-jar
            jar_path: /usr/bin/openapi-generator-cli.jar
            templates_dir: Java
        commands:
          - commandline:
            - rm
            - -rf
            - src/main/java/com/datadog/api/{{spec_version}}
            description: Remove old generated files
            - commandline:
              - function: openapi_generator_generate # expands to the default generation command
              - --additional-properties
              - java8=true,dateLibrary=java8
              description: Generate code using openapi-generator
          - commandline:
            - rm
            - -rf
            - docs
            description: Remove unwanted docs folder
        tests:
        - commandline:
          - mvn
          - test
          description: "Run tests using maven"
    downstream_templates:
      downstream-templates/java/README.md: README.md
    github_org_name: my-github-org
    github_repo_name: my-java-client
    library_version: "0.0.1"
    spec_versions: ["v1", "v2"]
    version_path_template: "myapi_{{spec_version}}"
spec_sections":
  v1: ["accounts.yaml", "users.yaml"]
  v2: ["users.yaml"]
spec_versions: ["v1", "v2"]
user_agent_client_name: "MyCompany"
validation_commands:
- commandline:
    - openapi-generator
    - validate
    - -i
    - "{{full_spec_path}}"
  description: "Validate full spec using openapi-generator"

The structure of the general config file is as follows, starting with top level keys:

container_opts

Since apigentools major version 1, all commands are expected to be executed inside containers by default. In order to achieve this a container_opts structure can be provided on different levels of the configuration file to modify how containers are executed.

The container_opts defined at various levels have certain inheritance set up. To compute resulting container_opts for a command, the container_opts are taken and considered in the following order:

If at any point an inherit: False value is reached, the inheritance process stops. The inheritance is mostly intuitive, for example if environment value FOO is defined for the command and for the language, the one for the command is used.

Following container_opts are recognized:

Preprocess Templates

Template preprocessing is a key feature of apigentools, as it allows for downstream modification of templates used for code generation. Preprocessing templates is a two-step operation:

Based on the above, the templates step contains two keys:

openapi-jar

Extract templates from openapi-generator JAR file.

templates:
  source:
    type: openapi-jar
    jar_path: /usr/bin/openapi-generator-cli.jar # path to the openapi generator JAR file
    templates_dir: Java # directory with templates for this language

openapi-git

Extract templates from openapi-generator Github repository.

templates:
  source:
    type: openapi-git
    git_committish: "v4.3.0" # git committish to checkout before extracting the templates
    templates_dir: Java # directory with templates for this language

directory

Extract templates from a directory on filesystem.

temlates:
  source:
    type: directory
    directory_path: /path/to/templates
    templates_dir: Java # directory with templates for this language

Commands

Commands provided inside generated.default.commands and generated.<spec_version>.commands are executed one by one inside the directory with code for currently generated spec version.

Commands provided inside generated.default.tests and generated.<spec_version>.tests are executed one by one inside the directory with code for currently tested spec version.

Each command has following attributes:

Validation commands

Validation commands have exactly the same structure as the above commands. They're run for every created full OpenAPI spec (since there might be more of these based on how different languages define their spec_sections).

The only templating variable these commands can use is {{full_spec_path}}, which is a path to the spec currently being validated.

Functions in Commands

This section lists recognized functions for language phase and validation commands, as mentioned in the section above.

When used as this: json { "function": "glob_re", "args": [ "*.go", ".*(?<!_test.go)$" ] } It will match all files ending with .go except files that end with _test.go.

Templating Commands

The commandline arguments of commands can also use following templating values (Mustache templating is used):

Downstream Templates

downstream_templates are a mapping of additional templates (relative to the Spec Repo directory) to files they'll be rendered as (relative the top level directory of the generated library).

Downstream templates are only generated once per language and have access to a subset of templating values available for commands:

Section Files

By design, apigentools doesn't operate on full OpenAPI specification files, but on "section files". These are, essentially, an exploded OpenAPI spec—these aid the development experience when working on these individual files instead of one huge file. Note that these files have to be listed explicitly in spec_sections.

On apigentools' invocation, these are merged into a single OpenAPI spec file and used as input for further operations, e.g. for openapi-generator.

We recommend having a header.yaml file to for your openapi, info and servers attributes and a shared.yaml file for entities shared among multiple spec sections.

pre-commit Hooks

You can make sure that all contributors create syntactically valid specification that also follow all rules from config.yaml by creating a pre-commit check. Here is an example, to add in your .pre-commit-config.yaml file:

---
# Update the rev variable with the release version that you want, from the apigentools repo.
- repo: https://github.com/DataDog/apigentools.git
  rev: v0.10.0
  hooks:
    - id: validate