Spec Repo
Note
This is documentation for apigentools major version 1, which has significant differences from the 0.X series. Refer to upgrading docs for further details.
A "spec repo" is a repository (or, more generally, a directory), that contains files necessary for apigentools to generate your client code. The recommended layout is as follows:
.
├── .gitignore
├── config # config directory is mandatory
│ ├── config.yaml # general config for apigentools, mandatory
│ └── languages # languages directory is mandatory when using openapi-generator
│ └── java_v1.json # openapi-generator config for Java client for v1 API
├── downstream-templates # optional directory for downstream templates, name can be arbitrary
│ └── java
│ └── LICENSE # a file/template to add to generated client
├── generated # generated code will end up in this directory, mandatory
│ └── .gitkeep
├── spec # directory with all OpenAPI spec files, mandatory
│ └── v1 # directory with spec of v1 of the API
│ ├── accounts.yaml # example: spec of the accounts API
│ ├── header.yaml # header of the OpenAPI spec
│ └── shared.yaml # entities shared between multiple parts of the OpenAPI spec
├── template-patches # optional directory for template patches, name can be arbitrary
│ └── java-01-minor-change.patch # a patch to apply to openapi-generator template
└── templates # openapi-generator templates with applied patches end up here, mandatory
└── .gitkeep
You can create a basic scaffold of a spec repo using the init
subcommand. For example:
apigentools init myspecrepo
Most of the paths in this layout can be overriden with command line arguments passed to the apigentools executable.
Overview of spec repo contents
.gitignore
This is a standard .gitignore file.
config/
This is a directory containing config.yaml, which is a configuration file for apigentools.
config/languages/
This directory contains openapi-generator configuration files. There must be one file for every language and major API version configuration in your config/config.yaml
. For information on what keys can go in these files, use openapi-generator config-help -g <LANGUAGE>
.
downstream-templates/
This directory contains downstream templates, if used (i.e. templates that are provided by the user, rather than openapi-generator).
generated/
This directory contains code generated by apigentools. It is usually empty when you clone this repository, as the generated clients are supposed to have their own repositories and be gitignored from the spec repo.
spec/
This directory contains per-major-API-version OpenAPI specifications of your API. For example, if your API has a v1
and v2
, the spec
directory would have v1
and v2
subdirectories. Each of these would contain at least header.yaml
and shared.yaml
"section files". Refer to section files reference for more information.
template-patches/
This directory contains your downstream template patches, if you decide to use these. These are applied to openapi-generator templates before the generation process using the apigentools templates
command, thus allowing you to customize the upstream templates.
templates/
This directory contains processed upstream templates, i.e. IOW templates from openapi-generator with your template patches applied on top. These are usually gitignored from the spec repo; to process templates without running the whole code generation, use the apigentools templates
command.
File Formats
This document serves as a reference of various file formats used by apigentools.
config/config.yaml
This is a configuration file for apigentools. It must be present in the spec repo for apigentools to work.
Example:
container_opts:
image: datadog/apigentools:latest
minimum_apigentools_version: 1.0.0
languages:
java:
generation:
default:
templates:
patches:
- template-patches/java-0001-custom-license-header.patch
source:
type: openapi-jar
jar_path: /usr/bin/openapi-generator-cli.jar
templates_dir: Java
commands:
- commandline:
- rm
- -rf
- src/main/java/com/datadog/api/{{spec_version}}
description: Remove old generated files
- commandline:
- function: openapi_generator_generate # expands to the default generation command
- --additional-properties
- java8=true,dateLibrary=java8
description: Generate code using openapi-generator
- commandline:
- rm
- -rf
- docs
description: Remove unwanted docs folder
tests:
- commandline:
- mvn
- test
description: "Run tests using maven"
downstream_templates:
downstream-templates/java/README.md: README.md
github_org_name: my-github-org
github_repo_name: my-java-client
library_version: "0.0.1"
spec_versions: ["v1", "v2"]
version_path_template: "myapi_{{spec_version}}"
spec_sections":
v1: ["accounts.yaml", "users.yaml"]
v2: ["users.yaml"]
spec_versions: ["v1", "v2"]
user_agent_client_name: "MyCompany"
validation_commands:
- commandline:
- openapi-generator
- validate
- -i
- "{{full_spec_path}}"
description: "Validate full spec using openapi-generator"
The structure of the general config file is as follows, starting with top level keys:
config_version
(Added in config version1.2
). - Version of the configuration file.container_opts
- Options for containerized command execution. See container_opts section below.minimum_apigentools_version
- Minimum required version of the apigentools package required for this spec repository.languages
- Settings for individual languages; contains a mapping of language names to their settings.container_opts
- See container_opts section below.- individual-language-settings:
container_opts
- See container_opts section below.generation
- Setting for code generation.default
- Default settings for code generations. When any of the keys defined in thedefault
mapping are not found in the spec-version-settings, it's taken from here.container_opts
- See container_opts section below.commands
- Commands to execute to generate code. See commands.templates
- Instructions on how to preprocess templates to pass to code generator.tests
- Commands to execute to test code usingapigentools test
. See commands.validation_commands
(Added in config version1.2
) - Same as top-level and per-languagevalidation_commands
, allows overriding both of these.
- spec-version-settings - Exactly the same as
default
above, but used for overriding operations specifically for the given major API version.
downstream_templates
- Downstream templates.github_org_name
- Name of the Github organization of the client for this language.github_repo_name
- Name of the Github repository of the client for this language.library_version
- Version of the generated library, for now this only serves as a variable useful in command templatingspec_sections
- Same as top-levelspec_sections
. Use to override the subset of spec sections to generate for each spec version of this language. For every spec version not specified as a key, the top-level list of sections for this spec version is used.spec_versions
- Same as top-levelspec_versions
. Use to override the subset of major versions to generate for this language. If not specified, the top-levelspec_versions
value is used.validation_commands
(Added in config version1.2
) - Same as top-levelvalidation_commands
, allows overriding the top-level value on per-language basis.version_path_template
- Mustache template for the name of the subdirectory in the Github repo where code for individual major versions of the API will end up, e.g. withmyapi_{{spec_version}}
as value and agithub_repo_name
of valuemy-java-client
, the code forv1
of the API will end up inmyapi-java-client/myapi_v1
spec_sections
- Mapping of major spec versions (these must be inspec_versions
) to lists of files with paths/tags/components definitions to merge when creating full spec. Files not explicitly listed here are ignored.spec_versions
- List of major versions currently known and operated on. These must be subdirectories of thespec
directory.user_agent_client_name
- The HTTP User Agent string will be set to{user_agent_client_name}/{package_version}/{language}
.validation_commands
- List of commands to run during validation phase. The commands have the same structure and mechanics like language commands (see above). Validation commands will be executed for each full spec that is created.
config_version
In apigentools 1.2.0, the concept of config_version
was introduced. Since this version, apigentools will verify that the config it's running against is of a supported version.
config_version
adopts a MAJOR.MINOR
versioning scheme. Configuration will only change in minor apigentools releases, so an X.Y.Z
version of apigentools always supports config_version: "X.Y"
and perhaps some older versions as well.
Configs created for apigentools between 1.0.0 and 1.2.0 (as well as these which don't have config_version
specified) are considered to be of version 1.0
.
container_opts
Since apigentools major version 1, all commands are expected to be executed inside containers by default. In order to achieve this a container_opts
structure can be provided on different levels of the configuration file to modify how containers are executed.
The container_opts
defined at various levels have certain inheritance set up. To compute resulting container_opts
for a command, the container_opts
are taken and considered in the following order:
container_opts
defined for the commandcontainer_opts
defined for the spec version that the command is being runcontainer_opts
for thedefault
generationcontainer_opts
for the languagecontainer_opts
for the top level
If at any point an inherit: False
value is reached, the inheritance process stops. The inheritance is mostly intuitive, for example if environment
value FOO
is defined for the command and for the language, the one for the command is used.
Following container_opts
are recognized:
environment
- Environment variables to pass to the container. These are merged while inheriting, so different environment variables from different levels will all be used.image
- What image to use to run the command. This is the only option that's not stopped by inheritance. If a command hasinherit: False
but noimage
defined, theimage
value is still inherited. There is always a default global valuedatadog/apigentools:latest
unless specified otherwise.inherit
- Stop the inheritance at this point (ifFalse
; the default isTrue
).no_container
- Run this command outside of container (recommended for development/debugging purposes only).
Preprocess Templates
Template preprocessing is a key feature of apigentools, as it allows for downstream modification of templates used for code generation. Preprocessing templates is a two-step operation:
- First, upstream templates are obtained (by default, these are obtained from the container specified by container_opts for given spec version;
container_opts
inheritance applies). - Second, template patches are applied (optional).
Based on the above, the templates step contains two keys:
patches
- List of paths to patches (inside the Spec Repo) to apply to templatessource
- Source for the upstream templates; recognized configurations follow in subsections bellow. Additionally, following configuration keys are recognized:no_container
- Don't obtain the templates from container, but from local host.
openapi-jar
Extract templates from openapi-generator JAR file.
templates:
source:
type: openapi-jar
jar_path: /usr/bin/openapi-generator-cli.jar # path to the openapi generator JAR file
templates_dir: Java # directory with templates for this language
openapi-git
Extract templates from openapi-generator Github repository.
templates:
source:
type: openapi-git
git_committish: "v4.3.0" # git committish to checkout before extracting the templates
templates_dir: Java # directory with templates for this language
directory
Extract templates from a directory on filesystem.
temlates:
source:
type: directory
directory_path: /path/to/templates
templates_dir: Java # directory with templates for this language
Commands
Commands provided inside generated.default.commands
and generated.<spec_version>.commands
are executed one by one inside the directory with code for currently generated spec version.
Commands provided inside generated.default.tests
and generated.<spec_version>.tests
are executed one by one inside the directory with code for currently tested spec version.
Each command has following attributes:
description
- A description of the command.commandline
- The command itself; the items in the list are:- templatable strings and, potentially
- functions that represent callbacks to the Python code. Each function is represented as a map:
function
- Name of the function (see below for list of recognized functions).args
- List of args to pass to the function in Python code (as in*args
).kwargs
- Mapping of args to pass to the function in Python code (as in**kwargs
).- The result of the function call is then used in the actual command line call (note that the functions are called by apigentools outside the container that actually executes the command).
container_opts
- container_opts describing how to execute the command inside a container. Do note that commands are executed in non-interactive containers.
Validation commands
Validation commands have exactly the same structure as the above commands. They're run for every created full OpenAPI spec (since there might be more of these based on how different languages define their spec_sections
).
New in 1.2.0: validation_commands
can be specified on multiple levels, with inheritance working like this: For a specific language version, validation_commands
are taken from:
validation_commands
definition for the specific version (if present there); elsevalidation_commands
definition in thedefault
section (if present there); elsevalidation_commands
definition for the specific language (if present there); elsevalidation_commands
definition from the top level (if present there); else- empty command list is used
The only templating variable these commands can use is {{full_spec_path}}
, which is a path to the spec currently being validated.
Functions in Commands
This section lists recognized functions for language phase and validation commands, as mentioned in the section above.
glob
- runs Python'sglob.glob
glob_re
- runs Python'sglob.glob
and then filters its results by callingre.match
using given Python regular expression
When used as this:
json
{
"function": "glob_re",
"args": [
"*.go",
".*(?<!_test.go)$"
]
}
It will match all files ending with .go
except files that end with _test.go
.
-
openapi_generator_generate
- expands to- "openapi-generator" - "generate" - "--http-user-agent" - "{{user_agent_client_name}}/{{library_version}}/{{language_name}}" - "-g" - "{{language_name}}" - "-c" - "{{language_config}}" - "-i" - "{{full_spec_path}}" - "-o" - "{{version_output_dir}}" - "--additional-properties" - "apigentoolsStamp='{{stamp}}'"
If the spec version that the default command is being run for is using a
templates
section, thecommandline
is further appended with:- -t - "{{templates_dir}}"
Templating Commands
The commandline
arguments of commands can also use following templating values (Mustache templating is used):
{{cwd}}
- Current working directory{{full_spec_path}}
- Path to the input full OpenAPI spec{{github_repo_name}}
- Value from config{{github_repo_org}}
- Value from config{{language_config}}
- Path to the proper config file underconfig/languages/
directory{{language_name}}
- Name of the language from the config{{library_version}}
- Value from config{{spec_version}}
- Version of the spec currently being processed{{stamp}}
- Same asapigentoolsStamp
mentioned in reproducibility{{templates_dir}}
- Directory with templates for this language/API version combination{{user_agent_client_name}}
- Value from config{{version_output_dir}}
- Directory to which the output for this spec version will be put{{top_level_dir}}
- Relative path (to{{version_output_dir}}
) to the top level directory of the generated language repo
Downstream Templates
downstream_templates
are a mapping of additional templates (relative to the Spec Repo directory) to files they'll be rendered as (relative the top level directory of the generated library).
Downstream templates are only generated once per language and have access to a subset of templating values available for commands:
{{full_spec_path}}
{{github_repo_name}}
{{github_repo_org}}
{{language_name}}
{{library_version}}
{{stamp}}
{{user_agent_client_name}}
Section Files
By design, apigentools doesn't operate on full OpenAPI specification files, but on "section files". These are, essentially, an exploded OpenAPI spec—these aid the development experience when working on these individual files instead of one huge file. Note that these files have to be listed explicitly in spec_sections
.
On apigentools' invocation, these are merged into a single OpenAPI spec file and used as input for further operations, e.g. for openapi-generator.
We recommend having a header.yaml
file to for your openapi
, info
and servers
attributes and a shared.yaml
file for entities shared among multiple spec sections.
pre-commit
Hooks
You can make sure that all contributors create syntactically valid specification
that also follow all rules from config.yaml
by creating a pre-commit
check.
Here is an example, to add in your .pre-commit-config.yaml
file:
---
# Update the rev variable with the release version that you want, from the apigentools repo.
- repo: https://github.com/DataDog/apigentools.git
rev: v0.10.0
hooks:
- id: validate