Skip to content

Commit

Permalink
Feature / API documentation update (#472)
Browse files Browse the repository at this point in the history
* Update model API doc comments

* Update Sphinx config for AutoAPI

* Custom templates for modules and classes in AutoAPI

* Use handwritten RST for top level reference pages

* Do not flat-pack modules in docgen-ctrl (this is now handled correctly in Sphinx)

* Qualify namespaces for types in the code generator for python docs

* Documentation theme updates

* Fix a bad doc reference in the runtime exception classes

* Fix doc layout for DECIMAL precision / scale info

* Fix Sphinx warnings

* Flatpack the runtime metadata module for documentation

* Update the code generator to handle type scoping for both regular Python and Sphinx AutoAPI

* Fix handling nested types in the generator when not running with doc_semantics

* Work around for scope resolution of one nested type (that cannot be solved in the regular structure)

* Doc updates in model_api module

* Exclude declare_ functions from public docs

* Doc updates in static_api module

* Doc updates in the launch module

* Add doc comments for top level constants in the api module

* Top level doc comments for public modules

* Minor tidy up in conf.py

* Update function references in @see annotations for metadata.proto

* Update for translating @see comments in the Python generator

* Remove chapter numbers from paths in tutorial files

* Fix references to class methods

* Fix references to class attributes

* Update attribute references defined in code

* Fix generating attribute references from @see annotations

* Update type references for some non-TRAC types in doc comments

* Update line numbers in modelling tutorials after header changes

* Update line numbers in app dev tutorials after header changes

* Use release instead of version in top level headline (to change in 0.8)

* Rename metadata listing and service listing references

* Fix width of code blocks in modelling tutorials

* Fix width of code blocks in app dev tutorials

* Fix headings in introduction

* Use chapter headings in tutorials

* Table headers in dark mode

* Use finTRAC CSS for docs styling

* Fix links in the main README file
  • Loading branch information
martin-traverse authored Nov 11, 2024
1 parent 68c5129 commit 221c6d4
Show file tree
Hide file tree
Showing 36 changed files with 1,308 additions and 553 deletions.
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ TRAC is designed to break the trade-off that has traditionally been required, be
platforms. It offers best of both worlds, power, control and analytical flexibility.

The core platform services - i.e. TRAC Data & Analytics Platform (or TRAC D.A.P.) - are maintained by
`finTRAC Limited <https://www.fintrac.co.uk>`_ in association with the `finos Foundation <https://www.finos.org>`_
under the `Apache Software License version 2.0 <https://www.apache.org/licenses/LICENSE-2.0>`_.
[finTRAC Limited](https://www.fintrac.co.uk) in association with the [FINOS Foundation](https://www.finos.org)
under the [Apache Software License version 2.0](https://www.apache.org/licenses/LICENSE-2.0).

## Documentation and Packages

Expand Down Expand Up @@ -65,7 +65,7 @@ With TRAC D.A.P. you can build and run production-ready models right on your des
All you need is an IDE, Python and the tracdap-runtime Python package.
TRAC D.A.P. requires Python 3.8 or later.

The [modelling tutorial](https://tracdap.finos.org/en/stable/modelling/tutorial/chapter_1_hello_world.html)
The [modelling tutorial](https://tracdap.finos.org/en/stable/modelling/tutorial/hello_world.html)
shows you how to get set up and write your first models. You can write models locally using
an IDE or notebook, once the model is working t can be loaded to the platform without modification.
TRAC D.A.P. will validate the model and ensure it behaves the same on-platform as it does locally.
Expand Down
209 changes: 125 additions & 84 deletions dev/codegen/generator.py

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions dev/codegen/protoc-ctrl.py
Original file line number Diff line number Diff line change
Expand Up @@ -265,11 +265,11 @@ def build_protoc_args(generator, proto_paths, output_location, packages):

elif generator == "python_doc":

options = "--trac_opt=target_package=tracdap.rt;doc_format"
options = "--trac_opt=target_package=tracdap.rt;doc_format;flat_pack"

elif generator == "api_doc":

options = "--trac_opt=flat_pack;doc_format"
options = "--trac_opt=doc_format;flat_pack"

else:

Expand Down
55 changes: 17 additions & 38 deletions dev/docgen/docgen-ctrl.py
Original file line number Diff line number Diff line change
Expand Up @@ -126,38 +126,44 @@ def python_runtime_codegen(self):

# Set up the tracdap.rt package
self._mkdir(doc_src.joinpath("tracdap"))
self._touch(doc_src.joinpath("tracdap/__init__.py"))
self._mkdir(doc_src.joinpath("tracdap/rt"))
self._touch(doc_src.joinpath("tracdap/rt/__init__.py"))

# Copy only API packages / modules from the runtime library
api_modules = [
"tracdap/rt/__init__.py",
"tracdap/rt/_version.py",
"tracdap/rt/api/",
"tracdap/rt/launch/",
"tracdap/rt/exceptions.py"]

api_exclude = ["experimental.py"]

for module in api_modules:

src_module = runtime_src.joinpath(module)
tgt_module = doc_src.joinpath(module)

if src_module.is_dir():
self._flatten_package(src_module, tgt_module, api_exclude)
self._cp_tree(src_module, tgt_module)
else:
self._cp(src_module, tgt_module)

# Note: DOCGEN_REMOVE is not strictly necessary for the trac.rt.api package after flattening

# We include the runtime metadata classes at global scope in the api package for convenience
# Having them show up in both places in the docs is confusing (for users, and the autoapi tool)!
# So, remove those imports from the API package before running Sphinx

# This can probably be improved by setting __all__ instead

self._log.info("* fix docgen imports")

for line in fileinput.input(doc_src.joinpath('tracdap/rt/api/__init__.py'), inplace=True):
if "DOCGEN_REMOVE" not in line:
print(line, end="")
fix_import_modules = [
'tracdap/rt/api/__init__.py',
'tracdap/rt/api/model_api.py',
'tracdap/rt/api/static_api.py',
]

for module in fix_import_modules:
for line in fileinput.input(doc_src.joinpath(module), inplace=True):
if "DOCGEN_REMOVE" not in line:
print(line, end="")

# We also want the runtime metadata
# There is a separate mechanism for generating this and getting it into the library for packaging
Expand All @@ -169,7 +175,7 @@ def python_runtime_codegen(self):
"--out", "build/doc/code/runtime_python"]

self._run_subprocess(codegen_exe, codegen_args)
self._mv(doc_src.joinpath('tracdap/metadata'), doc_src.joinpath('tracdap/rt/metadata'))
self._mv(doc_src.joinpath('tracdap/metadata.py'), doc_src.joinpath('tracdap/rt/metadata.py'))

def get_version_and_release(self):

Expand Down Expand Up @@ -252,33 +258,6 @@ def _rm_tree(self, target_dir):
if target_dir.exists():
shutil.rmtree(target_dir)

def _flatten_package(self, src_package: pathlib.Path, tgt_package: pathlib.Path, exclude_modules = None):

target_package_init = tgt_package.joinpath("__init__.py")

self._mkdir(tgt_package)
self._touch(target_package_init)

for module in src_package.iterdir():

if module.is_dir():
self._log.warning(f"Flatting sub-packages not supported, skipping {module}")
continue

# Skip __init__, __main__ and other special modules
if module.name.startswith("__"):
continue

if exclude_modules and module.name in exclude_modules:
continue

module_content = module.read_text()

with open(target_package_init, "at") as target_stream:
target_stream.write(module_content)
target_stream.write("\n")
target_stream.flush()

def _run_subprocess(self, sp_exe, sp_args, use_venv=True, capture_output=False):

self._log.info(f'* {sp_exe} {" ".join(sp_args)}')
Expand Down
32 changes: 32 additions & 0 deletions doc/_static/tracdap.css → doc/_static/fintrac-docs.css
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,13 @@ div.rst-content h6 {
text-transform: uppercase;
}

div.rst-content [id^=tracdap-] > h1,
div.rst-content [id^=module-tracdap] > h1,
span.code h1 {
text-transform: none;
}


/* Code - matches default link color */

code {
Expand All @@ -79,6 +86,19 @@ pre > span.linenos {
margin-right: 10px;
}

dt.sig {
background-color: #A3C4D9;
}

dl dt.sig {
margin: 1rem 0;
padding: 1rem;
box-shadow: 0 2px 4px 0 rgba(0,0,0,.2);
border-radius: .25rem;
overflow: hidden;
}


/* Main buttons (light & dark) */

.btn-primary:hover,
Expand Down Expand Up @@ -237,8 +257,16 @@ body.theme-dark code {

.table-dark,
body.theme-dark table.docutils:not(.field-list) {
color: #EAEAEA;
background-color: #1D3D59;
}

.table .thead-dark th,
table.docutils:not(.field-list) .thead-dark th,
body.theme-dark table.docutils:not(.field-list) thead th {
color: #EAEAEA;
background-color: #2A577D;
border-color: #3B668C;
}

.table-dark th,
Expand Down Expand Up @@ -313,6 +341,10 @@ body.theme-dark .toc .current > a {
}


body.theme-dark dt.sig {
background-color: #1D3D59;
}

/* Dark Mode - Sphinx Design Elements */

body.theme-dark .sd-card-header {
Expand Down
120 changes: 120 additions & 0 deletions doc/_templates/autoapi/python/class.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
{% if obj.display %}
{% if is_own_page %}
.. raw:: html

{{ '<span class="code">' }}

{{ obj.name }}
{{ "=" * obj.name | length }}

.. raw:: html

{{ '</span>' }}

{% endif %}
{% set visible_children = obj.children|selectattr("display")|list %}
{% set own_page_children = visible_children|selectattr("type", "in", own_page_types)|list %}
{% if is_own_page and own_page_children %}
.. toctree::
:hidden:

{% for child in own_page_children %}
{{ child.include_path }}
{% endfor %}

{% endif %}

{% set show_obj_args = False %}

.. py:{{ obj.type }}:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}{% if show_obj_args %}({{ obj.args }}){% endif %}
{% for (args, return_annotation) in obj.overloads %}
{{ " " * (obj.type | length) }} {{ obj.short_name }}{% if args %}({{ args }}){% endif %}

{% endfor %}

{% if is_own_page %}
.. py:currentmodule:: {{ obj.id.replace('.' + obj.name, '') }}
{% endif %}

{% if obj.bases %}
{% if "show-inheritance" in autoapi_options %}

Bases: {% for base in obj.bases %}{{ base|link_objs }}{% if not loop.last %}, {% endif %}{% endfor %}
{% endif %}


{% if "show-inheritance-diagram" in autoapi_options and obj.bases != ["object"] %}
.. autoapi-inheritance-diagram:: {{ obj.obj["full_name"] }}
:parts: 1
{% if "private-members" in autoapi_options %}
:private-bases:
{% endif %}

{% endif %}
{% endif %}
{% if obj.docstring %}

{{ obj.docstring|indent(3) }}
{% endif %}
{% for obj_item in visible_children %}
{% if obj_item.type not in own_page_types %}

{{ obj_item.render()|indent(3) }}
{% endif %}
{% endfor %}
{% if is_own_page and own_page_children %}
{% set visible_attributes = own_page_children|selectattr("type", "equalto", "attribute")|list %}
{% if visible_attributes %}
Attributes
----------

.. autoapisummary::

{% for attribute in visible_attributes %}
{{ attribute.name }}
{% endfor %}


{% endif %}
{% set visible_exceptions = own_page_children|selectattr("type", "equalto", "exception")|list %}
{% if visible_exceptions %}
Exceptions
----------

.. autoapisummary::

{% for exception in visible_exceptions %}
{{ exception.id }}
{% endfor %}


{% endif %}
{% set visible_classes = own_page_children|selectattr("type", "equalto", "class")|list %}
{% if visible_classes %}
Classes
-------

.. autoapisummary::

{% for klass in visible_classes %}
{{ klass.id }}
{% endfor %}


{% endif %}
{% set visible_methods = own_page_children|selectattr("type", "equalto", "method")|list %}
{% if visible_methods %}
Methods
-------

.. autoapisummary::

{% for method in visible_methods %}
{{ method.name }}
{% endfor %}


{% endif %}
{% endif %}
{% endif %}
38 changes: 38 additions & 0 deletions doc/_templates/autoapi/python/data.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{% if obj.display %}
{% if is_own_page %}
{{ obj.id }}
{{ "=" * obj.id | length }}

{% endif %}
.. py:{{ obj.type }}:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.name }}{% endif %}
:canonical: {{ obj.id }}
{% if obj.annotation is not none %}
:type: {% if obj.annotation %} {{ obj.annotation }}{% endif %}
{% endif %}
{% if obj.value is not none %}
{% if obj.value.splitlines()|count > 1 %}
:value: Multiline-String

.. raw:: html

<details><summary>Show Value</summary>

.. code-block:: python
{{ obj.value|indent(width=6,blank=true) }}
.. raw:: html

</details>

{% else %}
:value: {{ obj.value|truncate(100) }}
{% endif %}
{% endif %}

{% if obj.docstring %}

{{ obj.docstring|indent(3) }}
{% endif %}
{% endif %}
23 changes: 23 additions & 0 deletions doc/_templates/autoapi/python/function.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{% if obj.display %}
{% if ":display: False" not in obj.docstring %}
{% if is_own_page %}
{{ obj.id }}
{{ "=" * obj.id | length }}

{% endif %}
.. py:function:: {% if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}({{ obj.args }}){% if obj.return_annotation is not none %} -> {{ obj.return_annotation }}{% endif %}
{% for (args, return_annotation) in obj.overloads %}

{%+ if is_own_page %}{{ obj.id }}{% else %}{{ obj.short_name }}{% endif %}({{ args }}){% if return_annotation is not none %} -> {{ return_annotation }}{% endif %}
{% endfor %}
{% for property in obj.properties %}

:{{ property }}:
{% endfor %}

{% if obj.docstring %}

{{ obj.docstring|indent(3) }}
{% endif %}
{% endif %}
{% endif %}
Loading

0 comments on commit 221c6d4

Please sign in to comment.