Contributing to vws-python-mock

Contributions to this repository must pass tests and linting.

CI is the canonical source of truth.

Install contribution dependencies

Install Python dependencies in a virtual environment.

$ pip install --editable . --group dev

Spell checking requires enchant. This can be installed on macOS, for example, with Homebrew:

$ brew install enchant

and on Ubuntu with apt:

$ apt-get install -y enchant

Install pre-commit hooks:

$ prek install

Linting

Run lint tools either by committing, or with:

$ prek run --all-files --hook-stage pre-commit --verbose
$ prek run --all-files --hook-stage pre-push --verbose
$ prek run --all-files --hook-stage manual --verbose

Running Tests

Create an environment variable file for secrets:

$ cp vuforia_secrets.env.example vuforia_secrets.env

Some tests require Vuforia credentials. To run these tests, add the Vuforia credentials to the file vuforia_secrets.env. See Connecting to Vuforia.

Then run pytest:

$ pytest

Connecting to Vuforia

To connect to Vuforia, Vuforia target databases must be created via the Vuforia Web UI. Then, secret keys must be set as environment variables.

The test infrastructure allows those keys to be set in the file vuforia_secrets.env. See vuforia_secrets.env.example for the environment variables to set.

Do not use a target database that you are using for other purposes. This is because the test suite adds and deletes targets.

To create a target database, first create a license key in the Vuforia License Manager. Then, add a database from the Vuforia Target Manager.

To find the environment variables to set in the vuforia_secrets.env file, visit the Target Database in the Vuforia Target Manager and view the “Database Access Keys”.

Two Cloud databases are necessary in order to run all the Cloud Target tests. One of those must be an inactive project. The script creates the inactive project automatically by deleting its license.

VuMark tests require one VuMark database.

Targets sometimes get stuck at the “Processing” stage meaning that they cannot be deleted. When this happens, create a new target database to use for testing.

To create databases without using the browser, use admin/create_secrets_files.py:

$ export VWS_EMAIL_ADDRESS=...
$ export VWS_PASSWORD=...
$ export NEW_SECRETS_DIR=...
# You may have to run this a few times, but it is idempotent.
$ python admin/create_secrets_files.py
# Each generated file gets its own active Cloud database credentials.

For the complete archive and GitHub Actions setup procedure, see Continuous Integration.

Skipping Some Tests

The tests run against several backends: the real Vuforia, the in-memory mock, and the Flask applications of the mock served in the test process. The last of these exercises the handlers which the Docker deployment runs, but not the split between its containers. The tests in tests/mock_vws/test_docker.py build and run the containers for that. The pytest-multi-backend plugin adds options which skip some tests:

--skip-backend=real   Skip tests against the real Vuforia
--skip-backend=mock   Skip tests against the in-memory mock Vuforia
--skip-backend=flask_in_process
                      Skip tests against the Flask applications served
                      in the test process
--skip-marker=requires_docker_build
                      Skip tests which build and run the Docker images

Give an option once per backend or marker to skip.

Verifying signed Model Target requests

Creating an advanced Model Target dataset with a state-based configuration or a standard dataset with inline CAD data is a “signed” request: the real Vuforia signs the trained dataset, and each signing consumes the account’s Model Target training allowance. The allowance is small (roughly 20 signings), it is shared by every CI job and every concurrent run, and it cannot be raised or reset. Verifying signed requests on every run exhausted the allowance within hours and then made every CI run fail with TRAINING_ALLOWANCE_EXCEEDED.

The signed test cases therefore run against the mock backends on every run, but are skipped against the real Vuforia by default. To verify them against the real Vuforia, for example after the allowance has recovered, opt in with:

--verify-model-target-signing
                      Run signed Model Target dataset tests against
                      the real Vuforia

The equivalent unsigned requests (a standard dataset, or an advanced dataset without a state-based configuration) are far cheaper and are verified against the real Vuforia on every run. With enough traffic even unsigned dataset creation can be rejected with TRAINING_ALLOWANCE_EXCEEDED; an unexpected allowance rejection is reported as an expected failure rather than a test failure, and the affected tests pass again automatically once the allowance recovers.

Retrying transient real Model Target failures

The load balancer in front of the real Model Target Web API sometimes answers a perfectly good request with a gateway error, or drops the connection. That is nothing to do with the contract under test, and a rerun of the same job passes, so a small number of retries is applied to requests which are safe to repeat.

The policy is in tests/mock_vws/utils/model_target_retries.py. It applies only while a test is running against the real Model Target backend, and only to GET requests: repeating a dataset creation can create a second dataset and can consume the account’s Model Target training allowance, so mutating requests are sent exactly once. Three attempts are made, with an exponential backoff with jitter of up to ten seconds, using tenacity. A request which still fails transiently on the last attempt is not hidden: the response is returned as it is, so the assertion reports the real status and body.

The mock backends are unaffected, so tests which configure a mock to answer with a 5xx still get that response immediately.

This is separate from the repository-wide pytest-retry configuration, which retries a test only when it fails with one of the exception types returned by pytest_set_filtered_exceptions in the top level conftest.py. The Model Target tests assert on requests.Response objects, so a gateway error reaches pytest-retry as an AssertionError and is never retried by it. Widening that allowlist would retry every failing assertion in the suite, which is why the Model Target policy sits in the request helper instead, before the assertion.

Documentation

Documentation is built on Read the Docs.

Run the following commands to build and view documentation locally:

$ uv run --group=dev sphinx-build -M html docs/source docs/build -W
$ python -c 'import os, webbrowser; webbrowser.open("file://" + os.path.abspath("docs/build/html/index.html"))'

Continuous Integration

See Continuous Integration.

Learnings about VWS

Vuforia Web Services, at the time of writing, does not behave exactly as documented.

The following list includes details of differences between VWS and expected or documented behavior.

When attempting to delete a target immediately after creating it, a FORBIDDEN response is returned. This is because the target goes into a processing state.

image is required for POST /targets, but it is documented as not mandatory.

The tracking_rating returned by GET /targets/<target_id> can be -1. This happens only while a newly uploaded target is processing, and it lasts for about a second. Updating a target returns it to the processing state but does not return the rating to -1; the new image’s rating is reported straight away.

The database summary from GET /summary has multiple undocumented return fields.

The database summary from GET /summary is not immediately accurate.

The documentation page Vuforia Query Web API states that the Content-Type header must be set to multipart/form-data. However, it must be set to multipart/form-data; boundary=<BOUNDARY> where <BOUNDARY> is the boundary used when encoding the form data.

The documentation page Vuforia Query Web API states that Content-Type will be the only response header. This is not the case.

The documentation page Vuforia Query Web API states that 10 is the maximum allowed value of max_num_results. However, the maximum allowed value is 50.

A response to an invalid query may have an application/json content type but include text (not JSON) data.

After deleting a target, for up to approximately 30 seconds, matching it with a query returns a 500 response.

A target with the name \uffff gets stuck in processing.

The documentation page Vuforia Query Web API states that “The API accepts requests with unknown data fields, and ignore the unknown fields.”. This is not the case.

The documentation page Vuforia Query Web API states “Maximum image size: 2.1 MPixel. 512 KiB for JPEG, 2MiB for PNG”. However, JPEG images up to 2MiB are accepted.

There is no documented limit on the number of pixels in an image, but POST /targets returns ImageTooLarge for an image with more than 37748736 pixels, whatever its file size, aspect ratio or color space. An image of a single color has a tiny file size whatever its dimensions, which is how this limit is reached. The Query API applies no such limit. It applies only its maximum width and height of 30000 pixels.

The request_count in a database summary is always 0.

The documentation for the target summary report says “Note: tracking_rating and reco_rating are provided only when status = success.”. However, reco_rating is never provided and tracking_rating is provided even when the status is “failed”.

Release Process

See Release Process.