Properties Schema Validation #14
Comments
|
Thank you for contributing to RudderStack Transformations. Your submission will be reviewed soon. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Contact Details
ioannis.foukarakis@gmail.com
Language
Javascript
Category
Data Security & Governance
Description
Properties Schema Validation
Description
Asserts the event's
propertiesfollow the constraints of a JSON schema.One of the most common challenges when gathering data is agreement between all involved stakeholders on the format, structure, and semantics of data. One popular solution is to apply "data contracts" to ensure that different systems or components communicate effectively and accurately. This transformation uses JSON Schema specification to assert that event's properties obey the agreed constraints.
Events that fail to comply with the agreed constraint are currently dropped, but logic can be modified to re-route them to a DLQ-like destination.
Deployment
jsonschema.Notes/Troubleshooting
JSONSchema draft-2020-12is currently supported.awaitis before the call tocontracts.registerSchemaFrom....Configuration
Register schemas for event
Schemas can be loaded either from URLs or from JSON objects. For example, the following validation function registers two schemas for two events from two different URLs:
In the following example, a single schema is registered from a JSON Object:
Drop events in case there's no schema registered
Simply add
falseas an argument toContractsconstructor:const contracts = new Contracts(true);Developing
A sample github project for writing transformations is available at https://github.com/ifoukarakis/rudderstack-transformations.
Code Block
import { Schema } from 'jsonschema'; /* Class responsible for managing contracts. */ export class Contracts { /** * Create a new contracts instance. * * @param {Boolean} allowUnregisteredEvents whether to allow unknown events or not. */ constructor(allowUnregisteredEvents=true) { this.schemas = {}; this.allowUnregisteredEvents = allowUnregisteredEvents; } async registerSchemaFromJSON(event, schema) { this.schemas[event] = new Schema(schema); } async registerSchemaFromURL(event, url) { const response = await fetch(url); this.schemas[event] = new Schema(response); } validateProperties(event) { const schema = this.schemas[event.event]; if(schema) return schema.validate(event.properties); // If unregistered event, fallback. return this.allowUnregisteredEvents; } } const productSchema = { "$id": "https://example.com/person.schema.json", "$schema": "https://json-schema.org/draft/2020-12/schema", "title": "Product", "type": "object", "properties": { "product_id": { "type": "string", "description": "The product's ID." }, "name": { "type": "string", "description": "The person's last name." }, "price": { "type": "string", "pattern": "^(0|([1-9]+[0-9]*))(\\.[0-9]{1,2})?$", "minLength": 1, "description": "The product's price.", "examples": [ "0", "0.00", "0.05", "19.95", "255.5", "120000" ] } }, "required": ["product_id", "name", "price"] } export async function transformBatch(events, metadata) { // eslint-disable-line no-unused-vars // Replace following line with const contracts = new Contracts(false); if you want to consider unregistered events as invalid. const contracts = new Contracts(true); // Register events here await contracts.registerSchemaFromJSON("Add To Cart", productSchema); // Register more events here. return events.filter(event => contracts.validateProperties(event)) }Input Payload for testing
License
The text was updated successfully, but these errors were encountered: