JSON, Thrift and gRPC

Introduction (Contd.)

• In general (not only in web) to store or transport data, we need a common format, to specify the structure of data; e.g.,
• Documents: PDF, DOCx, PPTx, ...
• Objects: Java Object Serialization/Deserialization

• How to define the data structure?
  • Binary format (similar to binary files)
  • Difficult to develop & debug
  • machine depended
  • ...
  • Text format (similar to text files)
  • Easy to develop & debug
  • human readable
  • ...

Introduction (Cont.)

• Example: Data structure of a class
• Course name, teacher, # of students, each student information

type Course struct {
  Name     string
  Teacher  string
  Students []Student
  Capacity int
}

type Student struct {
  FirstName string
  LastName  string
  ID        string
}

c := Course {
  Name: "IE",
  Teacher: "Parham Alvani",
  Students: []Student{
    { FirstName: "Parham", LastName: "Alvani", ID: "9231058" },
  },
  Capacity: 30,
}
    

{
  "name": "IE",
  "teacher": "Parham Alvani",
  "students": [
    { "first_name": "Parham", "last_name": "Alvani", "id": "9231058" }
  ],
  "capacity": 30
}
    

The Same Course in CBOR

A4                                 # map(4)
   64 6E616D65                     # text(4) "name"
   62 4945                         # text(2) "IE"
   67 74656163686572               # text(7) "teacher"
   6D 50617268616D20416C76616E69   # text(13) "Parham Alvani"
   68 73747564656E7473             # text(8) "students"
   81                              # array(1)
      A3                           # map(3)
         6A 66697273745F6E616D65   # text(10) "first_name"
         66 50617268616D           # text(6) "Parham"
         69 6C6173745F6E616D65     # text(9) "last_name"
         66 416C76616E69           # text(6) "Alvani"
         62 6964                   # text(2) "id"
         67 39323331303538         # text(7) "9231058"
   68 6361706163697479             # text(8) "capacity"
   18 1E                           # unsigned(30)
    

The Same Course, Positional

No names, no types: both sides must agree on the order (name, teacher, capacity, number of students, then each student)

IE
Parham Alvani
30
1
Parham
Alvani
9231058
    

JSON Syntax

• Data is in name-value pairs
• Field name in double quotes, followed by a colon, followed by a value
• In JSON, the keysmust be strings
• Data is separated by commas
• Curly braces hold objects
• Square brackets hold arrays

• Data Types:
• string

"This is a string"
          

• number

  42
  3.1415926
          

• object

{ "key1": "value1", "key2": "value2" }
          

• array

[ "first", "second", "third" ]
          

• boolean

true
false
          

• null

null
          

JSON Schema

• JSON schema is JSON also
• The JSON document being validated or described we call the instance, and the document containing the description is called the schema.

Hello World

• This accepts anything, as long as it's valid JSON

{}
      

• The most common thing to do in a JSON Schema is to restrict to a specific type. The type keyword is used for that.

{ "type": "string" }
      

Declaring a JSON Schema

• Since JSON Schema is itself JSON, it's not always easy to tell when something is JSON Schema or just an arbitrary chunk of JSON.
• The $schema keyword is used to declare that something is JSON Schema.
• It's generally good practice to include it, though it is not required.

{ "$schema": "https://json-schema.org/draft/2020-12/schema" }
{ "$schema": "http://json-schema.org/draft-07/schema#" }
    

Declaring a unique identifier

• It is also best practice to include an $id property as a unique identifier for each schema.
• For now, just set it to a URL at a domain you control, for example:

{ "$id": "http://yourdomain.com/schemas/myschema.json" }
      

Annotations

• JSON Schema includes a few keywords, title, description, default, examples that aren't strictly used for validation, but are used to describe parts of a schema.
• The title and description keywords must be strings.
• A "title" will preferably be short, whereas a "description" will provide a more lengthy explanation about the purpose of the data described by the schema.
• The default keyword specifies a default value for an item.

String

{ "type": "string" }
    

• Length
• The length of a string can be constrained using the minLength and maxLength keywords.
• For both keywords, the value must be a non-negative number.
• Regular Expressions
• The pattern keyword is used to restrict a string to a particular regular expression.

String (Contd.)

• Format
• The format keyword allows for basic semantic validation on certain kinds of string values that are commonly used.
• Dates and times
• Email addresses
• Host names
• IP Addresses
• ...

Numeric types

• The integer type is used for integral numbers.

{ "type": "integer" }
    

• The number type is used for any numeric type, either integers or floating point numbers.

{ "type": "number" }
    

Numeric types (Contd.)

• Multiples
• Numbers can be restricted to a multiple of a given number, using the multipleOf
• It may be set to any positive number.
• Range
• Ranges of numbers are specified using a combination of the minimum and maximum keywords

Object

• Objects are the mapping type in JSON.

{ "type": "object" }
      

• Properties
• The properties (key-value pairs) on an object are defined using the properties keyword.
• The value of properties is an object, where each key is the name of a property and each value is a JSON schema used to validate that property.
• properties: Object<String, Schema>

Object (Contd.)

{
  "type": "object",
  "properties": {
    "number":      { "type": "number" },
    "street_name": { "type": "string" },
    "street_type": { "type": "string",
                     "enum": ["Street", "Avenue", "Boulevard"]
                   }
  }
}
    

Object (Contd.)

• The additionalProperties keyword is used to control the handling of extra stuff
• properties whose names are not listed in the properties keyword.
• By default any additional properties are allowed
• If additionalProperties is an object, that object is a schema that will be used to validate any additional properties not listed in properties.

{
  "type": "object",
  "properties": {
    "number": { "type": "number" },
    "street_name": { "type": "string" },
    "street_type": { "enum": ["Street", "Avenue", "Boulevard"] }
  },
  "additionalProperties": { "type": "string" }
}
    

{
  "type": "object",
  "properties": {
    "number": { "type": "number" },
    "street_name": { "type": "string" },
    "street_type": { "enum": ["Street", "Avenue", "Boulevard"] }
  },
  "additionalProperties": false
}
    

Object (Contd.)

• By default, the properties defined by the properties keyword are not required.
• The required keyword takes an array of zero or more strings.

Object (Contd.)

{
  "type": "object",
  "properties": {
    "name":      { "type": "string" },
    "email":     { "type": "string" },
    "address":   { "type": "string" },
    "telephone": { "type": "string" }
  },
  "required": ["name", "email"]
}
    

Array

• Arrays are used for ordered elements.
• In JSON, each element in an array may be of a different type.

{ "type": "array" }
      

• List validation is useful for arrays of arbitrary length where each item matches the same schema.
• For this kind of array, set the items keyword to a single schema that will be used to validate all of the items in the array.

Array (Contd.)

{
  "type": "array",
  "items": {
    "type": "number"
  }
}
    

Example

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Product",
  "type": "object",
  "properties": {
    "id": {
      "type": "number",
      "description": "Product identifier"
    },
    "name": { "type": "string" },
    "price": { "type": "number", "minimum": 0 },
    "tags": {
      "type": "array",
      "items": { "type": "string" }
    },
    "stock": {
      "type": "object",
      "properties": {
        "warehouse": { "type": "number" },
        "retail": { "type": "number" }
        }
      }
   }
}
    

Example (Contd.)

{
  "id": 1,
  "name": "Foo",
  "price": 123,
   "tags": [
    "Bar",
    "Eek"
  ],
  "stock": {
   "warehouse": 300,
   "retail": 20
  }
}
    

JSON Schema Validator

• Validators available
• As Online tools
• As programming languages libraries
• Standalone tools

Example Message Parser

{
  "type": "object",
  "properties": {
    "messages": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "from": {
            "type": "string"
          },
          "to": {
            "type": "string"
          },
          "body": {
            "type": "string"
          }
        }
      }
    }
  }
}
      

function parseJSON() {
  const input = document.getElementById("json-in-2").value;
  const jsonData = JSON.parse(input);
  const out = document.getElementById("json-out-2");

  out.replaceChildren();

  for (const msg of jsonData.messages) {
    const p = document.createElement("p");
    // textContent, never innerHTML: this text came from the user
    p.textContent = `${msg.from} sent to ${msg.to}: ${msg.body}`;
    out.append(p, document.createElement("hr"));
  }
}
document.getElementById("json-btn-2").onclick = parseJSON;
    

gRPC

• Thrift's idea, as it is usually met today: describe the service once, generate the client and the server
• Protocol Buffers for the payload, HTTP/2 for the transport
• HTTP/2 brings multiplexing and, unlike request/response ReST, streaming in both directions

Service Definition

syntax = "proto3";

package course;

message Student {
  string first_name = 1;
  string last_name  = 2;
  string id         = 3;
}

message CourseRequest {
  string name = 1;
}

message Course {
  string name = 1;
  string teacher = 2;
  repeated Student students = 3;
  int32 capacity = 4;
}

service Registration {
  rpc GetCourse(CourseRequest) returns (Course);
}
    

Field Numbers

• The numbers, not the names, are what goes on the wire, so a field can be renamed freely
• A number must never be reused: an old client would read the new field as the old one
• Unknown fields are preserved rather than rejected, which is what makes rolling upgrades possible

On the Wire

One Call over HTTP/2

Four Kinds of RPC

Compared to JSON over ReST

• Smaller and faster: binary, and no field names in the payload
• A real contract: the .proto is the schema, and the compiler enforces it on both sides
• Not readable with curl, and not directly callable from a browser without a proxy
• Rule of thumb: gRPC between services, JSON at the edge facing browsers