Save to a file: B64E, Zlib compressed MessagePack.

ASTER::ACTION::JSON::Serialize w/ MessagePack & Save to a file

JSON data currently in memory is stored in a file.

Binary data is stored as text data encoded in Base64. Since the binary is compressed with zlib before being serialized, the data size is reduced.

Parameter.1: Destination path of a file

ASTER rev.1.0.5 allows users to freely configure the data storage location. However, due to security concerns, future updates may change the specifications and restrict file storage destinations.

"./aster/savedata/b64eMsgPack.txt"

JSON is a text-based data format, but MessagePack is a format that allows handling it in binary form.

The conversion to MessagePack uses nlohmann/json’s to_msgpack.

Relevant URL :

• MessagePack - JSON for Modern C++

Generally, even without compression, converting JSON to MessagePack results in a smaller data size compared to JSON.

Data that contains a large number of numerical values benefits from binary representation, allowing for efficient reduction of data size.

For example, in nlohmann/json, there is a default setting when storing numbers in JSON, where all floating-point numbers are serialized as Double type strings.

Relevant URLs

• nlohmann::basic_json::number_float_t
• RFC 8259

/// a class to store JSON values
/// @sa https://json.nlohmann.me/api/basic_json/
template<template<typename U, typename V, typename... Args> class ObjectType =
         std::map,
         template<typename U, typename... Args> class ArrayType = std::vector,
         class StringType = std::string, class BooleanType = bool,
         class NumberIntegerType = std::int64_t,
         class NumberUnsignedType = std::uint64_t,
         class NumberFloatType = double,// <-------------------------------------------:
         template<typename U> class AllocatorType = std::allocator,
         template<typename T, typename SFINAE = void> class JSONSerializer =
         adl_serializer,
         class BinaryType = std::vector<std::uint8_t>, // cppcheck-suppress syntaxError
         class CustomBaseClass = void>
class basic_json;

MessagePack stores numerical data in the most optimal format—whether as integers or floating-point numbers—when converting to binary. This alone helps reduce data size.

JSON inherently includes essential characters such as " (double quotes), : (colon), and {} (curly brackets) to define its data structure. MessagePack replaces these structural elements with binary identifiers and tags, effectively reducing redundant notation.

Comparison of Compression Technologies

Starting from ASTER rev.1.0.5, data compression and decompression support zlib and LZ4. Since the decompression process for compressed data is publicly available as a Python 3 implementation, stored data can be used in other environments.

Python3 Implementation: Zlib version

import sys
import base64
import zlib
import msgpack
import json

# PowerShell >> python base64_zlib_msgPack_python3_.py | Out-Host

# Function to safely decode byte string values in JSON
def decode_bytes(obj):
    if isinstance(obj, dict):
        return {k: decode_bytes(v) for k, v in obj.items()}
    elif isinstance(obj, list):
        return [decode_bytes(v) for v in obj]
    elif isinstance(obj, bytes):
        try:
            return obj.decode('utf-8')  # バイト型の文字列をデコード
        except UnicodeDecodeError:
            return obj.hex()  # 失敗時は16進表記で返す
    else:
        return obj

# Data encoded in Base64 (Replace as needed)
base64_encoded_data = "eJxrWlZcUpSZl77+2fSlz+asebFq3ouFK57371uZmJOeX5RZkpE7ebFPlMkK5/zcgqLU4uJVLqnJUCYA3kkehw=="

# Base64 decoding
try:
    binary_data = base64.b64decode(base64_encoded_data)
except base64.binascii.Error:
    print("Error: Invalid Base64 encoding")
    sys.exit(1)

# Decompression using zlib
try:
    decompressed_data = zlib.decompress(binary_data)
except zlib.error:
    print("Error: Invalid zlib compressed data")
    sys.exit(1)

# Decoding using MessagePack
try:
    original_data = msgpack.unpackb(decompressed_data, raw=False)
except msgpack.ExtraData:
    print("Error: Invalid MessagePack format")
    sys.exit(1)

# Decode bytes in MessagePack (convert binary data to readable format)
print("Decoded data:")
decoded_data = decode_bytes(original_data)  # Converts binary values in MessagePack to UTF-8 or hex
print(json.dumps(decoded_data, ensure_ascii=False, indent=2))

Since the data is not encrypted, it remains editable for general developers. However, for regular users, modifying the stored data becomes difficult, making it somewhat more resistant to tampering compared to saving JSON in plain text.