In a major stride towards enhancing the functionality of Web3 and blockchain technology, Space and Time has launched the beta version of Python Data Jobs, a revolutionary solution designed to bridge the gap in handling long-running Python jobs. This groundbreaking development comes as a response to the growing need for seamless integration of Python in data processing, analytics, and smart contract functionalities within the Web3 ecosystem.
Challenges Addressed and Solutions Provided
If you’ve been following developments in the realm of Space and Time, you might be aware of its groundbreaking achievement – the creation of the first and only Zero-Knowledge (ZK) proof for SQL. The Proof of SQL stands as a remarkably potent tool, empowering smart contracts to securely retrieve and process data using SQL. This cryptographic assurance opens up a plethora of innovative possibilities for blockchain technology.
Despite the prowess of SQL as a robust and nearly Turing-complete language, it doesn’t cover every conceivable business scenario. To address the need for custom business logic, especially when deploying arbitrary code becomes essential, Chainlink has ingeniously introduced a solution: Chainlink Functions. These Functions enable the execution of Javascript redundantly on Chainlink nodes, reaching consensus on the output.
Consequently, smart contracts can now seamlessly access ZK-proven analytics and data processing through Proof of SQL, along with efficiently executing scripts using Functions. However, within the Web3 landscape, a particular category of use cases remains unaddressed – long-running Python jobs. Recognizing the significance of Python in the workflows of businesses and developers associated with Space and Time, the team aimed to tackle two key challenges.
Firstly, it sought to empower users to effortlessly employ Python for extracting, transforming, and loading data from their existing databases into Space and Time, all without the need for manual code creation. Secondly, it aimed to establish a cryptographically guaranteed connection between Python jobs and smart contracts. This led to the introduction of Space and Time Python Data Jobs, now available in beta through the Space and Time Studio.
Comprehensive Functionality of Python Data Jobs
Embarking on a transformative journey within the Web3 landscape, Space and Time’s Python Data Jobs introduces a paradigm shift in data processing and smart contract functionalities. As we delve into the intricate capabilities of this solution, a world of seamless integration, simplified ETL processes, and enhanced security unfolds, redefining the way developers and businesses harness the potential of Python in the blockchain ecosystem. Here is how this innovative solution works:
1. Getting data into Space and Time
Python Data Jobs streamline the data acquisition process into Space and Time from any off-chain origin, eliminating the need for manual code creation. Earlier this year, Space and Time introduced AI SQL, a service powered by OpenAI that enables users to formulate natural language prompts such as “show me the top 5 wallets on Sui with the most transactions ordered by balance.”
This prompt is then converted into a SQL query, yielding the desired result. Space and Time has announced that Houston, the AI chatbot within the Space and Time Studio, can now generate straightforward Extract, Transform, Load (ETL) scripts. These scripts are designed to fetch data from various sources, including Web2 databases or Web3 decentralized storage platforms. Houston handles the data preparation process and facilitates its transfer into Space and Time.
By creating a script that connects to databases like PostgreSQL (or alternatives such as Snowflake or IPFS), Houston comprehends the database content, performs necessary transformations, establishes tables in SxT, and incrementally loads data from PostgreSQL into SxT one row at a time. Typically, database migration is a laborious, costly, and time-consuming task that involves Python expertise. With Space and Time’s solution, developers can achieve this seamlessly using natural language input in a single operation.
2. Getting data out of Space and Time
Python Data Jobs also offers the capability to extract data from Space and Time, process it, and transmit it to a smart contract. The reason this has posed a challenge in the realm of Web3 is due to the prolonged execution times associated with Python jobs. Consider a scenario where you have a script tasked with calculating the probability that Bitcoin (BTC) remains above $40k for the rest of the year.
This script must collect data from the markets, process it, and execute a Monte Carlo simulation in Python, a process that may take approximately 20 seconds. When connecting the outcome to a smart contract, it becomes imperative to ensure its tamperproof nature. While consensus-based proving is well-suited for swiftly executing scripts, it becomes less effective for tasks running over an extended duration.
If, for instance, the computation is distributed redundantly across 30 nodes, node 1 might complete the task in 18 seconds, while node 5 finishes in 25 seconds, and node 15 takes 21 seconds. This necessitates the development of a new architecture to address the unique challenges associated with prolonged script execution.
During the Python Data Jobs beta, Space and Time is actively progressing towards achieving this through the implementation of Zero-Knowledge (ZK): specifically, a ZK proof for Python. Currently, the system relies on optimistic security, akin to an optimistic rollup approach. When executing a Python Data Job within SxT, the inputs, outputs, and the code itself undergo hashing and are recorded on a major chain.
The script runs only once, and if the outcome deviates from expectations, users have the option to request a proof. SxT then cryptographically verifies what was executed. Unlike real-time proofing with redundant computation and consensus, Space and Time’s approach involves a single execution followed by hashing all metadata. This process creates a tamper proof audit trail, serving as an incentive for node operators to refrain from tampering with the execution. Space and Time will share more details in the future about the ZK solution under development, which aims to enhance the real-time security of Python Data Jobs.
Real-world Applications and Use Cases
Space and Time’s Python Data Jobs transcend theoretical possibilities, stepping into the practical domain with real-world applications and use cases. These scenarios showcase the versatility and adaptability of Python Data Jobs, illustrating how this innovative solution addresses complex challenges across diverse industries, paving the way for a new era in data processing and smart contract functionalities.
1. Seamless Database Migrations
Python Data Jobs introduces a new era of simplicity to database migrations. Users can instruct Houston to generate Python scripts for complex tasks such as loading data from platforms like Snowflake into Space and Time. For instance, if you state, “generate a Python script to transfer my Snowflake data into SxT,” Houston will request access and produce a Python script.
This script will query Snowflake, retrieve the data, discern the schema, and replicate it to SxT in a single Large Language Model (LLM) inference. An illustrative use case involves Truflation, which ingests extensive real-time inflation data from numerous data feeds (commodities, bond rates, housing, etc.) into storage. Subsequently, Truflation constructs aggregations, such as inflation indexes, to be exposed on-chain via oracles.
Python Data Jobs efficiently processes and prepares these substantial data volumes for aggregation. Another use case is dClimate, which regularly performs Extract, Transform, Load (ETL) operations on weather data from multiple sources, loading this data into IPFS. Python Data Jobs can streamline this process by automating the extraction and transformation of weather data.
2. Complex Calculations for DeFi
Envision if your smart contract could execute intricate off-chain computations, such as predicting the future performance of a cryptocurrency based on diverse market conditions, in a securely tamperproof manner. Python Data Jobs empowers you to integrate sophisticated financial models, akin to those utilized for forecasting price movements or evaluating risk factors, into your smart contract with an optimistic security approach.
This capability enables DeFi protocols to employ more advanced business logic, surpassing the capabilities offered by Proof of SQL. For instance, dYdX conducts calculations for perpetual options/futures pricing off-chain, as these require historical pricing input data and involve complex computations that cannot be executed by smart contracts on-chain.
Python Data Jobs facilitates the execution of these calculations in a tamper proof manner. In another example, 3Commas employs off-chain machine learning models for decision-making in DeFi/CeFi activities (such as swaps, futures, bot trades, etc.) within a centralized compute container environment. Python Data Jobs provides a Web3-native alternative for executing these tasks.
The Way Forward
The beta launch of Space and Time’s Python Data Jobs marks a significant advancement in addressing the evolving needs of Web3 and blockchain technology. By recognizing and tackling the challenges associated with long-running Python jobs within the Web3 landscape, Space and Time has introduced a groundbreaking solution that not only streamlines data processing but also enhances the functionality of smart contracts.
The comprehensive functionality of Python Data Jobs is a testament to the transformative potential it brings to the Web3 ecosystem. From simplifying ETL processes to ensuring the security of Python jobs within smart contracts, Space and Time’s innovative solution opens up new possibilities for developers and businesses alike.
In an effort to encourage users to explore the transformative capabilities of Python Data Jobs, Space and Time is offering free access to the service for one month. Users can initiate their Python Data Jobs journey with Houston on the Space and Time Studio, heralding a new era of seamless integration between Python and Web3 technologies.
