Subsquid
Subsquid is an open-source framework for building GraphQL APIs for blockchain state history.
All edits
Subsquid is a query node framework for Substrate-based blockchains.In very simple terms, Subsquid can be thought of as an ETL tool, with a GraphQL server included.
Subsquid's multi-layer approach aims to pre-process and decode raw chain data and store it for easier access by the query nodes, providing increased performance over direct RPC calls.
A Squid Archive can constantly synchronize raw data with the chain and by simply defining the custom data model and Event handlers in the Squid project, it is possible to perform complex queries on top of it.
For example, expired Kusama Treasury spending proposals are pruned from the state of the Kusama blockchain, so querying one-year-old proposals would be problematic. With the traditional approach, it would be necessary to track the evolution of the state by sequentially applying the Treasury events and extrinsics in each historical block.
Subsquid takes a multi-layered approach that separates raw data ingestion from data processing and presentation.
This is very useful for the vast majority of developers, as it adds a layer of abstraction, taking care of data ingestion and synchronization, and exposing decoded Substrate chain entities (Events and Extrinsics, as well as entire Blocks), for developers to process.
The Squid data pipeline consists of two core components:
- Archive: data source, can be shared by multiple query nodes
- Squid: the data transformation part of the pipeline
An Archive can be thought of as a blockchain crawler, a data pipeline that systematically browses blockchains and that is operated by a decentralized network of squid archive node operators for the purpose of blockchain indexing.
In the context of a Squid query node, a Processor is responsible to get data from an Archive, transform it, and store it in the database.
The GraphQL Server is a separate web server providing a GraphQL API for the entities in the data store.
Archives should not be confused with Archive nodes, although the concept is vaguely similar, since both preserve the full blockchain history, without any pruning. The difference is that our Archives are special services, with specific endpoints, tailored for data retrieval.
An Archive is responsible for continuously ingesting raw data from the blockchain, process blocks, and save them in a database, along with Events and Extrinsics, for easier access through GraphQL APIs. Its main purpose is to provide a service, to be a high-performance, higher level data source for Squid(s).
An Archive includes the following components and services:
- Substrate archive
- Redis data storage
- Status service
- GraphQL Gateway
Ultimately, the Substrate Archive extracts block information, Events, and Extrinsics, then writes them to a Postgres database, while the status updates are saved in a Redis key-value database. Archived data is available for clients' queries, thanks to a GraphQL server running as part of every Squid Archive node.
An Archive can be shared by multiple Squids, which means it is possible to segment how data is presented, based on the user's needs, without having to replicate the data source.
To launch your own Archive for a particular Blockchain, head over to our dedicated page on the topic.
A Squid is a pipeline for transforming and presenting Substrate blockchain data. It consists of:
- Processor: each node has one processor but can have multiple if connected to multiple chains
- Database: a PostgreSQL database where processed data is stored
- GraphQL Server: every query node comes with a gateway to present processed data
A Squid replaces direct gRPC node access with more performant API calls to Archive gateways, allowing bandwidth reduction and quick synchronization of the API with the historical on-chain data. It can be run locally, on a server, or deployed to the Cloud using our SaaS solution.
Thanks to the Subsquid framework, it is possible to:
- define the database schema, data type, and entity definitions
- transform and store chain data efficiently
- present it thanks to the included GraphQL server
Subsquid provides developers with a high-level GraphQL-like schema and codegen tools to model blockchain data with Entities. One of the advantages, here, is the removal of boilerplate code to unbox. A more significant advantage is certainly the elimination of incorrect data types due to wrong decoding and missing parameters.
Even more importantly, when handling unstructured data from events, the data format may change from one block to the next, due to runtime upgrades. Without type safety and automation guaranteed by typegen, managing these alterations would be a nightmare.
The Processor extracts data from an Archive Endpoint and does Transform-Load operations, saving it to the database. The transform-load logic is fully custom and defined by the developer.
Once the schema and the mappings are set up, and the node is launched, it will start the continuous scan of the blockchain, processing the events through the Event handlers and updating the entities in the database.
The API requests are resolved by the GraphQL server by sourcing data from the Processor database. OpenCRUD filtering, entity relations, pagination, and text queries are supported out-of-the-box by the API.
February 14, 2022
Docs, docs, docs…
Updating our documentation has been a central task over the past few days, so, at the risk of sounding repetitive, I feel it’s worth saying a few words about the reasoning behind these changes.
I tend to take a methodical approach to projects, so prior to starting this task I took some time to do some research. I came across this documentation framework and found it really interesting — the division into four quadrants is very close to my natural approach towards docs as a consumer and something that was missing in our materials up until now.
My first priority was to improve the pages that explained the most basic concepts of Subsquid (the ‘explanation’ quadrant). For developers who are new to blockchain technology there are just so many new notions to absorb!
Of course, it wouldn’t be possible to explain every little detail about blockchain in this documentation, but we can at least try to provide the minimum necessary for those who are just starting to build on Subsquid.
I have also tried to retrace my own steps, scattering external links like breadcrumbs throughout our pages. Developers can follow these links to become better acquainted with any new or unfamiliar topics.
Getting up to speed, one line of code at a time
I ended last week by telling myself:
“If I am to be advocating for Subsquid developers, my first priority should be to know the ins and outs of the product itself!”
The task of restructuring, updating, and improving our documentation really helps in this sense, first because I need to read it, and second because while writing, all sorts of questions start to pop up.
Let’s take the example of our multi-layered approach. OK, so our architecture is divided into Archives and Squids. The former collects raw blockchain data and makes it exploitable, while the latter leverages this data and performs custom transformation on it. That’s easy enough to understand.
But while trying to explain it in more detail, I found that my initial level of understanding simply was not enough.
The inner workings of our Archive became clearer when I looked into how the IndexBuilder , BlockProducer, IndexerStatusService, and PooledExecutor classes interacted with each other:
IndexBuilder orchestrates blockchain data ingestion by fetching information from the BlockProducer
It then instantiates PooledExecutor so that multiple workers extract raw data, wrap it in TypeScript classes representing ORM entities (namely the Block itself, Extrinsics, Events), and saves it in the database.
It emits events, so that the IndexerStatusService is updated on the synchronisation status.
In short, it provides continuous ingestion of blockchain data, conveniently storing it in a more exploitable way.
As for the ‘Squid’ part of the ETL pipeline, it was quite clear from the start that the Processor is the key, so I went to take a look at the SubstrateProcessor class. In short, this class sets up ingestion metrics, groups blocks in batches, then starts a loop that ingests those batches.
The nice thing about it however is that it sets up methods for the developer to customise data processing by adding functions as Event Handlers, Extrinsic Handlers, and pre- or post-Block hooks.
I have used Django in the past and I instantly connected Block hooks to the concept of middleware in web servers (NestJS has a great explanation for its middleware — what a coincidence, I am taking inspiration from them for our docs!).
On the other hand, for the Handlers the pattern is akin to a Publish-Subscribe, where when calling
addEventHandler(eventName: QualifiedName, fn: EventHandler)
the developer has to specify the eventName as an argument. While processing a block, if that event is encountered, the Handler is triggered by calling the function fn specified as the second argument.
Once again, quite a clever use of tried and true software design patterns. It really does result in a polished implementation that is quite intuitive.
My first Dev interview!
As part of our Business Development initiative we are establishing various partnerships that often entail the development of software integrations. One of my roles during this process is to hold special meetings (called ‘developer interviews’) that help to make sure we gather all necessary information about how our technology is being implemented. This way I can communicate these things to our community later on.
Simply put, a dev interview is a meeting where the developer advocate (me!) sits down with developers to go over the integration that they are responsible for. During these meetings we discuss the work that needs to be done and highlight any noteworthy customisations or trade-offs that need to be made.
I had my very first of these meetings this morning, and you’ll be able to read an announcement about this upcoming integration in just a few days!
Join our Community!
To find out more about Subsquid, join us on our Discord server and chat to one of our helpful Subsquid team members or other users. For those who want some more background information on Subsquid and what it’s all about, our website and Medium page are the perfect places to start.
My hope for the dev interview process is that it will lead to interesting case studies that we can include in our documentation as neatly detailed ‘How-tos’ (referred to as Recipes in our docs). This is something I am very much looking forward to. Most importantly, dev interviews give me an excuse to get my hands dirty and do some coding .
Sounds simple, doesn’t it? That’s because it’s a very elegant implementation, in my opinion. Kudos to Eldar, our Senior Core Dev!
That is why, at some point, I took to old habits: I went to explore our codebase. And that’s where I finally started connecting the dots (Polkadot pun intended).
So, after settling in and getting to know my colleagues and establishing the first tasks I’d be getting into (spoiler: it’s documentation! ), it was finally time to sink my teeth into the tech.
One can think of these quadrants as the lens through which our new documentation should be viewed. Each part of the updated materials will fall into one of these categories and will be easy to find, depending on the needs of the reader.
February 11, 2022
Subsquid
Subsquid is an open-source framework for building GraphQL APIs for blockchain state history.
Subsquid
Subsquid is an open-source framework for building GraphQL APIs for blockchain state history.
Subspace Network has annonced Subspace Network Ambassador Program.
Subspace is an open, scalable platform for both storage and compute that is fully interoperable with any layer one.
Ambassador Benefits
Provide direct contributions to something you are interested in, gaining amazing experience along the way:
Resources: Direct access to the Subspace team and Lead Ambassadors for education and support
Early Access: Priority access to the Subspace testnets, apps, events, hackathons,and more
Networking: Access to private ambassador group and regular calls with the team and other ambassadors
How you can contribute
You can choose an area to contribute, depending on your skills and the area in which you want to grow:
Community: help to build a blossoming community of farmers
Technical: be among the first cohort of farmers and provide feedback
Content: write articles and create videos with Subspace support
Design: сreate NFTs, stickers, visual graphics, and other artwork
Translations: create accurate translations to help increase Subspace`s reach
Leadership: help set the standards of Subspace community,
Establishing the way forward for a truly decentralized protocol
Subspace Network has annonced Subspace Network Ambassador Program.
Subspace is an open, scalable platform for both storage and compute that is fully interoperable with any layer one.
Ambassador Benefits
Provide direct contributions to something you are interested in, gaining amazing experience along the way:
Resources: Direct access to the Subspace team and Lead Ambassadors for education and support
Early Access: Priority access to the Subspace testnets, apps, events, hackathons,and more
Networking: Access to private ambassador group and regular calls with the team and other ambassadors
How you can contribute
You can choose an area to contribute, depending on your skills and the area in which you want to grow:
Community: help to build a blossoming community of farmers
Technical: be among the first cohort of farmers and provide feedback
Content: write articles and create videos with Subspace support
Design: сreate NFTs, stickers, visual graphics, and other artwork
Translations: create accurate translations to help increase Subspace`s reach
Leadership: help set the standards of Subspace community,
Establishing the way forward for a truly decentralized protocol
Find more companies like Subsquid
