Sync Your Stacks Node and API Services Faster With the Hiro Archive

While Hiro runs a Stacks blockchain node and offers an instance of the Stacks Blockchain API for free—something we’ve put a lot of work into to scale to hundreds of millions of monthly requests—many developers and teams want to run their own node or instance of the Stacks API.

We support that! More running nodes and APIs makes the overall ecosystem more resilient. You may not trust us and want to have your own source of truth (no offense taken, we get it), and there are plenty of use cases where you may need to run your own node or API instead of relying on Hiro’s services if your app hits our API rate-limits.

While running a node or API is straightforward enough, it can take a while to get going. For example, syncing a node from genesis takes a long time—depending on your internet speed and computer specs, it can take anywhere from a few days to a little more than a week. And syncing an instance of the Stacks API can take even longer (there’s a lot of work that goes into indexing complex datasets).

This isn’t a problem for many developers. If you are spinning up a node for the first time and tinkering, start your sync and check back in in a few days. But sometimes that delay can be an issue.

Let’s say you run an app and have users that are dependent on you having a functioning node or API. What happens when your node goes down?

Maybe the data gets corrupted (rare but it happens), maybe there is a consensus-breaking protocol change that requires you to sync your node from genesis (such as recently happened with the Stacks 2.1 and 2.2 upgrades), or maybe it’s something else. Regardless, you have to take action to restore service, and having an outage of several days (or a month!) can severely affect your end users.

You need something faster. That’s where the Hiro Archive comes in.

What Is the Hiro Archive?

The Hiro Archive is a collection of snapshots of the Stacks network and the various Hiro APIs that are taken every 24 hours. These are effectively backups that can jumpstart your sync to the present day, and each snapshot includes all of the data on a Stacks node (or API instance) at that point in time. 

Using a snapshot from the Hiro Archive enables you to sync a new node dramatically faster than the process described above. Instead of taking days or weeks, it can take minutes or hours, all depending on how quickly you can download and unzip a snapshot (today in the ballpark of 40GB).

These snapshots are hosted in a publicly-readable Google Cloud bucket at https://archive.hiro.so/ (admittedly, this is a barebones interface). The archive includes a variety of snapshots:

In the archive, we take snapshots of both:

• Stacks mainnet
• Stacks testnet

And inside each of the mainnet and testnet folders, you'll find snapshots of:

• Stacks Blockchain: these snapshots are for running a Stacks blockchain node. 
• Stacks Blockchain API: these snapshots are TSV files of the raw unprocessed data from a blockchain node. There are situations where you may want the raw API data, such as when a Postgres database archive for a particular API versions is not available or when it cannot be used for any reason. Or maybe you want to minimize trusting Hiro and would simply rather process the data yourself.
• Stacks Blockchain API PG: these snapshots take info from Hiro’s hosted postgres database, so the data has already been cleaned and organized. This is the quickest and most direct snapshot and is a much faster restoration process than using the raw data TSV file described above. We generally recommend starting with this method.
• Token Metadata API PG: similar to the snapshot directly above, but for Hiro's Token Metadata API.

And for every snapshot, we offer both the file of the snapshot itself, as well as a file containing the sha256 hash of the snapshot. You can use this sha256 file to verify that the backup you downloaded is not corrupted or incomplete. We’ll get into how to do that in the next section.

Before we get into the details, a few more notes on using the Hiro archive:

• Running an instance of the API is dependent on running a Stacks blockchain node. If you are using the Hiro Archive, you should pick snapshots from the same date to sync your node and API; otherwise your sync may fail.
• Snapshots are organized by version and then date, so you can easily find the most recent snapshot.

How to Use the Hiro Archive

Booting up a node with the Hiro Archive is a straightforward process. If you need guidance on setting up a node or API, start with these resources:

• If you want to run a Stacks node, reference documentation here.
• If you want to run a Stacks blockchain API, reference Stacks documentation here and Hiro documentation here.

As for leveraging the Hiro Archive, all you need to do is download your snapshot from https://archive.hiro.so/. Once it finishes downloading, extract the file(s) if necessary, verify it, and import the file into your running node or API.

For detailed instructions on how to use the archive, please reference our docs.

How to Verify a Snapshot

Using the Hiro archive is a matter of trust. You are trusting that we are taking accurate snapshots, that we have not tampered with those snapshots, and that we are processing API data correctly (for the stacks-api-pg snapshots). However, there are steps you can take to verify the correctness of those snapshots.

Comparing Snapshot to SHA256

As mentioned above, for every snapshot in the Hiro Archive, we also provide a sha256 file that is a hash of that dataset. When downloading a snapshot, you can also download the associated shasum file and can use that shasum to make sure you downloaded the snapshot correctly.

To do that, you can use the sha256 command line in the final file to generate a hash and see if your downloaded file matches what is in the archive.

echo "$(cat  | awk '{print $1}')  " | shasum --check
: OK

# Example
echo "$(cat mainnet-stacks-blockchain-api-latest.sha256 | awk '{print $1}')  mainnet-stacks-blockchain-api-latest.gz" | shasum --check

# If the integrity check succeeds, a log will be printed indicating so
mainnet-stacks-blockchain-api-latest.gz: OK

# Otherwise a log will be printed indicating failure
mainnet-stacks-blockchain-api-latest.gz: FAILED
shasum: WARNING: 1 computed checksum did NOT match

If it matches, your snapshot is correct! If it doesn’t match, your download may have been corrupted, and you should check your network connection and try again (or try a different network connection if the issue persists).

Comparing Snapshot to Local /v2 Endpoint

You can also verify the correctness of a snapshot from the Hiro Archive, by comparing it to your local /v2 endpoints. To do that, you can use the following:

To compare the Stacks Blockchain node, you would compare values from your local device found at <code-rich-text>localhost:20443/v2/info<code-rich-text> to Hiro’s endpoints at <code-rich-text>api.hiro.so/v2/info<code-rich-text>. 

And for the Stacks Blockchain API, you would compare values from your local device at <code-rich-text>localhost:3999/extended/v1/status<code-rich-text> to Hiro’s endpoints at <code-rich-text>api.hiro.so/extended/v1/status<code-rich-text>.

From there, you can compare the block heights between your local instance and Hiro’s APIs by hitting the URLs above. The block heights shouldn’t match right away, but they should be relatively close. This is because we only take backups every 24 hours. During that time, the block height for our services will continue to progress. Once your node or API is running for a short while, it should catch up to the same block height they see from Hiro’s services.

Learn more about using the Hiro Archive in our documentation.

Conclusion

We host the Hiro Archive to make your life easier and take some work off your plate the next time you need to sync a node or API instance. We hope you find it helpful. If you have any questions on how to use the Hiro Archive, please reach out to us on the HIRO-PUBLIC channels on Discord.