Scalable storage for your Dapps using Filecoin

Blockchain is primarily used to store state information. Storage of assets is one of the inherent problem that you as a blockchain developer has to tackle while creating scalable Dapps. Using solutions like amazon s3 defeats the very purpose of blockchain – decentralisation. Filecoin attempts to solve this problem and offer reliable persistent decentralised storage. In this post we shall look at how can we store and retrieve data using Filecoin protocol.

Filecoin and IPFS

“Doesn’t IPFS solve the same problem?” you may ask.Filecoin and IPFS are two separate, complementary protocols, both created by Protocol Labs. IPFS allows peers to store, request, and transfer verifiable data with each other, while Filecoin is designed to provide a system of persistent data storage. Under Filecoin’s incentive structure, clients pay to store data at specific levels of redundancy and availability, and storage providers earn payments and rewards by continuously storing data and cryptographically proving it.

In short: IPFS addresses and moves content, while Filecoin is an incentive layer to persist data.

Filecoin Architecture

Filecoin is a peer-to-peer network that stores files on the internet, with built-in economic incentives to ensure files are stored reliably over time. Available storage and pricing is not controlled by any single company. Instead, Filecoin facilitates open markets for storing and retrieving files that anyone can participate in.

The two main entities in the protocol are Miners and Storage clients. Miner is a person or organization that supplies the hardware required to store data on the protocol in exchange for an incentive. Storage clients are people like you and me who want to store data on the Filecoin network. When a storage client uploads a file to the protocol it is accepted by a miner and the Filecoin protocol constantly ensures if the miners are storing the file correctly. At the time of retrieval the end user pays to retrieve the stored file as explained in the diagram below:

Before we go deeper into the protocol lets understand the Key Terms and phrases in the Filecoin protocol.

Now that we have our foundations covered, lets look how can we leverage Filecoin to store and retreive data over the blockchain.

Setting up your system

To connect to the Filcoin blockchain you need to run a Node on your computer simliar to how a web browser connects you to the Internet. Programs that interact with the Filecoin network are called implementations, and Lotus is a command-line interface (CLI) implementation which lets you run a Filecoin node. There are other implementation being created alongside Lotus, however Lotus is the only Filecoin implementation created and maintained by Protocol Labs.

A Lotus full-node is a computer running the lotus daemon. Full-nodes are unique because they have complete access to the Filecoin blockchain. The computer specifications required to run a Lotus full-node are relatively high and might be out of reach for most end-user laptops and PCs.

Usually, we’d have to spin up a full-node, but we’re going to use a Lotus full-node provided by Protocol Labs for this tutorial. This node, called api.chain.love, is only for practice sessions like this tutorial and should not be relied upon for any production or development purposes.

Installing a lite-node

Lotus full node is a very resource intensive program and most general computers may not meet the hardware requirements for it. Instead for the purpose of this tutorial you should install lite-node which is a light weight client that helps you connect to a Lotus full node.

Follow installation instruction for macOS and Ubuntu to get started with lite-node.

Once you install lite-node run the following command to run the lite-node:

FULLNODE_API_INFO=wss://api.chain.love lotus daemon --lite

Create a new wallet using the following command:

lotus wallet new 

Lotus outputs your public address. Public addresses always start with f1

Export your private key

lotus wallet export f1... > my_address.key

It’s important to keep this file safe. If anything happens to your Lotus node, you can still access your funds using this file.

Warning

Filecoin is optimized for public data and doesn’t yet support access controls. If storing private data, ensure you encrypt it before storage to ensure it remains unreadable by anyone without the ability to decrypt it. Keep in mind that if a vulnerability is found in your encryption process at any point in the future, then your data may be compromised.

Uploading data to Filecoin

Lets upload a file in our local system to Filecoin protocol. Navigate the the directory that contains the file and enter the following command:

lotus client import hello-world.docx

Lotus creates a directed acyclic graph (DAG) based off the payload. This process takes a few minutes. Once it’s complete, Lotus will output the payload CID.

Import 3, Root bafykb...

Make a note of the CID bafykb... This is your Data CID. We’ll use it in an upcoming section. Next up we need find a miner to store our file fore us. To find a miner:

1. Go to plus.fil.org/miners.
2. Using the table, find a couple of storage providers that suit your needs. Try to find storage providers that are geographically close to you.
3. Once you have found a couple of suitable storage providers, make a note of their miner IDs from the Storage Provider ID column:

Next lets create a Storage Deal

lotus client deal

The interactive deal assistant will now ask you some questions

Data CID (from lotus client import): bafykbz...

Wait for Lotus to finish calculating the size of your payload. Lotus calculates this size by counting the individual bits in your payload to ensure that the size is accurate.

Enter the deal duration and miner address when prompted. Finally confirm the transaction and you should get deal CIDs which you should keep note of.

Once the data has been sent to the storage clients, the storage deals can take up to 24 hours to complete. You can check the progress of your deals by running the following command:

lotus client list-deals --show-failed

DO NOT TURN OFF YOUR LOTUS NODE unleass the deal status has reached “StorageDealActive”

The table below indicates various deal statuses:

Key terms to understand when uploading file to Filecoin

Retrieving a file

As mentioned in the previous section it takes about 24 hours for a file to be successfully uploaded in Filecoin protocol. After that you may retrieve the file using the following command:

lotus client retrieve --miner <MINER ID> <DATA CID> ~/output-file

Note that you need FIL blanace in your Filecoin wallet to be able to retrieve the file.

> Recv: 66.33 KiB, Paid 0.00000000000013584 FIL, ClientEventPaymentSent (DealStatusFinalizing)
> Recv: 66.33 KiB, Paid 0.00000000000013584 FIL, ClientEventComplete (DealStatusFinalizingBlockstore)
> Recv: 66.33 KiB, Paid 0.00000000000013584 FIL, ClientEventBlockstoreFinalized (DealStatusCompleted)
Success

Done! The file must have been now downloaded to your speechified location.

Conclusion and next steps

Filecoin provides a great scalable alternative to traditional centralised storage mechanisms. You can definitely start building Filecoin integrations for your next Dapp and expect persistent and reliable storage. You may want to explore these resources to learn more about Filecoin:

• Filecoin Faq
• IPLD specifies data formats for content-addressed data like the blockchain or the way in which IPFS stores files.
• libp2p provides peer-to-peer network capabilities, connection security and key discovery and data distribution features like the DHT and Pubsub.
• Multiformats define future-proof identifiers and data-types.
• Graphsync and Bitswapenable fast and efficient IPLD data transfers between nodes.