Blockchain is primarily used to store state information. Storage of assets is one of the inherent problem that you as a blockchain developer has to tackle while creating scalable Dapps. Using solutions like amazon s3 defeats the very purpose of blockchain – decentralisation. Filecoin attempts to solve this problem and offer reliable persistent decentralised storage. In this post we shall look at how can we store and retrieve data using Filecoin protocol.
Filecoin and IPFS
“Doesn’t IPFS solve the same problem?” you may ask.Filecoin and IPFS are two separate, complementary protocols, both created by Protocol Labs. IPFS allows peers to store, request, and transfer verifiable data with each other, while Filecoin is designed to provide a system of persistent data storage. Under Filecoin’s incentive structure, clients pay to store data at specific levels of redundancy and availability, and storage providers earn payments and rewards by continuously storing data and cryptographically proving it.
In short: IPFS addresses and moves content, while Filecoin is an incentive layer to persist data.
Filecoin Architecture
Filecoin is a peer-to-peer network that stores files on the internet, with built-in economic incentives to ensure files are stored reliably over time. Available storage and pricing is not controlled by any single company. Instead, Filecoin facilitates open markets for storing and retrieving files that anyone can participate in.
The two main entities in the protocol are Miners and Storage clients. Miner is a person or organization that supplies the hardware required to store data on the protocol in exchange for an incentive. Storage clients are people like you and me who want to store data on the Filecoin network. When a storage client uploads a file to the protocol it is accepted by a miner and the Filecoin protocol constantly ensures if the miners are storing the file correctly. At the time of retrieval the end user pays to retrieve the stored file as explained in the diagram below:
Before we go deeper into the protocol lets understand the Key Terms and phrases in the Filecoin protocol.
| Word | Definition |
|---|---|
| Address | A string of letters and numbers that other users can send FIL to. |
| Block explorer | A service, usually a website, that lets you view details of a blockchain such as transactions, deals, and addresses. |
| Deal | An agreement between two computers about what to do with some data. |
| FIL | The shorthand representation of the filecoin cryptocurrency. For example: We charge 0.5 FIL per GiB. |
Filecoin (upper-case F) |
The network that transactions and storage deals take place on. For example: Museums can use the Filecoin network to store their digital archives. |
filecoin (lower-case f) |
The cryptocurrency that the Filecoin network runs on. For example: You can use filecoin to pay for your transactions. |
| Miner | An alternate name for a storage provider. |
| Private key | A string of letters and numbers that programs use to interact with the Filecoin network. Keep your private key safe, and don’t share it with anyone. |
| Storage deal | An agreement between a storage provider and a client about what data to store, how long for, and how much the storage provider can charge for storage. |
| Retrieval deal | An agreement between a storage provider and a client about how much the storage provider can charge to send data to a client. |
| Storage client | The user that wants to store something on the Filecoin network. In this tutorial, you are the storage client. |
| Storage provider | A computer on the Filecoin network offering storage space to other users who want to store data. Storage providers are sometimes called miners. |
| Wallet | A collection of addresses. Think of each wallet as a folder and each address as a single file in that folder. |
Now that we have our foundations covered, lets look how can we leverage Filecoin to store and retreive data over the blockchain.
Setting up your system
To connect to the Filcoin blockchain you need to run a Node on your computer simliar to how a web browser connects you to the Internet. Programs that interact with the Filecoin network are called implementations, and Lotus is a command-line interface (CLI) implementation which lets you run a Filecoin node. There are other implementation being created alongside Lotus, however Lotus is the only Filecoin implementation created and maintained by Protocol Labs.
A Lotus full-node is a computer running the lotus daemon. Full-nodes are unique because they have complete access to the Filecoin blockchain. The computer specifications required to run a Lotus full-node are relatively high and might be out of reach for most end-user laptops and PCs.
Usually, we’d have to spin up a full-node, but we’re going to use a Lotus full-node provided by Protocol Labs for this tutorial. This node, called api.chain.love, is only for practice sessions like this tutorial and should not be relied upon for any production or development purposes.
Installing a lite-node
Lotus full node is a very resource intensive program and most general computers may not meet the hardware requirements for it. Instead for the purpose of this tutorial you should install lite-node which is a light weight client that helps you connect to a Lotus full node.
Follow installation instruction for macOS and Ubuntu to get started with lite-node.
Once you install lite-node run the following command to run the lite-node:
FULLNODE_API_INFO=wss://api.chain.love lotus daemon --lite
Create a new wallet using the following command:
lotus wallet new
Lotus outputs your public address. Public addresses always start with f1
Export your private key
lotus wallet export f1... > my_address.key
It’s important to keep this file safe. If anything happens to your Lotus node, you can still access your funds using this file.
Warning
Filecoin is optimized for public data and doesn’t yet support access controls. If storing private data, ensure you encrypt it before storage to ensure it remains unreadable by anyone without the ability to decrypt it. Keep in mind that if a vulnerability is found in your encryption process at any point in the future, then your data may be compromised.
Uploading data to Filecoin
Lets upload a file in our local system to Filecoin protocol. Navigate the the directory that contains the file and enter the following command:
lotus client import hello-world.docx
Lotus creates a directed acyclic graph (DAG) based off the payload. This process takes a few minutes. Once it’s complete, Lotus will output the payload CID.
Import 3, Root bafykb...
Make a note of the CID bafykb... This is your Data CID. We’ll use it in an upcoming section. Next up we need find a miner to store our file fore us. To find a miner:
- Go to plus.fil.org/miners.
- Using the table, find a couple of storage providers that suit your needs. Try to find storage providers that are geographically close to you.
- Once you have found a couple of suitable storage providers, make a note of their miner IDs from the Storage Provider ID column:
Next lets create a Storage Deal
lotus client deal
The interactive deal assistant will now ask you some questions
Data CID (from lotus client import): bafykbz...
Wait for Lotus to finish calculating the size of your payload. Lotus calculates this size by counting the individual bits in your payload to ensure that the size is accurate.
Enter the deal duration and miner address when prompted. Finally confirm the transaction and you should get deal CIDs which you should keep note of.
Once the data has been sent to the storage clients, the storage deals can take up to 24 hours to complete. You can check the progress of your deals by running the following command:
lotus client list-deals --show-failed
DO NOT TURN OFF YOUR LOTUS NODE unleass the deal status has reached “StorageDealActive”
The table below indicates various deal statuses:
| State | Description |
|---|---|
| StorageDealError | There has been an unforeseen error. No further updates will occur. |
| StorageDealFailing | Something has gone wrong in a deal. Once data is cleaned up, the deal will finalize. |
| StorageDealProposalNotFound | Your full-node cannot find the deal you are looking for. This could be because it doesn’t exist, or your full-node is out of sync. |
| StorageDealProposalRejected | The storage provider, has chosen not to accept this deal. The storage provider may have provided a reason alongside this status message, but not always. |
| StorageDealRejecting | The storage provider has rejected the deal. This comes immediately before StorageDealProposalRejected. |
| StorageDealSlashed | The data was in a sector, and the storage provider got slashed for failing to prove that the data was available. |
Key terms to understand when uploading file to Filecoin
| Variable | Description | Example |
|---|---|---|
| Data CID | The content identifier (CID) of the data that you want to store using Filecoin. | bafk2bzaceajz56zudni2hli7id6jvvpo5n4wj5eoxm5xwj2ipthwc2pkgowwu |
| Miner ID #1 | The unique identifier for each storage provider. You need to have two storage provider IDs for this tutorial. | f01000 |
| Miner ID #2 | The unique identifier for each storage provider. You need to have two storage provider IDs for this tutorial. | f01000 |
| Deal CID | The content identifier (CID) for a deal made with a storage provider. | bafyreict2zhkbwy2arri3jgthk2jyznck47umvpqis3hc5oclvskwpteau |
Retrieving a file
As mentioned in the previous section it takes about 24 hours for a file to be successfully uploaded in Filecoin protocol. After that you may retrieve the file using the following command:
lotus client retrieve --miner <MINER ID> <DATA CID> ~/output-file
Note that you need FIL blanace in your Filecoin wallet to be able to retrieve the file.
> Recv: 66.33 KiB, Paid 0.00000000000013584 FIL, ClientEventPaymentSent (DealStatusFinalizing)
> Recv: 66.33 KiB, Paid 0.00000000000013584 FIL, ClientEventComplete (DealStatusFinalizingBlockstore)
> Recv: 66.33 KiB, Paid 0.00000000000013584 FIL, ClientEventBlockstoreFinalized (DealStatusCompleted)
Success
Done! The file must have been now downloaded to your speechified location.
Conclusion and next steps
Filecoin provides a great scalable alternative to traditional centralised storage mechanisms. You can definitely start building Filecoin integrations for your next Dapp and expect persistent and reliable storage. You may want to explore these resources to learn more about Filecoin:
- Filecoin Faq
- IPLD specifies data formats for content-addressed data like the blockchain or the way in which IPFS stores files.
- libp2p provides peer-to-peer network capabilities, connection security and key discovery and data distribution features like the DHT and Pubsub.
- Multiformats define future-proof identifiers and data-types.
- Graphsync and Bitswapenable fast and efficient IPLD data transfers between nodes.
Leave a Reply