It’s difficult to explain the importance of the internet to humanity. Many have stated that it’s our best invention ever.
So what’s IPFS and what does it have to do with the internet?
Is it the future of the internet? Who’s working on improving it’s protocols, systems, and tools? What is the Interplanetary file system aka IPFS and why is it important?
Protocol labs created this innovating tech. Essentially, it consists of a decentralized internet scheme on top of the existing one.
In more specific terminology, IPFS is a peer-to-peer version controlled file-system which seeks to model all data as part of the same Merkle Tree (hash tree) which is DAG (Directed acyclic graph) type of data structure.
How does this work? What does this mean for government censorship? How can you start using it now? Are there any dangers?
This article will cover many topics regarding IPFS, however, the interplanetary file system is a very complex subject, so to really dig in, check out the IPFS white paper after you’re done with this article.
My goal with this article is to introduce you to many of the related concepts, give you an understanding of the value of IPFS, and a generalized view of how it works.
What is Decentralized Resource Sharing?
You computer is likely only using part of it’s available CPU cycles, disk storage, and RAM.
What if you could use these resources to speed up a network you and your organization depend on?
What if you could make money of off it?
Examples of Distributed File Systems
The concept of distributed computing and file storage is not new, but IPFS certainly is.
Here are some technology precursors and inspirations.
AFS mentioned in the White Paper
The AFS is such a system, developed at Carnegie Mellon University.
Andrew File System (AFS) has succeeded widely and is still in use today.
ARIA
OPEN AFS
Home of the OPEN AFS.
Dat Project
A distributed data community aka Dat is a nonprofit-backed data sharing protocol for apps of the future.
Perkeep
This set of open source formats, protocols, and software, previously known as Camlistore (Content-Addressable Multi-Layer Indexed Storage), is used to model, store, search, share, and sync data.
Peer-to-Peer Examples
Some other examples include popular publicly available peer-to-peer file sharing applications like Napster, KaZaA, and BitTorrent.
These decentralized platforms offered audio, video, and other file formats but it fell short on providing a way to build on top of it.
Thus far, there hasn’t been a global low-latency, and decentralized distributed system that is designed as infrastructure to be built on.
MojoNation
MojoNation is an old version which attempted to implement a way to compensate people for contributing resources to a network.
It wasn’t successful due to the challenge of compensating contributors. This was before crypto-currencies, so now, it’s much simpler to compensate users of a decentralized network.
Introducing, IPFS.
What is InterPlanetary File System – IPFS?
IPFS is all about decentralized resource sharing.
It’s a technology that enables content to persist on the internet as long as someone on the network has a copy.
You can view IPFS as a single BitTorrent swarm, exchanging objects within one Git Repository (from the white paper).
The white paper goes on to put it in more technical terms.
IPFS provides a high throughput content-addressed block storage model, with content-addressed hyper links.
This forms a generalized Merkle DAG (directed acyclic graph), a data structure upon which one can build versioned file-systems, block-chains, and even a Permanent Web.
The IPFS stack uses IPNS and IPLD to define the data, and Libp2p to move the data.
The stack utilizes Filecoin. This part of IPFS is to provide a cryptocurrency incentivized storage network.
The Filecoin block-chain is used to compensate contributors to the network.
How Does IPFS Work?
Getting a resource from the network starts with a hash that identifies a resource. If the original publisher source isn’t available, the protocol will find someone on the network with a copy.
The protocol will connect to the node, fetch the resource, verify it’s authenticity, and then send it to you.
Protocol labs created it and continues to improve and maintain it. They do research, development, and conduct experiments with network protocols.
They create open-source technologies, which means that you can take a look at their projects.
IPNS
IPNS is the InterPlanetary Naming System.
It uses a Distributed Hash Table.
DHT or the Distributed Hash Table
DHTs are widely used to coordinate and maintain metadata about peer-to-peer system.
DHTs Mentioned in the White Paper
Kademlia DHT provides lookup through massive networks.
Coral DSHT extends Kademlia to address a handful of deficiencies.
- More efficient storage scheme to save storage and bandwidth.
- Increased efficiency of value distribution by only providing a subset.
- Creates regions, so queries can first apply only to a local region, and then if value isn’t found they can query more distant nodes.
S/Kademlia DHT extends Kademlia to address security.
- Prevents Sybill attacks in Node Id generation
- Enables lookup values over disjoint paths in order to ensure communication with a large number of adversarial nodes in the network
It provides a lookup service similar to a hash table. Key/value pairs are stored in the distributed hash table.
IPLD
IPLD stands for interplanetary linked data. Although, my initial guess was interplanetary link directory. Here’s the github repo.
The data model of the content-addressable web. With it, you can treat all hash-linked data structures as subsets of a unified information space.
In essence this means that you can unify any data models that link data with hashes as instances of IPLD.
- A common hash-chain format for distributed data structures.
- Allows system interoperability..
- Ex: Bitcoin to Git, Git the Ethereum, etc..
- Merkle-links
- secure, immutable
- Merkle-paths
- Universal nestable URIs
- Canonical hashing safe
libp2p
This peer-to-peer networking stack is open source. It’s called a modular network stack because you can run network apps free from run-time and address services, independent of your location.
What does this library enable you to do?
- Use several transports
- Native Roaming
- configuration step free migration between networks and machines
- Runtime Freedom
- Assume connectivity
- Protocol Muxing
- Reuse already established and secure connections with in-band protocol multiplexing.
- Work Offline
- libp2p is capable of discovering other peers without resourcing to centralized registries, enabling apps to work disconnected from the backbone.
- Encrypted Connections
- Set a crypto channel by default
- Upgrade without Compromises
- Upgrade your service while still supporting older versions.
- Work in the Browser
- Write your application to run in the browser.
- Good for High Latency Scenarios
- Not all connections are equal. Make sure your app is using the best transport for its environment.
Implementations in:
Implementations coming soon, as of 6-10-2018.
- Java
- Python
- Rust
- Haskell
Multiformats
This part of IPFS is a collection of protocols which aim to future-proof systems.
This is the name of the organization but also the name of the protocols.
In other words, self-describing values for future-proofing. This allows inter-operability, protocol agility, and helps us avoid lock in.
You can think of Multiformats as a specification that helps systems to evolve and grow.
The Multiformats github page, is a good source of examples and source of deeper understanding, for this part of the IPFS stack.
FileCoin
FileCoin is a decentralized storage network created by Protocol Labs. It’s a decentralized network and a protocol.
Basically, you can contribute your file space to the network to earn FileCoins which can be traded for other crypto-currencies or a fiat-currency like the US Dollar.
The idea is that over time the cost of storage needs to go down, since the amount of data will continue to grow. Hence, FileCoin’s goal of reliably storing files at hyper-competitive prices.
Core Concepts
The above are technologies that together help IPFS work. There are a number of concepts that will help solidify your understanding of IPFS.
Content Versus Link Based Addressing
The web as we know it works as a set of links. If the server hosting the domain of a link is down, then the content can’t be retrieved. You can check out the internet archive but this isn’t ideal.
IPFS uses content based addressing where the address is a hash. If someone has a copy of the content that this hash represents, others can view it, even if the origin server is down.
Being IPFS Friendly
Make sure that all the links on your site are relative. Or in other-words, they don’t include the domain.
Interviews
David Dias
Worked on peer to peer and received a masters in it’s study. Say IPFS’s ambitious goals and joined the team. He say that they have similar goals and problems to solve. He created the IPFS Javascript implementation so it can run in the browser.
He has an interview on Software Engineering Daily Podcast all about IPFS and his contribution.
Or if you prefer reading, you can view the transcript of the IPFS software engineering daily interview with David Dias.
Karl Floersh
Software daily discussion on Decentralization.
Juan Benet
Juan Benet, creator of IPFS and Filecoin, founder of Protocol Labs.
Juan Benet is the inventor of the InterPlanetary File System (IPFS). His goal is to make the web faster, safer, and more open. He studied Computer Science at Stanford University.
IPFS is a large open source revolution focused on security, and re-decentralizing the web.
Juan is the founder of Protocol Labs, the internet tech R&D lab that develops IPFS and Filecoin.
Youtube talk on IPLD called Enter the Merkle Forest. The concept is all about content and data moving on the web. The main data structure behind it is a Merkle Tree, which is a computer science concept that isn’t the easiest to understand.
He also has an interview on Software Engineering daily.
Government censorship
If the local government of a client blocks the original hosting server, the client can fetch the content from another node that the government hasn’t black listed. This can have earth changing consequences when it comes to dissemination and access to information on a global level.
Create Your Own IPFS Node
To join the network you have to create a public and private key pair. This will then register you on the IPFS network as a contributing node.
First your node will check your Local Area Network or LAN to find other local nodes. This is done using a multicast DNS or Domain Name System.
Resources, Demos and Tutorials
Collaborative Editing Application with IPFS using CRDT video on youtube.
Create a React App to Store Data on IPFS
Conclusion
This type of technology can change the world, so if you’re in the field, it’s a good idea to look into it, as you may be building on it in the future.
Stay tuned as more IPFS related article will be published here as I learn more.