Author of this article: Yang Wei of Xianhe system; original works, reprint please indicate the source
Catalog
Link: 4 Analysis of top-level architecture of filecoin source code
Link: 5 heartbeat protocol of filecoin source protocol layer analysis
Link: the hello handshake protocol of 6 filecoin source protocol layer analysis
-
Storage protocol for analysis of filecoin source protocol layer
- 7.1 agreement overview
- 7.2 source information
- 7.3 source code analysis
- 7.3.1 storage miner
- 7.3.2 storage customers
7.1 agreement overview
storage_protocol.png
7.2 source information
- version
- master branch 619b0eb1 (March 2, 2019)
- package
- storage
- location
- protocol/storage
7.3 source code analysis
7.3.1 storage miner
▼ package storage ▶ imports ▼ constants //Waiting for seal data prefix -dealsAwatingSealDatastorePrefix // Storage transaction agreement Name: "/ fil/storage/mk/1.0.0" -makeDealProtocol // Miner data store prefix -minerDatastorePrefix // Storage query protocol name: '/ fil/storage/qry/1.0.0' -queryDealProtocol // Gas and gas limitation -submitPostGasLimit -submitPostGasPrice // Waiting time for payment channel establishment -waitForPaymentChannelDuration ▼ variables -log ▼+Miner : struct [fields] // Transaction set -deals : map[cid.Cid]*storageDeal // Waiting for sealing structure -dealsAwaitingSeal : *dealsAwaitingSealStruct // Resource object of transaction -dealsDs : repo.Datastore // Transaction lock -dealsLk : sync.Mutex // Storage miner address -minerAddr : address.Address // The Owner address of the node -minerOwnerAddr : address.Address // Node object, which has an interface defined to be implemented by the storage miner -node : node // High level API for storing miners -porcelainAPI : minerPorcelain // Whether it is in the generation of spatiotemporal proof and the corresponding lock -postInProcess : *types.BlockHeight -postInProcessLk : sync.Mutex // Accept and reject transactions -proposalAcceptor : func(ctx context.Context, m *Miner, p *DealProposal) *DealResponse, error -proposalRejector : func(ctx context.Context, m *Miner, p *DealProposal, reason string) *DealResponse, error [methods] // When the sealed message is submitted to the blockchain, the callback function executed is executed in the node // If 1 fails, dealsAwaitingSeal.fail will be called // 2. If successful, dealsAwaitingSeal.success will be called // 3. After success, you need to save the seal sector information. If it fails, call dealsAwaitingSeal.fail +OnCommitmentAddedToChain(sector *sectorbuilder.SealedSectorMetadata, err error) // Callback generated by new block, called by node, will trigger new storage proof // If the space-time proof expires, it will start again in the new cycle +OnNewHeaviestTipSet(ts types.TipSet) // Called by handleQueryDeal to return the query result +Query(ctx context.Context, c cid.Cid) : *DealResponse // Generative space-time proof -generatePoSt(commRs []proofs.CommR, challenge proofs.PoStChallengeSeed) : proofs.PoStProof, []uint64, error // Access to payment channel information // 1 wait for the establishment of payment channel // 2 get payment channel information and return // 3. Payment information includes: contract address, payer address, channel information, payment channel message cid, payment voucher collection -getPaymentChannel(ctx context.Context, p *DealProposal) : *paymentbroker.PaymentChannel, error // Get new time to prove space-time -getProvingPeriodStart() : *types.BlockHeight, error // Get specific deals for storage miners -getStorageDeal(c cid.Cid) : *storageDeal // Get quotes for storage miners -getStoragePrice() : *types.AttoFIL, error // Store the entry method of transaction request, handle function of transaction request flow // 1 read transaction request information in the stream // 2 call receiveStorageProposal to process transaction request // 3 reply processing reply -handleMakeDeal(s inet.Stream) //Analyze the specific flow information, process the Query request, and call the Query request -handleQueryDeal(s inet.Stream) // Load transaction information from resource directory to Miner instance -loadDeals() : error // Load information to be sealed -loadDealsAwaitingSeal() : error // Sealing failed, update response information -onCommitFail(dealCid cid.Cid, message string) // Seal successful, update response information // 1 switch state to Posted // 2 update proof information: sector ID, copy information, original data information -onCommitSuccess(dealCid cid.Cid, sector *sectorbuilder.SealedSectorMetadata) // Handling storage transactions // 1. Access to stored transaction information // 2. Data processing and sealing -processStorageDeal(c cid.Cid) // Process transaction request // 1 check the correctness of the signature // 2 check the correctness of payment information and call the validateDealPayment method // 3 illegal call to proposalrejector (rejectproject) to reject the request; legal call to proposalacceptor (acceptproject) to reply -receiveStorageProposal(ctx context.Context, sp *SignedDealProposal) : *DealResponse, error // Store transaction information from Miner object to resource directory -saveDeal(proposalCid cid.Cid) : error // Store information to be sealed to resource directory -saveDealsAwaitingSeal() : error // Submit time and space certificate // 1 generate random seeds // 2 according to the input length of spatiotemporal proof, generate copy slice // 3. Time and space proof of random seed + copy slice as input generation // 4 call the high-level interface to send messages -submitPoSt(start, end *types.BlockHeight, inputs []generatePostInput) // Update transaction response message -updateDealResponse(proposalCid cid.Cid, f func(*DealResponse)) : error // Check the correctness of payment information // 1 the customer's offer must be higher than the miner's offer // 2 payee must be a miner at this node // 3. The total fund of payment channel must be greater than the miner's quotation // 4. There must be a transaction receipt, and the total amount of the transaction receipt must be greater than the miner's quotation -validateDealPayment(ctx context.Context, p *DealProposal) : error [functions] // Instantiation storage miner // 1 parameter assignment via node // 2 specifies the callback function that failed to seal successfully // 3. Set the flow handle method of transaction request and transaction query +NewMiner(ctx context.Context, minerAddr, minerOwnerAddr address.Address, nd node, dealsDs repo.Datastore, porcelainAPI minerPorcelain) : *Miner, error ▼-dealsAwaitingSealStruct : struct [fields] // Get failure information from sector id +FailedSectors : map[uint64]string // Get the cid of the transaction from the sector id +SectorsToDeals : map[uint64][]cid.Cid // Get sector metadata from sector id +SuccessfulSectors : map[uint64]*sectorbuilder.SealedSectorMetadata -l : sync.Mutex // The failure processing callback points to onCommitFail in the instantiated Miner -onFail : func(dealCid cid.Cid, message string) // Callback processed successfully. When Miner is instantiated, it points to onCommitSuccess -onSuccess : func(dealCid cid.Cid, sector *sectorbuilder.SealedSectorMetadata) [methods] // Seal data -add(sectorID uint64, dealCid cid.Cid) // Seal failure handling dealsAwaitingSeal.onFail -fail(sectorID uint64, message string) // Seal successfully processed dealsAwaitingSeal.onSuccess -success(sector *sectorbuilder.SealedSectorMetadata) ▼-generatePostInput : struct [fields] // Replica merkle root -commD : proofs.CommD // Raw data merkle root -commR : proofs.CommR // Middle data merkle root -commRStar : proofs.CommRStar // Sector ID -sectorID : uint64 ▼-storageDeal : struct [fields] // Transaction request structure +Proposal : *DealProposal // Transaction request response structure +Response : *DealResponse // Storage miner high level API ▼-minerPorcelain : interface [methods] // block height +ChainBlockHeight(ctx context.Context) : *types.BlockHeight, error // Get configuration +ConfigGet(dottedPath string) : interface{}, error // Send, query, wait for message +MessageQuery(ctx context.Context, optFrom, to address.Address, method string, params ...interface{}) : [][]byte, *exec.FunctionSignature, error +MessageSend(ctx context.Context, from, to address.Address, value *types.AttoFIL, gasPrice types.AttoFIL, gasLimit types.GasUnits, method string, params ...interface{}) : cid.Cid, error +MessageWait(ctx context.Context, msgCid cid.Cid, cb func(*types.Block, *types.SignedMessage, *types.MessageReceipt) error) : error ▼-node : interface [methods] // block height +BlockHeight() : *types.BlockHeight, error // Block service +BlockService() : bserv.BlockService // Block time +GetBlockTime() : time.Duration // Host information +Host() : host.Host // Sector creation, including // 1. Add and read piece; // 2 seal all non empty staged sectors // 3. The sealing result is returned through channel // 4 get the maximum piece byte size in the sector // 5. Space time proof of generation +SectorBuilder() : sectorbuilder.SectorBuilder ▼ functions // After storing the transaction information, call processStorageDeal to process the transaction information -acceptProposal(ctx context.Context, sm *Miner, p *DealProposal) : *DealResponse, error // Get specific file size -getFileSize(ctx context.Context, c cid.Cid, dserv ipld.DAGService) : uint64, error -init() // Store transaction information, update response message and return -rejectProposal(ctx context.Context, sm *Miner, p *DealProposal, reason string) : *DealResponse, error
7.3.2 storage customers
▼ package storage ▼ imports ▼ constants +ChannelExpiryInterval // Gas and gas limitation +CreateChannelGasLimit +CreateChannelGasPrice +ErrDupicateDeal // Establish the period of the Voucher +VoucherInterval // Storage prefix -clientDatastorePrefix ▼ variables +Errors ▼+Client : struct [fields] // Storage customer high level API -api : clientPorcelainAPI // Transaction set -deals : map[cid.Cid]*clientDeal // Trading resource directory object and lock -dealsDs : repo.Datastore -dealsLk : sync.Mutex // Storage customer node -node : clientNode [methods] // Load vouchers for a specific transaction +LoadVouchersForDeal(dealCid cid.Cid) : []*paymentbroker.PaymentVoucher, error // Initiate storage transactions // 1 get file size, miner quotation, block height, destination address // 2 establish payment channel // 3 call MakeProtocolRequest to initiate transaction request // 4 check transaction response // 5. Response and reply to persistent transaction +ProposeDeal(ctx context.Context, miner address.Address, data cid.Cid, askID uint64, duration uint64, allowDuplicates bool) : *DealResponse, error // Enquiry transaction // 1 get miner information, address, node ID // 2 call MakeProtocolRequest to initiate the request +QueryDeal(ctx context.Context, proposalCid cid.Cid) : *DealResponse, error // Check transaction response -checkDealResponse(ctx context.Context, resp *DealResponse) : error // Judge whether it is a duplicate transaction -isMaybeDupDeal(p *DealProposal) : bool // Load transaction information -loadDeals() : error // Return to target miner address -minerForProposal(c cid.Cid) : address.Address, error // Persistent transaction response -recordResponse(resp *DealResponse, miner address.Address, p *DealProposal) : error // Save transaction information -saveDeal(cid cid.Cid) : error [functions] // Instantiate storage customers +NewClient(nd clientNode, api clientPorcelainAPI, dealsDs repo.Datastore) : *Client, error ▼+ClientNodeImpl : struct [fields] -blockTime : time.Duration -dserv : ipld.DAGService -host : host.Host [methods] //Implement the clientNode interface +GetBlockTime() : time.Duration // Get file size +GetFileSize(ctx context.Context, c cid.Cid) : uint64, error // Initiate protocol request // 1. Establish the corresponding protocol flow of storage transaction or request // 2 initiate request +MakeProtocolRequest(ctx context.Context, protocol protocol.ID, peer peer.ID, request interface{}, response interface{}) : error [functions] // Instantiate customer node +NewClientNodeImpl(ds ipld.DAGService, host host.Host, bt time.Duration) : *ClientNodeImpl ▼-clientDeal : struct [fields] // Target miner, request and response +Miner : address.Address +Proposal : *DealProposal +Response : *DealResponse ▼-clientNode : interface // Implemented by ClientNodeImpl [methods] +GetBlockTime() : time.Duration +GetFileSize(context.Context, cid.Cid) : uint64, error +MakeProtocolRequest(ctx context.Context, protocol protocol.ID, peer peer.ID, request interface{}, response interface{}) : error ▼-clientPorcelainAPI : interface [embedded] +types.Signer [methods] // Get block height +ChainBlockHeight(ctx context.Context) : *types.BlockHeight, error // Create payment channel // Including source and destination address, price, time, payment interval, channel timeout, Gas and restrictions +CreatePayments(ctx context.Context, config porcelain.CreatePaymentsParams) : *porcelain.CreatePaymentsReturn, error // Get target address +GetAndMaybeSetDefaultSenderAddress() : address.Address, error // Get miner's offer +MinerGetAsk(ctx context.Context, minerAddr address.Address, askID uint64) : miner.Ask, error // Get miner Owner address +MinerGetOwnerAddress(ctx context.Context, minerAddr address.Address) : address.Address, error // Get miner node ID +MinerGetPeerID(ctx context.Context, minerAddr address.Address) : peer.ID, error ▼ functions -init()