New state block design

As we continue to develop the protocol and the node implementation, we are constantly discovering opportunities for improvement. Some improvements would be very welcome on the network, but aren't possible or as effective with the current state block structure. Because of this, the introduction of a new state block version with structural changes is becoming a priority.

This topic is aiming to discuss the changes we are considering and why we think they are worth the effort. Before formalizing the specification for a new state block, we want to explore all possible additions to help reduce the need for additional versions in the future. This is especially important because each new block version not only requires an upgrade process with multiple node releases, canary blocks or epoch blocks, but also requires hardware wallets and other custom integrations to change their payload setup and signing code.

Considerations

With some of these changes it is important to remember that even though the data being considered for addition to the block may be available already either explicitly or implicitly from other data the node has access to, in many cases additional computational or disk IO resources are required to access it. Adding them directly to the block may save on these resources but of course comes with tradeoffs including additional bandwidth usage.

As the theoretical bottleneck with Nano is bandwidth, we must do our best to weigh these tradeoffs to ensure their long term impact is properly balanced.

Candidates for new fields

Version

Size: 1 byte

This would replace previous epoch version markers and provide a point in each account chain to transition certain behaviors in the node. For example, when the new work algorithm is implemented an incrementing of the block version could be used to signal an account switching over.

This also provides a clear indication at the network level of what epoch or version a block is, allowing for better filtration and management of traffic before further resources are used to validate these details.

Block height

Size: 64 bits (Max block height per account: 18,446,744,073,709,551,615)

This is a powerful addition to the block payload as it allows easier filtering of unwanted traffic in certain scenarios:

  • If the block may be discarded due to it being below the existing confirmation height for the account

  • If the block could possibly be a fork without having to reference the root

It also makes it easier to understand the relationship of the block to the ledger when it does not fit on a known chain. Currently blocks for an account that don't fit the existing chain don't provide that context to know how close they are to the current frontier (unless an open block), but the block height provides that context.

Subtype flags

Size: 1 byte

The original state block design removed the need for different types of blocks having different structures. This improvement made some elements of the node and protocol simpler, but also removed context which can be useful in certain scenarios, including during block validation.

Although more detail is needed to finalize the flags, the main considerations are indicators of the signature being used (epoch vs. self signed) and how to interpret the link field value (hash for receive, account for send, etc.). Requiring these will help streamline management of the block on arrival and offer additional protection against certain accidental changes such as sends to the burn account.

Not all bits in this field would be used in the current design, saving room for additional uses for this field in the future.

Amount

Size: 16 bytes

Although the amount of Nano being transferred by a block is easily calculated by comparing balances between the current and previous blocks, adding an amount field gives more explicit details on the intended transfer.

This is helpful for various offline wallet or lighter node implementations, as well as with some pruning scenarios where keeping the previous to pending send blocks may be necessary if this field wasn't available.

Feedback

We are proposing these block changes after careful consideration of the benefits and tradeoffs. We are interested feedback and analysis of these additions, especially around possible drawbacks seen against potential benefits.

4 Likes

In this case why not adding in the same time a reference/memo field of few bytes ?

If this were to be implemented, services would not need to create + subscribe to block for every payment.

But only one address and it would actually save bandwidth and storage for the whole network.

Because they need to pocket then transfer to their own address then repocket which is 3 block instead of only one

Thanks for mentioning the memo piece, it is one of the items that is frequently asked about in regards to block content. A couple reasons why this hasn't been included:

  • Any fields that allow arbitrary data to be added increase the chances that ledger space will be used up for holding data aimed for purposes other than transferring value via Nano, which represents an increase in resource usage and lowers efficiency of the network
  • Payment IDs and similar data represent proprietary accounting data from external systems that is being forced on all network participants via the ledger

On the second point we think there are alternative approaches here that can be established for communication between the wallet/sender and merchant/receiver to allow easier accounting for incoming payments and potentially even avoiding the need to use unique payment addresses.

We will be pushing some more detail around some possibilities for this soon, but there are also protocols like Manta and community suggestions to look at other payment schemes like BIP70 or the W3C payment protocol.

What are your thoughts on these?

2 Likes

So in total this would be an increase of ~26 bytes per block? Any idea what it would be with database overhead? If my math is correct, we're currently at ~550 bytes/block average (28,468,649,984 bytes for data.ldb vs 51,808,653 blocks), and the proposed change would represent a ~5% increase per block.

I like the version change, because for 1 byte per block (e.g. 51,808,653 new bytes) we can get rid of 1blockPerEpoch*numberOfNanoAccounts. For 434,781 current accounts that's 239,129,550 bytes per epoch. Easy trade off, not even including the usability/pruning/light node benefits.

I'm not as sure about the other three fields though. Since nodes (including light nodes) always have to compare the incoming transaction to the current frontier anyways, do we really get that much benefit from adding block height, subtype, and amount (besides perhaps slightly easier service implementations from explicit definition vs contextual definition)? Isn't the "previous" field an implicit block height field anyways?

We appreciate the feedback and questions here. Not sure about the database overhead, but can look further into that.

If the block included the amount field then light nodes technically could operate without the previous block by requesting the frontier on their account from a number of other nodes - probably not the biggest use case. For certain payment scenarios it can make the validation of the payment easier by not requiring the previous block as well, as you mentioned. Also having the amount also allows more options for PoW validation against amount without requiring disk reads, which we try to avoid for DoS protection.

With block height we can get it from previous, but including it allows checks for where a non-fitting block might would line up in the ledger even if we don't have the previous. In the future it may be worth limiting the height above confirmation that a node will accept blocks for and having this value in the block would make filtering those lighter weight.

And as mentioned the fork detection based on height may also be helpful earlier in the resource consumption path for a block, but that is still a bit TBD. One other thing not mentioned is that with a block height included even newly syncing nodes (using top-down bootstrapping) can get a general sense of how synced their account set is are by summing the frontier blocks even before they've fully bootstrapped to genesis, or understand how closely a set of a frontiers gathered from the network is to being complete vs. block counts seen (along with account counts) via telemetry, for instance.

The subtype does provide more explicitness to the block and although many of these details can be gathered by pulling a previous block (send vs. receive for instance), it does help with a bit more DoS protection when work validation is different between receive vs. send for instance. And with regards to the signature, it can help avoid some extra signature checks when blocks are marked for version change (was it the epoch signer or not - for self upgrades).

Based on these additional details, are you still hesitant with the tradeoffs of some of these fields?

1 Like

Thank you for the detailed response!

Light nodes directly querying for a specific account frontier makes sense, but I'm still struggling to understand how a dedicated amount field helps. If the light node has an old frontier that they're updating, surely it's not that difficult to calculate the transfer amount if they really want that information, and if they don't have the old frontier then what use is getting a transfer amount? I can see how it might make light implementations slightly easier, but I'm not sure that's worth the cost of 16 bytes (x2 for send/receive). That being said, if PoW by transfer amount is being seriously considered, then this field might make more sense to me since we'd be trading some block space for long-term ledger space (i.e. reducing spam and bloat). Is there a lot of demand for this field?

The block height field makes more sense to me now. I can see how having it in the current block allows for quick processing (potentially discarding) of a block in certain scenarios. Similar to PoW (assuming you know the block subtype :grin:)

Some ignorant brainstorming on block height here: but is there a way to get "close enough" without reserving a full 8 bytes? If it was just the last 3 digits or something - that wouldn't help much for bootstrapping, but maybe it would be enough for live tracking of new block heights vs the old ledger block height? For example, if I had BH 12,345 and a new block came in with 349, could I reasonably assume that the new block's height is 12,349?

The subtype field also makes more sense to me now, and 1 byte isn't enough to make me overthink it too much haha

Tl;dr, I guess my biggest concern is about the amount field and whether or not it's really worth permanently increasing block sizes for something that only applies to very specific usecases (light nodes?) and seems to really only improve the relatively infrequent service integration/implementation process?

1 Like

I have a similar feeling on the amount field, it seems if one is replacing an old frontier with a new one they can simply calculate the amount value. That being said we've had difficulties before when something needs to be calculated based on a different block because we need to load two blocks to calculate it.

I think we calculated a 5% overhead for this field. Opinions for/against are definitely welcome.

It's possible for the block height to be less than 64-bits, even 40 or 48-bits could be enough.

I'll make a layout/alignment diagram and maybe we can see if we can conveniently pack things in an aligned width.

2 Likes

There's one change we're considering on the signature: signing the block contents instead of the block hash. Since the signature algorithm applies a digest to what you're hashing, and we're hashing the block hash, this involves 2 blake2 operations per signature and this would be reduced to 1.

2 Likes

Something else I thought of today - since the "type" field for all blocks is always "state" now, could that be overloaded to be used as subtype instead of creating a new subtype field? Maybe in combination with the new version field?

I guess the pain point with that might be the required service integration updates just to save one byte ha

Good thinking, block type goes into the extensions of the header so not really necessary, however we're analyzing how to optimize the subtype flags.

1 Like

I think that it would be better to make a new flexible-block, instead of every change needs a new type of block and makes it impossible to parse them.


Let's consider a node (any node, light-wallet, pruned...) that is running today (not aware of that new block-type). If it gets the "State Block V2", they must refuse that block, without any fallback. It is impossible to parse. Because every block is not flexible and there's currently no promises that future blocks will follow the State pattern.


There are two solutions here.


What I mean about "State pattern" is:

Every block, will need to have the "State Block" + extra-fields. All extra-fields must be "hashable" (signed). It can't change the length of any data (such as PoW or PublicKey), it also can't remove any existent data.

That makes possible to parse future blocks, ignoring the extra-content. Of course, the old block must be supported (with a higher cost of PoW, for instance).


What I mean with "flexible", is every field most have something like:

[Field_ID][Field_Len][Field_Version][Content]

All fields (except from PoW will be signed, forever). So, using that "Flexible-Block" currently, will be something like:

[0][32][0][{Public_Key}]
[1][32][0][{Previous}]
[2][32][0][{Representative}]
[3][16][0][{Balance}]
[4][32][0][{Link}]
[5][64][0][{signature}]
[6][8][0][{PoW}]

What is the difference now?! It's possible to add new fields or change them, without a break all the compatibility. It's possible to add new fields, and then you could choose what you do: "ignore the block", "try to parse", "request votes", "request the user to update the software (if the block is valid, but not supported)"...


@clemahieu suggests a change in the signature method, which I agree, but that shows the reason behind the "flexible block". If that "new flexible-block", it could be a new "Field_Version".

If I get a block with unsupported "Field_Version" I can ignore that (NF will do that, of course), or I could parse the block (ignoring the signature) and request votes. Why? Because I can still check the amount by getting the balance (Field_ID = 3) and the Link (Field_ID = 4) and the previous (Field_ID = 1).

There's some talk about changing the PoW algorithm too. If the PoW changes (taking consideration the height or new algorithm) it could be a simple new Field_Version at the same Field_ID of the old PoW.

The Field_Len makes it possible to change the length. So, if there's a multi-signature public key, it can be a new version and set the length to bigger than 32 bytes. So, I can ignore that field, if unsupported, but... I can still do the hash, asking votes and generate a "receive block". Even, without support multi-signature transactions. It never would be possible with new block-type for "multi-sig" (imagine like State MultiSig), because it wouldn't be able to parse the block and get the balance/link.

Some fields (like Previous and Balance) maybe don't need that, since change it breaks everything.


Of course, that still problem with the current Epoch. That new block also supposes that some fields can still be omitted. So, even after V3, V4, V100, would always be possible to send a transaction following the State V1 (or V2). The same applies to "State Pattern", which is only usefull if still possible to use the V1.

3 Likes

We did consider some form of extensible block design and like you pointed out it could include must understand/optional fields, fields that are included in the block hash or not, etc. The biggest difficulty we run in to is less with the wire format and parsing the block and more with the semantics around adding new fields that nodes must understand. Even if they can parse the block they may still need to reject the block because they don't know how to interpret new fields.

I think we can fix some of the pain around parsing new wire formats by using a parser-generators that builds from a generic specification, maybe even flatbuffers. It's possible we could specify upgrades to the block, which hopefully will remain minimal if any more, in terms of using new field numbers. Maybe this is a good compromise?

We have to be careful with a few things:

  • Hash malleability: we don't want equivalent semantics in the field to yield different hashes.
  • Mutually exclusive fields: Let's say we had a 40-bit block height field and we found a need to upgrade it to 48-bit. We need a way to say, use the new field and not the old field, it can't use them both, especially if they conflict.
  • Eliminating block-bloat: we don't want a bunch of historical fields sitting around in a block forever into the future just on the off chance someone doesn't want to upgrade their parsing code from 10 years ago.
  • Denial of service: We don't want someone to be able to put a bunch of arbitrary data in the block in order to bloat its size while not providing any useful semantics to processing currency.

The work field is an interesting thing, it's not in the block hash payload so this gives flexibility to wire-parsing but it doesn't affect signing.

Right now the epoch version number effectively defines the required fields. If we did define the block with something like flatbuffers, we could have this version number define which fields must be used and which ones must not. Then the block parsing doesn't need manual work but the semantic interpretation would still need to be upgraded.

3 Likes

I honestly would also like to see a field for some kind of reference. I wouldnt even make it a big text field but just big enough that some hash can fit in because currently neither side can exactly prove what a payment is for, which can get quite chaotic, especially if there are multiple things that can go for the same amount, or other order details.

for example with Fiat transactions in Germany for sending money from one bank account to another, we have 140 characters (yeah the exact length of a twitter post up to a while ago) to express what any transaction is for, which is very useful obviously, however Nano obviously wouldnt need that much of space if a hash is used (which obviously should be because personal data and all.

for example in case of SHA256 it would be just 32 Bytes

also in the same vein you can easily prove something has NOT been paid if you search for the hash of that purpose and dont get your results.

might it not even be that the block size is a bit larger than a send and recieve block because with unique addresses, you need an open block which does representative stuff?
but yeah if you had a refernce it would just be

send-> recieve
instead of
send->recieve(open)->send->recieve (on original account)

Currently, the only way to store the data is abusing of the "Representative" field. It's 32 bytes and, for the most case, no one cares. I think we need to create a new standard around the current URI (https://docs.nano.org/integration-guides/the-basics/#uri-and-qr-code-standards). Something like [&][representative=<data>], I think you get the idea.

The NF idea of "saving space", is kinda the opposite: because we need at least 3 transactions (one send with custom representative, another to reset the representative and another receive).

The same happens to create a new address, has @Jav said, just worse. To identify each payment needs to create a new address/account, which is terrible for pruning and also takes four blocks in total (1 Send for the buyer -> 1 Open + 1 Send -> 1 Receive for the primary account).

The BIP-0070 seems to have other priority. The main one is replacing the "random-looking address" by the website domain, it also gives the refund-address to the seller... It's not in a network-protocol level, but the wallet implementation, that may help, but need to have wallets capable of handling such protocol. That can also abuse of the "Representative" field if needed.

Ok here's an outside-of-the-box way to get a 32 byte memo field with minimal (or zero) impact to the block size.

  • Rename the "Representative" field to "metadata"
  • Find a bit from elsewhere or add a 1 byte "selector" field
  • When the selector is 0, the value in "metadata" changes the account's representative
  • When the selector is 1, the value in "metadata" does not change the account's representative

Nonzero selector values essentially make the 32 bytes of our current "Representative" field free for 2nd-Layer applications to interpret. And since the vast majority of transactions don't involve changing the representative, this is a more efficient use of our block data.

Ok here's another way to do it but without the pruning implications of the last one.

We bite the bullet and add a 32-byte "forward" field.

  • For send blocks this field gets updated to the block hash of the receive block after the other account has received it
  • For receive blocks this field acts as purely application-specific metadata

We then get the block traversal improvements in the ledger, and the ability to add metadata, all in one field. Granted this assumes there's use in a metadata field just for receive blocks, when I expect most applications would only find metadata useful in send blocks.

Even fancier, perhaps for nodes that are currently online, the "forward" field could be metadata for each send block, but upon the other chain's receive it gets overwritten with the forward link to their block hash.

We could store that type of information in the ledger in the sideband information, but since the sender can't know the hash of the block that will receive this send, it wouldn't be able to be in the signed payload. If it's not in the signed payload it's really something that's just stored in the ledger and wouldn't need to be transmitted with the block over the network.

The link field already has a link-interpretation enumeration, if there were a metadata bit, we'd probably have it use the link field rather than the representative field. We had problems in the past where, in order to find an account's representative, we had to traverse back an arbitrary number of block to find it, hence the reason to put the representative in each black.

We've done a fairly large amount of brainstorming on this, I don't like the account-per-transaction strategy for a couple reasons. It usually requires sweeping out of that account into a main account, requiring multiple transactions. We broke the interaction down this way: What is the payment provider doing with the account-per-transaction that gives them the ability to identify it for a particular purchase? They're essentially giving a random, secret number to the payer in order to identify a particular transaction. So we explored if this could be done in a different, more efficient way, ideally allowing direct payments to one main payment account from everyone transacting in the system. This seems possible in a different way, rather than using a signal from the network that a payment has arrived, by watching account balances, the payment provider can provide a unique callback URL where the payer sends the block after signing. The unique URL is ephemeral so it doesn't need to go in the ledger, and it uniquely identifies a particular payment. They PP can then forward the block on to the network to ensure it's valid. The process would be:

  • Create PP DB entry and associate it with a callback URL and mark the entry as waiting
  • On receipt of a block through the URL, mark the DB entry as in-progress
  • Check the amount, if incorrect, don't publish the block
  • Publish the block to the network
  • Wait for confirmation if successful, mark the DB entry as paid

There is a side-benefit of this callback URL strategy: it doesn't require the signing device to have internet access. This could be the case in low-cost devices, no network coverage, or high congestion areas (sports stadiums). The PoS devices are usually wired or have high quality connections and since the PoS devices would be responsible for publishing the block to the network, it would give a much higher quality experience.

1 Like

A new suggestion for the new state block format is to encapsulate old legacy blocks and the current state blocks in its fields.

The motivation for this change is to simplify node code by having only one block class and give hints to the node as far as a block's account, version, and height.

This will shift complexity from different processing functions in different classes to a single processing function for all block types though this seems like it will likely decrease overall complexity.

Since existing blocks cannot be re-signed, the hashables and will need to be interpreted differently depending on the block that's being encapsulated.

There is an option for this implementation to either:

  • Create an additional bitfield that defines the hashables or
  • Replace the signer field with this bitfield and add an extra enumeration value defining state2/self or state2/epoch signing.
1 Like

Hi,
I was kind-of invited here from https://github.com/nanocurrency/nano-node/issues/2942 .
(I feel like an ant between giants, so I hope I can contribute a bit, and sorry if I step "out of bounds" somewhere, as I didn't understand all here.)

There was a lot of discussion about memo field (32B or something like that), but I have another spin.
Add just 8 Bytes int64, which is ~19digit number, enough for any invoice number id identification.

I won't write all what is in the issue, only mention what was written here.

8 Bytes are (by the numbers in this thread) just 1.5% added. Which might not even increase disk used, if the underlying database uses data blocks of fixed size for adressing.
And current method of account per payment uses far more resources and storage, and can not be pruned!

Transactions in the world require 4 main ingredients sender, reciever, amount, reason(identification). Nano blocks do not contain any info about why the payment exists.
"Why in ledger" is about trust. Signed blockchain can not be changed.
"forced on all" If the people want it, need it, why not add it, as it won't affect others. I'm not forcing anyone to parse/use my data.

I'm hope for a future, where you could use also cryptocurrency ledger for accounting or taxes. And for that is some trusted memo field required.

This is about identification of the payment, an association to something in the real world (invoice), that is why we want it in an imutable ledger. Informational data, messages to reciever etc. should be outside L1 ledger.

If I understand, it is the Direct block handoff option.
It requires some api on merchant side, so it's unusable for some average joe who wants to accept nano.
It's still online QRcode-only payment, and it has to contain api address where to send the block.
Payment block still does not contain a reason identification for payment.
I can not just take merchant address, amount and invoice number to pay. Any type of memo field would allow it. And it's also, how people pay for currently, and would be much easier to undestand for any 'grandma'.
I'm not saying, that Direct block handoff is bad... just that it is not a wide solution to the problem.

Well, what about the compromise to make such data only part of pending blocks? (The data will be discarded as soon as the receive block is processed.)