Standardizing transaction metadata for an inter-wallet format

The last time I was looking into paying taxes it seemed that all the available solutions involved pointing the software at the assorted sources of the information needed, (e.g. to the crypto ledger via your address, your exchange accounts, etc) rather than getting all of that information from a standardized wallet source. This doesn't mean there are no wallet solutions, but at least they weren't widely adopted enough that the accounting solutions relied on them.

Perhaps it would be better to design a crypto-agnostic solution, creating a standardized format that can be used by all other cryptocurrency wallets and accounting software.

Well I think it should be pretty easy to distinguish which of these fields are general to currencies and which are specific to nano. Any conflicts would arise with the overall structure: are there cryptocurrenceis which can't be organized by account + transaction? Otherwise, you could have a field at the "account" level which identifies which cryptocurrency the account belongs to. The standard would then have a set of required fields (the general ones) and each cryptocurrency could then have a set of additional fields that make sense in the context of their own protocol.

Things that would probably be general:

  • crypto protocol
  • address
  • private name
  • public name
  • modified
  • accounting unit
  • identifier
  • amount
  • value
  • memo
  • transactee

Things that would not be general:

  • index
  • hash

Things that I'm not sure of:

  • time published
  • time confirmed

I would think these two are general, as long as the cryptocurrency has a reasonably identifiable confirmation point (e.g. bitcoin reaching N blocks deep).

Provide a way for additional data fields to be defined (either custom metadata from the wallet, or crypto-specific parameters)

I'm not sure this even needs to be defined. Anyone adopting this format could add their own fields, and any software reading the format could just skip fields that it doesn't understand.

Personally I don't feel this would provide much of a purpose to the handoff protocol — that requires its own set of parameters and is for dealing with the processing of blocks, whereas the standard you propose seems to be aimed at storing and transferring historical data of already existing blocks

Ideally all of these fields would be populated at the time of transaction, rather than after the fact, and some of them would ideally be populated by the opposing party. In practice I would expect that the transaction process would simply involve the exchange of a generic "metadata" chunk that contains an assortment of fields like this from each party. The wallet of the other party can then choose which metadata fields it cares to integrate into its own records of the transaction. Sort of like "throwing it at the wall and seeing what sticks." For instance, the merchant might store the "transactee" field from the customer. The customer might store the "memo" and "transactee" fields from the the merchant, and if their "accounting_unit" matches your own you would store their "value" field because the merchant is likely the one defining the exchange rate of the transaction.

  • Personally I'm not so sure about the inclusion of the fiat currency - exchange rates can differ between sources, and different apps may expect or support different currencies. Perhaps including the crypto amount would be better, instead allowing the wallet/accounting software to make the conversions?

The crypto amount could be included. It would be redundant since it could be read from the ledger, but there's no strong reason not to have the field and it would make it easier for accounting software to just read this value rather than interact with the network. If the amounts disagree it might be cause for further error checking.

But the fiat currency should be included, because there's no other good place to store this information. In principle two parties could agree to make an exchange where the Nano is valued drastically differently from the current market price, and you would need to record that. Not everyone has to use this field, but if they do use it this is the logical place to have it. Eventually of course we would hope that people are accounting in nano and this field becomes obsolete. :slight_smile:

I feel I'm misunderstanding how people are suggesting this could be integrated into the handoff protocol, so if you could help clarify it for me (or provide any other suggestions) I'd greatly appreciate it.

I might have answered this question earlier in this reply. Basically at some point during the transaction communication they simply trade a chunk of metadata that they populate with their own info, and let the other do with it what they will. For a merchant it might make sense to include their chunk in the payment request, which lets the customer wallet display all of this information while they authorize the tx. The customer they could include their chunk it when they hand off the block, since they are already sending data at this step.

The first JSON schema is very similar to what we have for the current revised handoff protocol, though with a few more options in addition to that.

I see what you mean. I still don't understand what the need for this specific standard/schema is though (in regards to the handoff protocol) — as far as the payment processor/merchant is concerned, all they would really need is the ID and block hash, so all the additional metadata would be irrelevant.

The newer handoff protocol makes some additional changes that also break this, namely the fact that the wallet no longer publishes the block themselves; instead, they pass ("hand off") the block directly to the payment processor for them to validate, generate work, and publish on the sender's behalf.

What do you think is unique to Nano that isn't present in other cryptocurrencies, in terms of transaction history? Most seem to use a reusable address, with an identifier for each transaction (in the case of Nano, the block hash).

I'm not sure what uses that might have, but it's a good shout and definitely something to consider. I'd suggest it may be better to have just one signature that covers all blocks under an account to reduce the overall file size, rather than signing each individual transaction.

For something like this, I'd recommend at least defining a separate key to contain custom info — perhaps 2 sections, one for crypto-specific attributes, and one for wallet/service data. While version numbering can help to prevent it somewhat, you don't want cryptos/wallets/services to start using a custom key name that could conflict with a later version of the format.

Hash probably should be general, though perhaps named something different like uid. This would act as a unique identifier for the transaction (I can't think of any cryptos that don't have a way to identify them) — in the case of Nano it'd be the block hash.

I thought one of the main purposes was to make it easier to integrate into external systems, which perhaps don't have access to the ledger? For instance, accounting tools are unlikely to need this, and it'd just add additional complexity to their integration.

Its true all a merch needs is block hash + ID to validate but the other data cannot hurt. They might do something with the (approximate) timestamps.

I suppose in this thread we are talking about two different standards - one for a "payment-id" handoff protocol, and one for metadata transfer. The way I see it these are both cross wallet standards that should go into one master specification. Its hard enough to ask multiple different companies/groups to implement one spec so it should all go in one.

I see value to both "payment-id" approaches and think it would be best to include both in a final specification. The signed pre-published block handoff method is useful if client is offline or has unstable wireless connection. The post-published block handoff method is more geared towards online merchants. As I said an online merch wouldn't need to operate a node (a huge benefit IMO).

Putting these thoughts together then, here is what a new "payment-id" QR code spec could look like:

{
    "dest_accout": "nano_merch_address",
    "amount": "1 nano",
    "memo": "online merch from xyz merchant",
    "identifier": "123 merch id for tracking",
    "handoff_type": "expects either pre or post published client block data",
    "mech_backend_url": "https://merch_backend/validate"
}

Also for the metadata specification: I think signing each transaction with account key is important. It makes it much harder for someone to fake memo, timestamp, and any other important data for tax / accounting. Imagine I bought a house and then faked the transaction metadata as a donation for taxes.

On that note I wonder if having a signed link between merch and client metadata accounts would be even better.

Looking up whether a block was confirmed via a third-party service (like NanoCrawler) really isn't recommended, as there's no guarantee of accuracy of information. They could always resort to using an external service to publish the block either way.

Originally, the design did include the option for both (hash and block handoff), however the hash handoff was removed as it added additional complexity with no benefit. The other benefit of the block handoff is that a payment processor can validate a block before the user sends it, reducing the chance of stuck funds. I won't go into it all that much more as it'll be announced over the next month, and it'll still just be a prototype/RFC at that as we try to collect further feedback.

You could sign the transaction and it still doesn't prove the data is true. All it verifies is that the sender 'approved' the message, not whether they published the block or whether it was confirmed. The point about a merchant signing transaction metadata could provide some use, though I'm still not sure what practical benefit it'd offer.

This risk could be mitigated by checking two separate explorer services. A small trade off for a merch not having to run a node. This is still a huge advantage I think. Not having to run a node to verify transactions lowers the barrier to entry. I am thinking mostly of online micro transactions here.

I agree this would also work but I don't know of any current, free, services doing this.

Basically my main point is to prevent accounting fraud. Nano confirms a transaction occurred but says nothing about why it occurred. There needs to be a way for 3rd parties / gov to verify the 'why'. If both merch and client sign thier local metadata with each others pub keys it creates a useful link for gov to verify the 'why'. If I say transaction A was a donation while merch says it was a purchase, then something obviously wrong and requires manual investigation.

That's actually a good point, I can see it having some practical uses acting as a proof of intent. That then raises the question of how you'd verify that their pubkey belongs to the business. For instance, what's to prevent me from creating a new Nano account and claiming it belongs to a charity, while still being able to sign the metadata? I suppose for large businesses, this information could be public knowledge or in a government database, but I feel that's not gonna be necessary for decades to come.

There's also the issue of security; requiring a merchant to sign metadata means their private key must be stored on yet another server, increasing the risk of an attack or leaking the key. While it should still be relatively simple to secure, it's likely going to be viewed as an additional risk services wouldn't be willing to take. But for smaller merchants with fewer funds attached to the account, I can't see it being much of an increased risk.

An updated version of this format to incorporate some of the recommended changes:

  • Make it protocol-agnostic using a "protocol" field at the account level.
  • Allow for protocol- and service-specific info at the account and transaction levels.
  • Include the transaction amount (the amount of crypto) and whether the transaction is sending or receiving funds.
  • Change name of "hash" field to "uid"

This time I've put example values in each field.

{
  "version": "1.0",
  "accounts": [
    {
      "protocol":"Nano",
      "address": "nano_1ebq356ex7n5efth49o1p31r4fmuuoara5tmwduarg7b9jphyxsatr3ja6g8",
      "private_name":"Bob's gas n' grocery wallet",
      "public_name":"Bob Dobson",
      "modified": 1634718547,
      "accounting_unit": "USD",
      "protocol_spec":{
        "representative":"nano_1ebq356ex7n5efth49o1p31r4fmuuoara5tmwduarg7b9jphyxsatr3ja6g8"
       },
      "wallet_spec":{
         "wallet_name":"Natrium",
         "custom_fields":"Arbitrary data"
       },
      "transactions": [
        {
          "uid": "C1150C61D10C7E42F7F9494C205A924D0CEB55683F8BAA723C89DCC038E84C42",
          "identifier": "65146",
          "published": 1624430775,
          "confirmed": 1624430776,
          "sendrec":"send",
          "amount": 6782692307000000000000000000000,
          "value": 35.27,
          "memo": "10.003 gallons unleaded at pump #7",
          "transactee": "Circle K Gas Station, 2011 N Lincoln Ave, Urbana IL 61801",
          "protocol-spec":{
              "index":107,
              "transactee_address":"nano_1natrium1o3z5519ifou7xii8crpxpk8y65qmkih8e8bpsjri651oza8imdd"
           },
           "wallet_spec":{}
        }, ... ]
    }, ... ]
}

If you want the ability for the transactee to sign the transactee and memo fields, you could append the signature to the end of the message. However, we would need to identify what data would be used alongside this to make the signature secure. The easy thing would be the block hash but that requires the block to have been created already, which means you couldn't send a signed memo until after the block has handed off, which might require an extra communication step.

2 Likes

Looks very nice!
Few questions.

  1. Does the "value": 35.27 field represent the dollar/accounting_unit amount?
    I think it might be smart add a few more decimals to avoid accounting issues. Readability wise I would prefer the accounting_unit and value to be a bit closer; either both in transaction or both not.
  2. Does this allow for encrypting the info, or does that require an extra field? Because I think this would be a valuable feature; that fiat banking also brings. It was already implemented here.

Does the "value": 35.27 field represent the dollar/accounting_unit amount?
I think it might be smart add a few more decimals to avoid accounting issues. Readability wise I would prefer the accounting_unit and value to be a bit closer; either both in transaction or both not.

Yes, the "value" field represents the asserted fiat/accounting currency value, typically based on whatever the current exchange rate is. It's just a JSON number so you can certainly include more decimal places. For tax purposes (cost basis, gross proceeds) the nearest cent is usually fine. As far as putting them together: the value will change with each transaction so it must be recorded at the transaction level. If you put the accounting unit in each transaction it would just increase the file size and leave it open for inconsistency. (What happens if you change accounting unit half way through?) This file is not really intended for readability, it will be managed behind the scenes by the wallet.

Does this allow for encrypting the info, or does that require an extra field?

Encryption could be done on the file as a whole. If you wanted to do it on a per-account basis it might make sense to put them into separate files rather than trying to do multiple encryptions within the container. This is pushing my knowledge on best practices.

Okay I looked up how nanomemo encrypts it. It's described here. You'd just need to somehow encrypt the message field with a public key, but you'd keep using the same field the message would just be gibberish:

So nothing needs to change, only the values would be different in case of encryption. And if you use a Diffie-Hellman key both the sending and receiving party would be able to decrypt the message.

"transactions": [
  {
   "memo": "8743b52063cd84097a65d1633f5c74f5",
   "transactee": "f0fda58630310a6dd91a7d8f0a4ceda2",
  }

So I guess the next question would be where would this metadata be hosted? Is it a possibility to (optionally) add this to the Nodes that are running the network?

@SomeNano already hosts something similar. Do you have any ideas?

So I guess the next question would be where would this metadata be hosted? Is it a possibility to (optionally) add this to the Nodes that are running the network?

It usually wouldn't be hosted, though the wallet service could store backups for you. This would be sotred on your phone, PC, business database, point-of-sale machine, etc. Certainly it wouldn't be stored by nodes on the network, else we might as well just add it to the nano ledger. If you need to transfer it from one device or wallet to another, just copy the file. That's what I was talking about in the initial post.

So nothing needs to change, only the values would be different in case of encryption. And if you use a Diffie-Hellman key both the sending and receiving party would be able to decrypt the message.

The point was not to encrypt the memo to keep it from being read. The entire transaction interaction will presumably be encrypted between the parties using HTTPS, NFC, etc. The point was to prove that the memo was written by the other party. For instance, if you donate to a charity they could sign it and you could prove that you donated to charity, assuming the address of the charity is static and publicly known. The message would say something like "charitable donation" as opposed to "conference room rental fees" or some other payment to the address. My understanding is that you would need to include some non-repeating information like the block hash so that you couldn't forge that signature again for some future transaction.

Okay, but the wallet apps currently just communicate with the Nano network. I assume you'd want to be able to send a message from, say, Natrium to WeNano and have the information be included right?

So this would require some service that provides this extra layer of communication for you. And I didn't mean all the nodes to host it, but for example you could optionally run a "heavy node" that also handles meta-data. But yeah as soon as the message is received it could be removed and then both the sender and receiver would take care of the storage.

It could be that I'm missing something simple here though. Also, I just found out that WeNano already has a message feature build in.

I assume you'd want to be able to send a message from, say, Natrium to WeNano and have the information be included right?

I think you need to clarify what use case you're envisioning. If you're just sending your friend some money, presumably you have a way to communicate with them already. (How did you get their nano address?) In that case you just make the transaction and tell them thru your communication channel that you sent them some money for X. You each enter your own respective memos for your own records, but otherwise the transaction is done. You probably already have them in your address book so you don't need to fill out the transactee field unless it's the first time you're transacting.

That being said, if someone wants to use a wallet for accepting payments in a more systematic way then the wallet would probably want to implement a payment request function and set up a URL to receive payment data, much like merchants would be doing. They would make a payment request link and share that, again over whatever their existing communication channel is. This would allow them to include a memo with their request.

Okay, but the wallet apps currently just communicate with the Nano network.

What other communications the wallet wants to implement is up to them. Making a fully fledged messaging service in addition to the nano wallet would be more work, but having payments integrated with messaging apps is certainly not a new idea. WeNano does messaging, GPS communication, and shares spot data between users, but those are all outside the scope of nano transactions themselves.

Ideally memo and some other basic data would be passed via QR codes. Heres the basic end2end flow

The case I'm envisioning is the same one I have with a normal banking app. I send money to an address, and I'll be able to add a message and it shows up next to their transaction. The app takes care of the message. I don't have to text it separately.

Did you mean the metadata just to be stored in the app alone? The memo was just for yourself?

Did you mean the metadata just to be stored in the app alone? The memo was just for yourself?

Yes. The purpose of this format was storing it for future reference by the user/wallet. But it was still important that we define it, so that we can then discuss what fields might be populated automatically by the other party in various transaction (seems like memo/transactee are the main ones).

The case I'm envisioning is the same one I have with a normal banking app. I send money to an address, and I'll be able to add a message and it shows up next to their transaction. The app takes care of the message. I don't have to text it separately.

P2P payment + messaging would be a separate feature to develop. It would be easy enough if they are using the same wallet, but if they are using different wallets then the wallet developers will need to come to an agreement on this standard as well. It would probably end up looking similar to email, e.g. nano_abc...xyz@natirum.io At least by defining the metadata format here we can easily tell what fields should be transmitted in this interaction.

1 Like