Standardizing transaction metadata for an inter-wallet format

A key UX feature that seems to be missing from wallets at the moment is the ability to record metadata associated with a transaction, such as what the transaction was for, when it took place, who the recipient/sender is, how much it was worth in your accounting currency, and so forth. This information is not only useful to a user for understanding their own transaction history, but it would also form the basis for important functions like calculating capital gains tax or leaving paper trails, which are directly relevant to business adoption.

Metadata that is only of interest to one or a few parties should not be recorded on the chain of course, but it should certainly be supported as local/centralized data stored by the user's wallet. This data should be able to be backed up or imported to other wallets, making it useful to them even if they use a different software or if they lose their device. To that end, I think we as a community should go ahead and create a standard format for this data, which can then be implemented by wallet developers.

Here is the concept in json form:

{
  "version": "Version number of the format",
  "accounts": [
    {
      "protocol":"Which cryptocurrency this account represents",
      "address": "Public address of the account",
      "private_name":"String for the user to identify the account",
      "public_name":"String that can be automatically provided to other wallets' transactee field",
      "modified": "Unix timestamp of when this account metadata was last updated",
      "accounting_unit": "String representing what currencny accounting is done in, if any. For instance, USD. This is relevant to the value field for transactions.",
      "protocol_spec":{
           "eg":"account fields that are specific to the crypto protocol, such as representative in Nano"
       },
      "wallet_spec":{
           "eg":"account fields that are specific to the wallet or service managing this file"
       },
      "transactions": [
        {
          "uid": "Unique identifier for the transaction in the ledger (in Nano, the block hash) that this entry corresponds to",
          "identifier": "Arbitrary id string for associating transactions with other databases, e.g. a business's records",
          "published": "Unix time stamp of when the wallet first broadcast the block to the network",
          "confirmed": "Unix time stamp of when the wallet received confirmation of the block from the network",
          "sendrec":"Whether this transaction is sending or receiving funds (send/rec respectively)",
          "amount":"Amount of the cryptocurrency transferred in this transaction",
          "value": "Amount that this transaction was taken to be worth in whatever the accounting_unit is. For instance, a business that accounts in USD would record the USD value of the transaction. This can automatically be populated using current market price, but it's possible to exchange nano at non-market prices as well. This is relevant to calculating capital gains taxes.",
          "memo": "Human-readable string describing the purpose of the transaction. Can be entered by the user manually, or populated by the wallet software automatically.",
          "transactee": "Human-readable string describing the person or entity the block is interacting with; can be automatically populated using the address book, if applicable",
          "protocol_spec":{
             "eg":"Transaction fields that are specific to the crypto protocol, such as block height or nano address of trasnsactee"
           },
           "wallet_spec":{
               "eg":"Transaction fields that are specific to the wallet or service managing this file"
           },
        }, ... ]
    }, ... ]
}

These files could be given a file extension of *.txmd ("transaction metadata"). Below is an example of how this format might look for a real account.

{
  "version": "1.0",
  "accounts": [
    {
      "protocol":"Nano",
      "address": "nano_1ebq356ex7n5efth49o1p31r4fmuuoara5tmwduarg7b9jphyxsatr3ja6g8",
      "private_name":"Bob's gas n' grocery wallet",
      "public_name":"Bob Dobson",
      "modified": 1634718547,
      "accounting_unit": "USD",
      "protocol_spec":{
        "representative":"nano_1ebq356ex7n5efth49o1p31r4fmuuoara5tmwduarg7b9jphyxsatr3ja6g8"
       },
      "wallet_spec":{
         "wallet_name":"Natrium",
         "custom_fields":"Arbitrary data"
       },
      "transactions": [
        {
          "uid": "C1150C61D10C7E42F7F9494C205A924D0CEB55683F8BAA723C89DCC038E84C42",
          "identifier": "",
          "published": 1624430775,
          "confirmed": 1624430776,
          "sendrec":"send",
          "amount": 6782692307000000000000000000000,
          "value": 35.27,
          "memo": "10.003 gallons unleaded at pump #7",
          "transactee": "Circle K Gas Station, 2011 N Lincoln Ave, Urbana IL 61801",
          "protocol-spec":{
              "block_height":107,
              "transactee_address":"nano_1natrium1o3z5519ifou7xii8crpxpk8y65qmkih8e8bpsjri651oza8imdd"
           },
           "wallet_spec":{}
        }, ... ]
    }, ... ]
}

There's little reason for the wallet to fill out the "identifier" field in this case since Bob isn't running a business and storing mixed transactions in a database.

Feel free to comment with any thoughts on what information should be included in this standard, how it should be formatted, etc.

2 Likes

Creating this standard would also be a good starting point for building block handoff or other inter-party transaction standards where transaction metadata is provided by the other party. For instance, if you buy something from a store, their PoS machine could automatically provide data for the transactee field (name of the store) and the memo field (itemized receipt and/or order number).

Love this idea. I also agree we need a standard for sharing meta data across services.

A standardized QR code format would go nicely with this to provide a webhook callback url to send the meta data too. This would basically solve the payment ID problem.

In the official payment ID solution proposal the focus is passing a signed block back to the merchant for them to post via their own node. I think its a bit over complicated and a simple webhook callback url with block metadata would be perfect.

So basically standardized QR fields:
dest_account (required),
amount (required),
webhook_url (optional),
memo (optional)

After confirmation the wallet populates metadata locally with above format and sends a copy to webhook_url

1 Like

Agreed that it seems very overcomplicated.

See also my comment here, which also links back to related discussions going back ~2 years.

I really like this concept, it seems like it'd prove to be very useful for managing wallets and importing into accounting tools as you mentioned.

Perhaps it would be better to design a crypto-agnostic solution, creating a standardized format that can be used by all other cryptocurrency wallets and accounting software. Are you aware of any existing formats or standards that are available for other cryptocurrencies that could instead be adapted to support Nano?

Personally I don't feel this would provide much of a purpose to the handoff protocol — that requires its own set of parameters and is for dealing with the processing of blocks, whereas the standard you propose seems to be aimed at storing and transferring historical data of already existing blocks. I've been working on designing and refining the handoff protocol the last few months, there'll be more info on that published soon if all goes well.

Some quick suggestions:

  • Provide support for other cryptocurrencies (perhaps allowing multiple types to be defined in a single instance of the format?)
  • Provide a way for additional data fields to be defined (either custom metadata from the wallet, or crypto-specific parameters)
  • Personally I'm not so sure about the inclusion of the fiat currency - exchange rates can differ between sources, and different apps may expect or support different currencies. Perhaps including the crypto amount would be better, instead allowing the wallet/accounting software to make the conversions?

I think they are still related concerns. For instance the memo field for each transaction can be auto populated by a merchant via the QR code. As well, if a webhook url is passed through QR code and confirmed block hash + metadata sent there, it makes payment tracking very simple. You wouldn't even need to run a node to track payment - a NanoCrawler api lookup of the block hash would suffice to verify details.

Nice! Looking forward to hearing your ideas

I can see how they'd work in conjunction with each other, but I'm not sure how this transaction export format could be used directly within the handoff protocol itself (if that's what you're suggesting?).

As the OP's concept shows, the format is for a list of blocks within a list of accounts, rather than just a single block. What metadata do you feel would be relevant for the payment processor? Surely this information would only be relevant to the account holder?

I feel I'm misunderstanding how people are suggesting this could be integrated into the handoff protocol, so if you could help clarify it for me (or provide any other suggestions) I'd greatly appreciate it.

Here is an end2end flow with the proposed standards working together:

  1. Merchant presents QR code with embedded fields:
{
    "dest_accout": "nano_merch_address",
    "amount": "1 nano",
    "memo": "online merch from xyz merchant",
    "identifier": "123 merch id for tracking",
    "webhook_url": "https://merch_backend/validate"
}
  1. User scans code and wallet completes transfer like normal.
  2. Wallet then creates a new metadata object (using OP's format):
{
    "index": "Index of the block in the account chain that this record applies to",
    "hash": "Hash of the block that this record applies to",
    "identifier": "123 merch id for tracking",
    "published": "1633821520",
    "confirmed": "1633821530",
    "value": "5.53 USD (1 nano at this time)",
    "memo": "online merch from xyz merchant",
    "transactee": "online merch from xyz merchant (if in address book)"
}
  1. Wallet updates local metadata with json (3) by adding to OPs "transactions" list
  2. Wallet posts json (3) to provided webhook_url: https://merch_backend/validate

Later on a user can export the local metadata to accounting software or other nano wallets.

A couple other thoughts:

  • Not sure if other standards exist. I suspect Nano will require something unique due to its DLT nature.
  • each metadata transaction object should also be signed with account key so once its exported anyone can verify the metadata relates to that nano account.
1 Like

The last time I was looking into paying taxes it seemed that all the available solutions involved pointing the software at the assorted sources of the information needed, (e.g. to the crypto ledger via your address, your exchange accounts, etc) rather than getting all of that information from a standardized wallet source. This doesn't mean there are no wallet solutions, but at least they weren't widely adopted enough that the accounting solutions relied on them.

Perhaps it would be better to design a crypto-agnostic solution, creating a standardized format that can be used by all other cryptocurrency wallets and accounting software.

Well I think it should be pretty easy to distinguish which of these fields are general to currencies and which are specific to nano. Any conflicts would arise with the overall structure: are there cryptocurrenceis which can't be organized by account + transaction? Otherwise, you could have a field at the "account" level which identifies which cryptocurrency the account belongs to. The standard would then have a set of required fields (the general ones) and each cryptocurrency could then have a set of additional fields that make sense in the context of their own protocol.

Things that would probably be general:

  • crypto protocol
  • address
  • private name
  • public name
  • modified
  • accounting unit
  • identifier
  • amount
  • value
  • memo
  • transactee

Things that would not be general:

  • index
  • hash

Things that I'm not sure of:

  • time published
  • time confirmed

I would think these two are general, as long as the cryptocurrency has a reasonably identifiable confirmation point (e.g. bitcoin reaching N blocks deep).

Provide a way for additional data fields to be defined (either custom metadata from the wallet, or crypto-specific parameters)

I'm not sure this even needs to be defined. Anyone adopting this format could add their own fields, and any software reading the format could just skip fields that it doesn't understand.

Personally I don't feel this would provide much of a purpose to the handoff protocol — that requires its own set of parameters and is for dealing with the processing of blocks, whereas the standard you propose seems to be aimed at storing and transferring historical data of already existing blocks

Ideally all of these fields would be populated at the time of transaction, rather than after the fact, and some of them would ideally be populated by the opposing party. In practice I would expect that the transaction process would simply involve the exchange of a generic "metadata" chunk that contains an assortment of fields like this from each party. The wallet of the other party can then choose which metadata fields it cares to integrate into its own records of the transaction. Sort of like "throwing it at the wall and seeing what sticks." For instance, the merchant might store the "transactee" field from the customer. The customer might store the "memo" and "transactee" fields from the the merchant, and if their "accounting_unit" matches your own you would store their "value" field because the merchant is likely the one defining the exchange rate of the transaction.

  • Personally I'm not so sure about the inclusion of the fiat currency - exchange rates can differ between sources, and different apps may expect or support different currencies. Perhaps including the crypto amount would be better, instead allowing the wallet/accounting software to make the conversions?

The crypto amount could be included. It would be redundant since it could be read from the ledger, but there's no strong reason not to have the field and it would make it easier for accounting software to just read this value rather than interact with the network. If the amounts disagree it might be cause for further error checking.

But the fiat currency should be included, because there's no other good place to store this information. In principle two parties could agree to make an exchange where the Nano is valued drastically differently from the current market price, and you would need to record that. Not everyone has to use this field, but if they do use it this is the logical place to have it. Eventually of course we would hope that people are accounting in nano and this field becomes obsolete. :slight_smile:

I feel I'm misunderstanding how people are suggesting this could be integrated into the handoff protocol, so if you could help clarify it for me (or provide any other suggestions) I'd greatly appreciate it.

I might have answered this question earlier in this reply. Basically at some point during the transaction communication they simply trade a chunk of metadata that they populate with their own info, and let the other do with it what they will. For a merchant it might make sense to include their chunk in the payment request, which lets the customer wallet display all of this information while they authorize the tx. The customer they could include their chunk it when they hand off the block, since they are already sending data at this step.

The first JSON schema is very similar to what we have for the current revised handoff protocol, though with a few more options in addition to that.

I see what you mean. I still don't understand what the need for this specific standard/schema is though (in regards to the handoff protocol) — as far as the payment processor/merchant is concerned, all they would really need is the ID and block hash, so all the additional metadata would be irrelevant.

The newer handoff protocol makes some additional changes that also break this, namely the fact that the wallet no longer publishes the block themselves; instead, they pass ("hand off") the block directly to the payment processor for them to validate, generate work, and publish on the sender's behalf.

What do you think is unique to Nano that isn't present in other cryptocurrencies, in terms of transaction history? Most seem to use a reusable address, with an identifier for each transaction (in the case of Nano, the block hash).

I'm not sure what uses that might have, but it's a good shout and definitely something to consider. I'd suggest it may be better to have just one signature that covers all blocks under an account to reduce the overall file size, rather than signing each individual transaction.

For something like this, I'd recommend at least defining a separate key to contain custom info — perhaps 2 sections, one for crypto-specific attributes, and one for wallet/service data. While version numbering can help to prevent it somewhat, you don't want cryptos/wallets/services to start using a custom key name that could conflict with a later version of the format.

Hash probably should be general, though perhaps named something different like uid. This would act as a unique identifier for the transaction (I can't think of any cryptos that don't have a way to identify them) — in the case of Nano it'd be the block hash.

I thought one of the main purposes was to make it easier to integrate into external systems, which perhaps don't have access to the ledger? For instance, accounting tools are unlikely to need this, and it'd just add additional complexity to their integration.

Its true all a merch needs is block hash + ID to validate but the other data cannot hurt. They might do something with the (approximate) timestamps.

I suppose in this thread we are talking about two different standards - one for a "payment-id" handoff protocol, and one for metadata transfer. The way I see it these are both cross wallet standards that should go into one master specification. Its hard enough to ask multiple different companies/groups to implement one spec so it should all go in one.

I see value to both "payment-id" approaches and think it would be best to include both in a final specification. The signed pre-published block handoff method is useful if client is offline or has unstable wireless connection. The post-published block handoff method is more geared towards online merchants. As I said an online merch wouldn't need to operate a node (a huge benefit IMO).

Putting these thoughts together then, here is what a new "payment-id" QR code spec could look like:

{
    "dest_accout": "nano_merch_address",
    "amount": "1 nano",
    "memo": "online merch from xyz merchant",
    "identifier": "123 merch id for tracking",
    "handoff_type": "expects either pre or post published client block data",
    "mech_backend_url": "https://merch_backend/validate"
}

Also for the metadata specification: I think signing each transaction with account key is important. It makes it much harder for someone to fake memo, timestamp, and any other important data for tax / accounting. Imagine I bought a house and then faked the transaction metadata as a donation for taxes.

On that note I wonder if having a signed link between merch and client metadata accounts would be even better.

Looking up whether a block was confirmed via a third-party service (like NanoCrawler) really isn't recommended, as there's no guarantee of accuracy of information. They could always resort to using an external service to publish the block either way.

Originally, the design did include the option for both (hash and block handoff), however the hash handoff was removed as it added additional complexity with no benefit. The other benefit of the block handoff is that a payment processor can validate a block before the user sends it, reducing the chance of stuck funds. I won't go into it all that much more as it'll be announced over the next month, and it'll still just be a prototype/RFC at that as we try to collect further feedback.

You could sign the transaction and it still doesn't prove the data is true. All it verifies is that the sender 'approved' the message, not whether they published the block or whether it was confirmed. The point about a merchant signing transaction metadata could provide some use, though I'm still not sure what practical benefit it'd offer.

This risk could be mitigated by checking two separate explorer services. A small trade off for a merch not having to run a node. This is still a huge advantage I think. Not having to run a node to verify transactions lowers the barrier to entry. I am thinking mostly of online micro transactions here.

I agree this would also work but I don't know of any current, free, services doing this.

Basically my main point is to prevent accounting fraud. Nano confirms a transaction occurred but says nothing about why it occurred. There needs to be a way for 3rd parties / gov to verify the 'why'. If both merch and client sign thier local metadata with each others pub keys it creates a useful link for gov to verify the 'why'. If I say transaction A was a donation while merch says it was a purchase, then something obviously wrong and requires manual investigation.

That's actually a good point, I can see it having some practical uses acting as a proof of intent. That then raises the question of how you'd verify that their pubkey belongs to the business. For instance, what's to prevent me from creating a new Nano account and claiming it belongs to a charity, while still being able to sign the metadata? I suppose for large businesses, this information could be public knowledge or in a government database, but I feel that's not gonna be necessary for decades to come.

There's also the issue of security; requiring a merchant to sign metadata means their private key must be stored on yet another server, increasing the risk of an attack or leaking the key. While it should still be relatively simple to secure, it's likely going to be viewed as an additional risk services wouldn't be willing to take. But for smaller merchants with fewer funds attached to the account, I can't see it being much of an increased risk.