This section describes several messages that are exchanged to set up and complete an audit. The following message flow diagram shows the message flow for the successful case when the audit of the entire database is successful.
More explanation is provided after the diagram to describe the purpose of each step in the process. Later, in this chapter, each message is described in detail.
Successful Audit High Level Message Flow
Disclaimer: Any images in this article are intended to be used as a guide for educational purposes only and they may differ slightly from the Production Environment.
The diagram shows both the TSS and the TFT Router making copies of the database. They use the copyAtSequence to coordinate the point when the copy should be made. If all is well, the two databases match perfectly -- and the way the databases are compared is by having both sides calculate the hash of the entire database and then the TSS compares the hashes to know if the databases match. The calculation of the hash has to be done exactly the same by both sides, and it is described in more detail later. Also, the above diagram has embedded steps that are explained in more detail here:
The TFT Router periodically initiates the audit. Ask the TSS operator for the frequency that the audits are expected. If an audit is not performed in a timely manner, the TSS can use its discretion as to what action to take.
The TFT Router sends a REST/JSON message to the “/tft-audit” URL. This message is described in more detail later in this chapter.
Upon receipt of the audit URL, the TSS makes a copy of its database (a snapshot). The copy is actually not mandatory -- what is mandatory is that the data being used during the audit does not change. To accommodate failure scenarios, the audit may take several minutes. Therefore, for convenience, the word “copy” is used loosely in this document. Since data is continually being added and removed, some type of snapshot (or download suspension) provides a fixed target to audit. The TSS and TFT Router can decide independently on their approaches so long as the audited database is not changing during the resolution of audit failure scenarios.
When the TSS makes the copy of the database, it also records the “last” sequence number that is waiting to be downloaded by the TFT Router (copyAtSequence). That sequence number has already been consumed by the TSS, so the TFT Router must consume it as well before the audit can continue. Once that is recorded, and since the TSS DB has been copied, the TSS can resume normal operations and work only on the DB copy for the audit.
The TFT Router downloads the events up to the copyAtSequence value provided by the TSS. It leaves any remaining events on the download queue until it can make a copy of the DB. The diagram does not show the downloading of those events - they occur through the normal download URL process. Please refer to the Download section of this document.
Once the copyAtSequence value has been consumed by the TFT Router (and importantly before it consumes any events after that sequence number) the TFT Router creates a copy of its DB to use for the audit. At this point, both the TSS and the TFT Router have DB snapshots that were made immediately after the copyAtSequence event was consumed by them. Ideally, the two database copies match. The rest of the steps determine if that is the case.
The TFT Router calculates a hash value for its audit database copy. The algorithm is described later in this document and requires data being fed into the SHA-1 algorithm in a very specific order -- this allows the TSS and the TFT Router to use independent schemas, so long as they adhere to the exact same hash algorithm and the exact order of input to that algorithm. The reader will also notice that the string “8” is passed into the calculateHash(“8”) algorithm (see the hashing section later in this chapter). The “8” is intentional and indicates to the algorithm to hash all TFNs that start with 8. In regular expression terms one could think of it as “8\d{9}” or “8\d\d\d\d\d\d\d\d\d” or perhaps, more loosely, “8*”. In SQL terms, this can be thought of as “WHERE (tfn LIKE '8%')”.
Once the TFT Router has the hash value, it sends both the partialTfn value (“8”) and the result of calculateHash - the SHA-1 40 printable character hash - to the TSS.
The TSS independently performs the exact same hash calculation on its DB. The diagram shows this as happening after the prior step, but in fact, the TSS could have performed this calculation at any time after it created the audit database copy.
At this point, the TSS is in possession of both the TFT Router’s hash as well as its own hash, of their respective audit database copies. If the two hashes match exactly, the TSS concludes that the databases match exactly. Understanding the distribution of the SHA-1 algorithm is outside of the scope of this document and is regarded as “off the shelf” since there are many implementations available to calculate a SHA-1 message digest (hash). Upon success, the TSS returns a REST/JSON message indicating success.