Indexing in the TON Blockchain: An Insight into TON-20 Token Operations

Understanding the TON-20 Indexer Architecture

The indexer on the TON Blockchain plays a critical role in scanning all transactions on the network. Unlike traditional blockchain tokens that use smart contracts to enforce rules on-chain, TON-20 tokens rely entirely on off-chain indexers to interpret and validate token operations embedded in transaction data.

graph TD
    A[TON Blockchain<br/>All Shards] -->|Continuous Monitoring| B[Indexer Scans<br/>Every Transaction]
    B --> C{Contains<br/>TON-20 Inscription?}
    C -->|No| D[Skip Transaction]
    C -->|Yes| E[Extract JSON Data]
    E --> F{Validate Format<br/>& Operation}
    F -->|Invalid| G[Reject & Log Error]
    F -->|Valid Deploy| H[Register New Token<br/>First-Come-First-Serve]
    F -->|Valid Mint| I[Check Supply Limits]
    F -->|Valid Transfer| J[Verify Balance]
    H --> K[Update State Database]
    I --> K
    J --> K
    K --> L[Serve API Queries<br/>to Wallets & dApps]

    style A fill:#2196F3
    style B fill:#FF9800
    style F fill:#9C27B0
    style K fill:#4CAF50
    style L fill:#00BCD4

5-Stage Indexing Pipeline:

1. Transaction Monitoring: Scan all shards continuously
2. Inscription Detection: Identify p: "ton-20" JSON data
3. Operation Classification: Deploy/Mint/Transfer validation
4. State Management: Update balances in off-chain database
5. API Service: Provide balance queries to applications

The indexer’s primary function is to parse the inscription content of each transaction initiated by an address, which may include deployment, minting, and transfer operations. This architecture draws inspiration from Bitcoin’s Ordinals and BRC-20 standards, where inscription data is embedded in blockchain transactions but interpreted by external systems.

The Indexing Process: A Technical Deep Dive

The TON-20 indexer operates through a multi-stage pipeline:

1. Transaction Monitoring: The indexer continuously scans the TON blockchain, examining every transaction across all shards. Given TON’s dynamic sharding architecture capable of processing up to 100,000 transactions per second, this represents a significant computational challenge.

2. Inscription Detection: When a transaction contains data matching the TON-20 inscription format (JSON data with p: "ton-20"), the indexer extracts and validates the inscription content.

3. Operation Classification: The indexer categorizes inscriptions into three types:

   • Deploy operations that establish new token parameters
   • Mint operations that create new token units
   • Transfer operations that move tokens between addresses

4. State Management: The indexer maintains a comprehensive database of all token states, including total supply, individual balances, and transaction history.

5. Validation Logic: The indexer enforces rules such as “first-deploy-wins” (only the first deployment inscription for a ticker is recognized) and supply limits defined in deployment inscriptions.

The indexer recognizes minting and transfer inscriptions of TON-20 tokens only after a deployment inscription for TON-20 has occurred. This sequential dependency creates a critical ordering requirement that the indexer must maintain across potentially millions of transactions.

Key Technical Observations

1. Off-Chain Computation Model

Inscriptions for TON-20 Deployment, Minting, and Transfer: These inscriptions are recorded on the blockchain but are not computed by the blockchain system itself. This represents a fundamental architectural difference from smart contract-based tokens like Jettons:

• On-Chain: Transaction data is immutable and publicly verifiable
• Off-Chain: Token state calculations happen in the indexer’s database
• Trust Assumption: Users must trust the indexer’s implementation and data integrity

This design trade-off prioritizes simplicity and low transaction costs at the expense of requiring trusted indexer infrastructure.

2. Sequential Dependency Chain

Dependence on Indexer for Token Operations: For TON-20 tokens, both deployment and minting transactions must be indexed successfully; otherwise, subsequent transactions are considered unsuccessful.

Consider this scenario:

Block 1000: Deploy "NANO" token (max supply: 21M)
Block 1001: User A mints 100 NANO
Block 1002: User A transfers 50 NANO to User B

If the indexer fails to process Block 1001, the transfer in Block 1002 becomes invalid despite being recorded on-chain. This creates a critical single point of failure where indexer downtime or desynchronization can invalidate otherwise valid transactions.

3. Stateful Balance Aggregation

Balance Calculation for TON-20 Tokens: The computation of individual TON-20 token balances is dependent on the indexer, which aggregates data from both deployment and minting transactions.

The indexer maintains complex state through cumulative operations:

• Starting from genesis, it processes every relevant transaction chronologically
• For each address, it calculates: Balance = Σ(mints) + Σ(incoming transfers) - Σ(outgoing transfers)
• This requires scanning potentially millions of historical transactions to compute current balances

Unlike smart contract tokens where balances are stored in contract state, TON-20 balances must be recomputed from transaction history. This makes balance queries computationally expensive and introduces latency.

4. Optimistic Transaction Model

Transactions Without Indexer Verification: Users can initiate transactions without consulting the indexer for balances, potentially leading to transactions that are successfully launched but not recognized by the indexer due to insufficient funds.

This “optimistic” model creates several edge cases:

• A user can broadcast a transfer transaction with insufficient balance
• The transaction succeeds on-chain (costs gas fees) but is rejected by the indexer
• Users lose gas fees for invalid transactions
• Wallet applications must query indexers before allowing transactions to prevent this

This differs from smart contract tokens where the contract itself validates sufficient balance before allowing transfers.

Critical Challenges with Indexer Centralization

The reliance on centralized indexer infrastructure introduces several systemic risks and technical challenges:

Transparency and Completeness Concerns

While blockchain data is publicly available, there’s no guarantee that the indexer has captured all transactions. Several factors can cause indexing gaps:

• Network Partitions: During the December 2023 TON-20 surge, shard desynchronization caused ~2.5 million transactions to be temporarily unprocessed
• Implementation Bugs: Different indexer implementations may interpret edge cases differently, leading to inconsistent token states
• Historical Gaps: If an indexer starts syncing after genesis, it must replay all historical transactions—any missing blocks create permanent inconsistencies

Timeliness and Real-Time Processing

Ensuring real-time indexing of transactions is crucial but challenging at TON’s scale:

• Throughput Bottleneck: With TON capable of 100,000 tps, indexers must process transactions faster than they’re created to avoid falling behind
• Sharding Complexity: TON’s dynamic sharding means indexers must monitor multiple shards simultaneously and maintain correct transaction ordering across shards
• Reorg Handling: Blockchain reorganizations (though rare on TON) require the indexer to roll back state and reprocess transactions

Computational Accuracy and Verification

Even with timely indexing, the correctness of the computational functions used by the indexer must be verified:

• Open Source Requirement: For trustless verification, indexer source code must be public and auditable
• Deterministic Replay: Different indexers running the same code should produce identical token states
• Edge Case Coverage: The indexer must handle malformed inscriptions, duplicate ticker symbols, and adversarial inputs correctly

Real-world example: If two users simultaneously deploy inscriptions for ticker “DOGE”, the indexer must consistently recognize only the first one based on block height and transaction index.

Latency in Balance Updates

There are significant delays and inaccuracies in the TON-20 balance calculations as performed by the current indexer:

• Query Response Time: Balance queries may take seconds as the indexer computes cumulative state from thousands of transactions
• Cache Invalidation: When new transactions arrive, cached balances become stale and must be recomputed
• Race Conditions: Users querying balances while transactions are being processed may see inconsistent states

Solutions and Future Enhancements

To enhance the performance and accuracy of TON-20 token operations, the community is exploring several technical approaches:

Decentralized Indexer Network

Development of Multiple Independent Indexers: Rather than relying on a single indexer, the ecosystem benefits from multiple independent implementations:

• Consensus Mechanism: Multiple indexers can cross-validate token states, with disagreements flagged for manual review
• Redundancy: If one indexer fails or falls behind, others can continue serving queries
• Open Source Competition: Multiple teams building indexers drives innovation and bug discovery

Projects like ton-nft-explorer and various community indexers are already emerging to provide alternative implementations.

Enhanced Transparency

Transparency in Computation Logic: Publicly revealing the logic behind balance computations allows for transparent and precise verification by all users:

• Open Source Repositories: All indexer code should be publicly auditable on GitHub
• State Snapshots: Publishing periodic state snapshots (e.g., all balances at block height X) enables independent verification
• API Standards: Standardized indexer APIs allow wallets to switch between indexers seamlessly

Performance Optimization

Rapid and Accurate Balance Retrieval: Ensuring that users can quickly and accurately access their TON-20 balances is vital:

• Database Indexing: Optimized database schemas with proper indexing on address and ticker fields
• Caching Strategies: Multi-layer caching (in-memory, Redis, CDN) to serve frequent queries instantly
• Incremental Updates: Rather than recomputing all balances, maintain incremental state changes
• Parallel Processing: Distribute indexing across multiple worker threads to keep pace with blockchain throughput

Comparison with Other Inscription Systems

TON-20’s indexer architecture shares similarities with other inscription-based token systems but has unique characteristics:

TON-20 indexers face unique scaling challenges due to TON’s high throughput and sharding, but benefit from faster finality reducing reorg concerns.

Practical Implications for Users and Developers

For Token Holders

• Multiple Indexer Queries: Check balances across multiple indexers before executing large transfers
• Transaction Confirmation: Wait for indexer confirmation (not just blockchain confirmation) before considering transfers complete
• Trusted Interfaces: Use wallet applications that integrate with reliable, well-maintained indexers

For Developers

• Error Handling: Implement robust retry logic for indexer API failures
• Balance Validation: Always query indexer balance before allowing user-initiated transfers
• Indexer Selection: Provide users with options to select their preferred indexer
• Local Validation: Consider running light indexer clients for critical applications

For Indexer Operators

• Monitoring: Implement comprehensive monitoring for indexing lag, error rates, and state consistency
• Disaster Recovery: Maintain regular state backups to enable quick recovery from failures
• API Rate Limiting: Protect infrastructure while ensuring fair access for legitimate applications
• Documentation: Provide clear API documentation and status pages for developers

Conclusion: The Path Forward

This detailed examination sheds light on the intricate workings of the TON-20 token operations within the TON Blockchain. The indexer-based architecture represents an innovative approach to token systems that prioritizes simplicity and low transaction costs, drawing inspiration from Bitcoin’s OP_RETURN inscriptions.

The article highlights the crucial role of the indexer in managing token operations and the need for improvements in its efficiency and transparency to ensure the reliability and accuracy of transactions on the blockchain. As the TON ecosystem matures, we expect to see:

1. Standardization: Community-driven standards for indexer behavior and APIs
2. Decentralization: Multiple independent indexers providing redundancy
3. Optimization: Performance improvements enabling sub-second balance queries at scale
4. Hybrid Approaches: Potential integration with smart contracts for critical validations

The evolution of TON-20 indexing infrastructure will be crucial for the broader adoption of inscription-based tokens on the TON blockchain, balancing the trade-offs between decentralization, performance, and user experience.