Octav Runs Local AI on Apple M3 Ultra with Dual-Instance Setup

🖥️ Octav ditches cloud providers

By Octav
Feb 9, 2026, 2:03 PM
twitter

Octav is running LlamaCPP locally on Apple M3 Ultra hardware using the GLM-4.​7-Flash Q4_K_M model to eliminate cloud provider dependencies.​

Technical setup:

  • 2 instances running simultaneously
  • Each instance handles 2 parallel slots for controlled concurrency
  • Nginx load balancer distributes requests across instances
  • Metal acceleration utilized for performance optimization

Key benefits:

  • Removes reliance on external cloud providers
  • Enables reliable 24/7 task execution
  • Provides full control over context and throughput
  • Maintains low latency through efficient load balancing
  • Prevents memory limit issues through predictable concurrency management

The local deployment strategy prioritizes infrastructure independence and operational reliability over cloud-based alternatives.​

Sources
Read more about Octav

Octav Optimizes Local LLM Setup with Llama Throughput Lab

Octav fine-tuned their local LLM infrastructure using **Llama Throughput Lab** to maximize performance on Apple M3 Ultra hardware. **Key optimizations:** - Batch and micro-batch sizes - Slot management - Parallelism configuration - Context window efficiency The setup runs **LlamaCPP locally** with GLM-4.7-Flash Q4_K_M model, using 2 instances with 2 parallel slots each, fronted by Nginx load balancer. **Why local deployment?** - Removes cloud provider dependency - Enables reliable 24/7 task execution - Provides full control over context and throughput The optimizations achieved maximum speed without crashing Metal, demonstrating practical approaches to self-hosted AI infrastructure. Tool: [Llama Throughput Lab](https://github.com/alexziskind1/llama-throughput-lab)

Octav Launches Virtual Balance Feature for Off-Chain Crypto Tracking

**Octav has introduced Virtual Balance**, a new feature that allows users to manually track cryptocurrency holdings that exist outside blockchain networks. **Key capabilities:** - Add manual balances for custodial accounts and OTC deals - Track off-chain holdings alongside on-chain wallets - View complete net worth in one unified dashboard The feature addresses a gap in portfolio management, as not all crypto assets are stored on public blockchains. Users can now bundle their exchange balances, custodial accounts, and traditional wallet addresses for comprehensive tracking. **Available now** at [pro.octav.fi](http://pro.octav.fi)

Octav Integrates Webacy Risk Intelligence Widget for Wallet Security Monitoring

Octav has partnered with Webacy to integrate a Risk Scores widget directly into the Octav dashboard, enabling users to monitor wallet security in real-time. **Key Features:** - Check if wallets have interacted with risky smart contracts - Identify rug pull exposure - Detect scam connections The widget is now live at [pro.octav.fi/widget-explorer/walletRiskScore](https://pro.octav.fi/widget-explorer/walletRiskScore). Additional details about the integration are available at [Webacy's announcement](https://world.webacy.com/dd-in-the-wild-octavs-integrated-risk-intelligence/). This integration provides Octav users with enhanced security monitoring capabilities, allowing them to assess potential risks associated with their wallet activities directly from their dashboard.

Octav Partners with Otomato to Add DeFi Position Alerts

Octav has integrated with Otomato to enable users to set up alerts for their DeFi positions directly within the app. Users can now receive email notifications about their portfolio activity, providing real-time insights and increased transparency for their DeFi holdings. **Key features:** - In-app alert configuration for DeFi positions - Email notification system - Real-time portfolio monitoring This partnership follows Octav's recent collaboration with Blockscout, which made Blockscout the default blockchain explorer in the Octav app.

DeFiKnowledge