Unnoticeably, time has already slipped into the second day of 2026. I wanted to write something so that 2025 wouldn't end up without a single article, but alas, I'm lazy. Recently on holiday at home, I've basically done nothing except tinker with various things, but I haven't even gotten around to redoing the waterproof sealant in the kitchen. Here's just a summary of my recent journey of tinkering with tools to save my history logs.

At the very beginning, I used Google's Timeline History, which I recall started in 2010. In 2025, this service officially entered Google's graveyard. I went to export the data once, but unfortunately, before I executed the export, it had already expired my data from the 2010s, so basically nothing was exported. In 2015, I started using Moves. Later, this app was acquired by Facebook, and in 2018, it also entered Facebook's graveyard. After tinkering around in 2018, I started using Arc and continued until recently when I switched to Overland.

Arc's data is stored on iCloud. Initially, I was somewhat critical of its requirement to download a model based on my location to identify various types of activities. But in reality, I didn't care about this identification at all (Moves had it too, and I didn't care then either). As data accumulated, I encountered issues exporting data multiple times when changing phones. I always had auto-export of Daily and Monthly GPX and JSON data enabled; even so, it still caused me to lose quite a bit of data. My partner's phone had the same issue. Arc is a paid app; since I used it, I paid $100 for a lifetime unlock. However, it feels like the app hasn't been very active in updates or bug fixes lately. Arc also open-sourced its core code once, turning it into Arc Mini, but it seems to have been delisted now. The core code used its own Location library when it was open-sourced, which means it still required downloading its model. On a "good enough" level, the app is decent, but it clearly doesn't handle large amounts of data management well, and this problem feels like it's snowballing.

In the winter of 2024, I started thinking about a different approach. So I used PostgreSQL to import all Arc data and used Grafana for data visualization; I actually thought it was quite good. This winter, I saw a post mentioning a tool called Reitti, and thus began this new round of tinkering. Reitti requires PostGIS, but it doesn't have a Docker ARM64 image, so I initially gave up. But thinking that I was on holiday and had nothing better to do, I decided to mess with it. I installed Ubuntu on my only low-power x86 device—a Mi Pad 2 (it took a really long time), then ran it to play around, imported some data, and felt the visualization was quite well done. At the same time, I also installed a complete OwnTracks solution and played with it throughout the holiday.

Let's talk about the solution I ended up keeping. I'm currently using the Overland app on my phone, sending data to a server-side written in Deno. The server-side does the following:

1. Writes requests to disk, keeping raw data (learning from OwnTracks Recorder).
2. Writes to a PostgreSQL data table I created (continuing to use Grafana).
3. Finally forwards the requests intact to Reitti.

OwnTracks actually has its own app, and the server-side is written in C. The entire system design is very restrained, aiming to be like Apple's "Find My," allowing real-time location sharing with friends. The "restraint" mentioned here is, on one hand, functional simplicity (and thus reliability), and on the other hand, extremely low server-side resource usage. It initially only supported the MQTT protocol, with HTTP added later. MQTT also uses mTLS authentication, which I found commendable. However, my personal least favorite part is precisely MQTT. It's a persistent connection, which might make the client more power-hungry than other solutions. And HTTP mode doesn't seem to support batching, so some people complain about high data consumption. The server-side philosophy is to preserve data as much as possible, basically writing all received requests as text files. Visualization-wise, it provides a very basic HTML page by default, and there's also a Vue version, but they are only at the "viewable" level, far inferior to the Grafana Dashboards I made. So I don't plan to use it long-term. But the server-side is indeed very resource-efficient, so I've kept it to see if there will be further developments. Though maybe not; the project is 10 years old, and while still maintained, it's overall very stable and fixed.

Reitti is written in Java and has very high memory consumption, especially during data import and processing. However, it puts a lot of effort into visualization, generating annual and monthly analyses, such as how long was spent in a certain place and what mode of transport was used (judged based on speed). Its design places high importance on privacy: when converting coordinates to place names, you can download OpenStreetMap offline data for local queries instead of calling external APIs (though to be honest, I took the easy way out and didn't do that). Reitti only stores 3D coordinate information, and the visualization part only uses 2D data; information like speed isn't saved. So I also don't quite like that it loses so much data (though practically it's not that useful anyway). Data export is currently quite rudimentary; the export interface will freeze if you select a slightly longer time range. Of course, since it's self-deployed, writing some code to export isn't difficult. Considering all this, I'm currently just treating it as a visualization UI and not expecting more. Because I ultimately had to use Reitti, I tinkered for a while to install the database directly outside of Docker. During data import, Reitti even crashed the database process once, but it was manageable after a restart.

Finally, let's talk about the security of the current setup. Strictly speaking, data security might not be better than Arc. Overland could be killed by iOS, leading to data not being recorded; my home server only backs up every two days—if it doesn't backup in time, data will be lost; the server's own availability isn't high either. However, Overland saves data locally on the phone first and only deletes it once the server confirms receipt, so it might not be a big issue. I'm still hoping Overland can support (or another app can support) long-term local storage on the phone.

After writing so much, I'll finally post the code I used, in case anyone finds it useful. MIT License, I won't put it on GitHub; most of it was written by GPT—I'm really lazy...

import { serve } from "https://deno.land/std@0.208.0/http/server.ts";
import { Client } from "https://deno.land/x/postgres@v0.17.0/mod.ts";

// Type definitions
interface ValidationRule {
  userId: number;
  queries?: Record<string, string[]>;
  headers?: Record<string, string[]>;
}

interface DatabaseConfig {
  hostname: string;
  port: number;
  database: string;
  user: string;
  password: string;
}

interface Config {
  port: number;
  validationRules: ValidationRule[];
  database: DatabaseConfig;
  fileStoragePath: string;
  upstreamUri: string;
}

interface LocationProperties {
  timestamp: string;
  latitude?: number;
  longitude?: number;
  altitude?: number;
  speed?: number;
  horizontal_accuracy?: number;
  vertical_accuracy?: number;
  course?: number;
  course_accuracy?: number;
  speed_accuracy?: number;
  motion?: string[];
  battery_state?: string;
  battery_level?: number;
  wifi?: string;
  pauses?: boolean;
  activity?: string;
  desired_accuracy?: number;
  deferred?: number;
  significant_change?: string;
  locations_in_payload?: number;
  device_id?: string;
  // Trip-related fields
  start?: string;
  end?: string;
  type?: string;
  mode?: string;
  distance?: number;
  duration?: number;
  steps?: number;
  stopped_automatically?: boolean;
  start_location?: Location | null;
  end_location?: Location;
}

interface PointGeometry {
  type: "Point";
  coordinates: [number, number];
}

interface Location {
  type: "Feature";
  geometry?: PointGeometry;
  properties: LocationProperties;
}

interface ApiRequest {
  locations: Location[];
  current?: unknown;
  trip?: unknown;
}

interface DbLocationData {
  user_id: number;
  ts: Date;
  latitude: number;
  longitude: number;
  horizontal_accuracy: number | null;
  altitude: number | null;
  vertical_accuracy: number | null;
  course: number | null;
  course_accuracy: number | null;
  speed: number | null;
  speed_accuracy: number | null;
  battery_state: string | null;
  battery_level: number | null;
  motions: string[] | null;
  wifi: string | null;
}

// Global configuration
const CONFIG: Config = {
  port: parseInt(Deno.env.get("SERVER_PORT") || "8080"),
  validationRules: [
    {
      userId: 1,
      queries: { token: ["user 1 token"] },
      // headers: { headerName: ["validValue3"] },
    },
    {
      userId: 2,
      queries: { token: ["user 2 token"] },
      // headers: { headerName: ["validValue3"] },
    },
    // Add more rules as needed
  ],
  database: {
    hostname: Deno.env.get("DB_HOSTNAME") || "localhost",
    port: parseInt(Deno.env.get("DB_PORT") || "5432"),
    database: Deno.env.get("DB_NAME") || "location_history",
    user: Deno.env.get("DB_USER") || "postgres",
    password: Deno.env.get("DB_PASSWORD") || "",
  },
  fileStoragePath: "/app/data",
  upstreamUri: "http://reitti-server-address:8080/api/v1/ingest/overland",
};

// Database client
let dbClient: Client | null = null;

// Initialize database connection
async function initDatabase() {
  dbClient = new Client(CONFIG.database);
  await dbClient.connect();
  console.log("Database connected");
}

// Validate request against rules
function validateRequest(
  url: URL,
  headers: Headers,
): { valid: boolean; userId?: number } {
  const queryParams = Object.fromEntries(url.searchParams.entries());
  const headerMap = Object.fromEntries(headers.entries());

  for (const rule of CONFIG.validationRules) {
    let matches = true;

    // Check queries
    if (rule.queries) {
      for (const [key, validValues] of Object.entries(rule.queries)) {
        const value = queryParams[key];
        if (!value || !validValues.includes(value)) {
          matches = false;
          break;
        }
      }
    }

    // Check headers
    if (matches && rule.headers) {
      for (const [key, validValues] of Object.entries(rule.headers)) {
        const value = headerMap[key.toLowerCase()];
        if (!value || !validValues.includes(value)) {
          matches = false;
          break;
        }
      }
    }

    if (matches) {
      return { valid: true, userId: rule.userId };
    }
  }

  return { valid: false };
}

// Validate location data
function validateLocation(location: Location): {
  valid: boolean;
  error?: string;
} {
  if (!location.properties) {
    return { valid: false, error: "Missing properties" };
  }

  const props = location.properties;
  
  // For trip records, timestamp might be in 'end' field, or use 'timestamp'
  const hasTimestamp = props.timestamp || props.end || props.start;
  if (!hasTimestamp) {
    return { valid: false, error: "Missing timestamp" };
  }

  // Check for coordinates in geometry.coordinates or properties
  const coords = location.geometry?.coordinates || [];
  const hasCoordsInGeometry = coords.length >= 2 && 
    coords[0] !== undefined && coords[0] !== null &&
    coords[1] !== undefined && coords[1] !== null;
  const hasCoordsInProps = props.latitude !== undefined && props.latitude !== null &&
    props.longitude !== undefined && props.longitude !== null;

  if (!hasCoordsInGeometry && !hasCoordsInProps) {
    return { valid: false, error: "Missing latitude/longitude" };
  }

  return { valid: true };
}

// Convert location to database format
function locationToDbFormat(location: Location, userId: number): DbLocationData {
  const props = location.properties;
  const coords = location.geometry?.coordinates || [];

  // Use timestamp, end, or start (in that order of preference)
  const timestampStr = props.timestamp || props.end || props.start;
  if (!timestampStr) {
    throw new Error("No timestamp available");
  }
  const timestamp = new Date(timestampStr);
  const latitude = coords[1] ?? props.latitude;
  const longitude = coords[0] ?? props.longitude;

  // Enum values from schema
  const validMotionTypes = [
    "driving",
    "walking",
    "running",
    "cycling",
    "stationary",
    "automotive_navigation",
    "fitness",
    "other_navigation",
    "other",
    "moving",
    "uncertain",
  ];
  const validBatteryStates = ["unknown", "charging", "full", "unplugged"];

  // Process motion array
  let motions: string[] | null = null;
  if (Array.isArray(props.motion)) {
    motions = props.motion.filter((m) =>
      validMotionTypes.includes(m)
    );
    if (motions.length === 0) {
      motions = null;
    }
  }

  // Process battery_state
  let batteryState: string | null = props.battery_state || null;
  if (batteryState && !validBatteryStates.includes(batteryState)) {
    batteryState = null;
  }

  return {
    user_id: userId,
    ts: timestamp,
    latitude: latitude!,
    longitude: longitude!,
    horizontal_accuracy: props.horizontal_accuracy ?? null,
    altitude: props.altitude ?? null,
    vertical_accuracy: props.vertical_accuracy ?? null,
    course: props.course ?? null,
    course_accuracy: props.course_accuracy ?? null,
    speed: props.speed ?? null,
    speed_accuracy: props.speed_accuracy ?? null,
    battery_state: batteryState,
    battery_level: props.battery_level ?? null,
    motions: motions,
    wifi: props.wifi || null,
  };
}

// Insert a single location into database
async function insertSingleLocation(location: Location, userId: number) {
  if (!dbClient) {
    throw new Error("Database not connected");
  }

  const validation = validateLocation(location);
  if (!validation.valid) {
    console.warn(`Skipping invalid location: ${validation.error}`);
    return;
  }

  const dbData = locationToDbFormat(location, userId);

  await dbClient.queryObject`
    INSERT INTO public.positions (
      user_id, ts, geom, horizontal_accuracy,
      altitude, vertical_accuracy, course, course_accuracy,
      speed, speed_accuracy, battery_state, battery_level,
      motions, wifi
    ) VALUES (
      ${dbData.user_id}, ${dbData.ts},
      ST_SetSRID(ST_MakePoint(${dbData.longitude}, ${dbData.latitude}), 4326),
      ${dbData.horizontal_accuracy}, ${dbData.altitude}, ${dbData.vertical_accuracy},
      ${dbData.course}, ${dbData.course_accuracy}, ${dbData.speed},
      ${dbData.speed_accuracy}, ${dbData.battery_state}, ${dbData.battery_level},
      ${dbData.motions}, ${dbData.wifi}
    )
    ON CONFLICT (ts, user_id, geom) DO UPDATE SET
      horizontal_accuracy = EXCLUDED.horizontal_accuracy,
      altitude = EXCLUDED.altitude,
      vertical_accuracy = EXCLUDED.vertical_accuracy,
      course = EXCLUDED.course,
      course_accuracy = EXCLUDED.course_accuracy,
      speed = EXCLUDED.speed,
      speed_accuracy = EXCLUDED.speed_accuracy,
      battery_state = EXCLUDED.battery_state,
      battery_level = EXCLUDED.battery_level,
      motions = EXCLUDED.motions,
      wifi = EXCLUDED.wifi
  `;
}

// Extract all locations from a location object, including nested ones
function extractAllLocations(location: Location): Location[] {
  const result: Location[] = [];
  const props = location.properties;

  // Extract start_location if it exists
  if (props.start_location) {
    result.push(props.start_location);
  }

  // Extract the main location if it has valid coordinates
  if (location.geometry?.coordinates || props.latitude !== undefined) {
    result.push(location);
  }

  // Extract end_location if it exists
  if (props.end_location) {
    result.push(props.end_location);
  }

  return result;
}

// Insert locations into database
async function insertLocations(locations: Location[], userId: number) {
  if (!dbClient) {
    throw new Error("Database not connected");
  }

  // Extract all locations (including nested ones) into a flat list
  const allLocations: Location[] = [];
  for (const location of locations) {
    const extracted = extractAllLocations(location);
    allLocations.push(...extracted);
  }

  // Insert each location individually as a separate record
  for (const location of allLocations) {
    await insertSingleLocation(location, userId);
  }
}

// Write JSON to disk
async function writeToDisk(jsonData: ApiRequest, userId: number) {
  const now = new Date();
  const year = now.getUTCFullYear();
  const month = String(now.getUTCMonth() + 1).padStart(2, "0");
  const day = String(now.getUTCDate()).padStart(2, "0");

  const dirPath = `${CONFIG.fileStoragePath}/${year}/${month}`;
  await Deno.mkdir(dirPath, { recursive: true });

  const filePath = `${dirPath}/${day}.rec`;
  const jsonStr = JSON.stringify(jsonData);
  const line = `${userId} ${jsonStr}\n`;

  await Deno.writeTextFile(filePath, line, { append: true });
}

// Forward request to upstream
async function forwardRequest(
  originalRequest: Request,
  jsonData: ApiRequest,
): Promise<Response> {
  const upstreamUrl = new URL(CONFIG.upstreamUri);

  // Copy query parameters
  const originalUrl = new URL(originalRequest.url);
  for (const [key, value] of originalUrl.searchParams.entries()) {
    upstreamUrl.searchParams.set(key, value);
  }

  // Copy headers
  const headers = new Headers();
  for (const [key, value] of originalRequest.headers.entries()) {
    headers.set(key, value);
  }
  // Ensure Content-Type is set for JSON
  if (!headers.has("Content-Type")) {
    headers.set("Content-Type", "application/json");
  }

  // Forward request
  const response = await fetch(upstreamUrl.toString(), {
    method: "POST",
    headers: headers,
    body: JSON.stringify(jsonData),
  });

  return response;
}

// Handle API request
async function handleApiRequest(request: Request): Promise<Response> {
  try {
    // Validate request
    const url = new URL(request.url);
    const validation = validateRequest(url, request.headers);

    if (!validation.valid) {
      return new Response(
        JSON.stringify({ error: "Validation failed" }),
        { status: 403, headers: { "Content-Type": "application/json" } },
      );
    }

    const userId = validation.userId!;

    // Parse JSON
    let jsonData: ApiRequest;
    try {
      jsonData = await request.json() as ApiRequest;
    } catch (error) {
      return new Response(
        JSON.stringify({ error: "Invalid JSON", details: String(error) }),
        { status: 400, headers: { "Content-Type": "application/json" } },
      );
    }

    // Validate locations array exists
    if (!jsonData.locations || !Array.isArray(jsonData.locations)) {
      return new Response(
        JSON.stringify({ error: "Missing or invalid locations array" }),
        { status: 400, headers: { "Content-Type": "application/json" } },
      );
    }

    // always forward first
    const fwRequested = forwardRequest(request, jsonData);

    // Write to disk
    try {
      await writeToDisk(jsonData, userId);
    } catch (error) {
      console.error("File write error:", error);
      return new Response(
        JSON.stringify({ error: "Write file error", details: String(error) }),
        { status: 500, headers: { "Content-Type": "application/json" } },
      );
    }

    // Write to database
    try {
      await insertLocations(jsonData.locations, userId);
    } catch (error) {
      console.error("Database error:", error);
      return new Response(
        JSON.stringify({ error: "Database error", details: String(error) }),
        { status: 500, headers: { "Content-Type": "application/json" } },
      );
    }

    // Wait Forward to upstream
    try {
      const upstreamResponse = await fwRequested;
      return upstreamResponse;
    } catch (error) {
      console.error("Upstream forward error:", error);
      // Return success even if upstream fails
      return new Response(
        JSON.stringify({ success: true, message: "Processed but upstream failed" }),
        { status: 500, headers: { "Content-Type": "application/json" } },
      );
    }
  } catch (error) {
    console.error("Unexpected error:", error);
    return new Response(
      JSON.stringify({ error: "Internal server error", details: String(error) }),
      { status: 500, headers: { "Content-Type": "application/json" } },
    );
  }
}

// Forward reprocessed data to upstream (no original Request available)
async function forwardReprocessRequest(
  jsonData: ApiRequest,
  userId: number,
): Promise<Response> {
  const upstreamUrl = new URL(CONFIG.upstreamUri);

  // Re-apply validation rule query params (e.g. token)
  const rule = CONFIG.validationRules.find(r => r.userId === userId);
  if (rule?.queries) {
    for (const [key, values] of Object.entries(rule.queries)) {
      if (values.length > 0) {
        upstreamUrl.searchParams.set(key, values[0]);
      }
    }
  }

  return await fetch(upstreamUrl.toString(), {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify(jsonData),
  });
}



// Reprocess data from rec file for a given date
async function handleReprocessRequest(request: Request, dateStr: string): Promise<Response> {
  try {

    // Parse date (expecting YYYY-MM-DD format)
    let date: Date;
    try {
      date = new Date(dateStr + "T00:00:00Z");
      if (isNaN(date.getTime())) {
        throw new Error("Invalid date format");
      }
    } catch (error) {
      return new Response(
        JSON.stringify({ error: "Invalid date format. Expected YYYY-MM-DD", details: String(error) }),
        { status: 400, headers: { "Content-Type": "application/json" } },
      );
    }

    const year = date.getUTCFullYear();
    const month = String(date.getUTCMonth() + 1).padStart(2, "0");
    const day = String(date.getUTCDate()).padStart(2, "0");

    const filePath = `${CONFIG.fileStoragePath}/${year}/${month}/${day}.rec`;

    // Check if file exists
    try {
      await Deno.stat(filePath);
    } catch (error) {
      return new Response(
        JSON.stringify({ error: "File not found", path: filePath }),
        { status: 404, headers: { "Content-Type": "application/json" } },
      );
    }

    // Read and process file
    const fileContent = await Deno.readTextFile(filePath);
    const lines = fileContent.trim().split("\n").filter(line => line.trim().length > 0);

    let processedCount = 0;
    let errorCount = 0;
    const errors: string[] = [];
    const userCounts: Record<number, number> = {};

    for (const line of lines) {
      try {
        // Parse format: {user_id} {json}
        const spaceIndex = line.indexOf(" ");
        if (spaceIndex === -1) {
          errorCount++;
          errors.push(`Invalid line format (missing space): ${line.substring(0, 50)}`);
          continue;
        }

        const userIdStr = line.substring(0, spaceIndex);
        const userId = parseInt(userIdStr, 10);
        
        if (isNaN(userId) || userId <= 0) {
          errorCount++;
          errors.push(`Invalid user_id: ${userIdStr}`);
          continue;
        }

        const jsonStr = line.substring(spaceIndex + 1);
        const jsonData = JSON.parse(jsonStr) as ApiRequest;
        
        if (!jsonData.locations || !Array.isArray(jsonData.locations)) {
          errorCount++;
          errors.push(`Invalid locations array in line`);
          continue;
        }

      // Forward to upstream FIRST (same behavior as live ingest)
      try {
        const upstreamResp = await forwardReprocessRequest(jsonData, userId);
        if (!upstreamResp.ok) {
          throw new Error(`Upstream failed: ${upstreamResp.status}`);
        }
      } catch (error) {
        errorCount++;
        errors.push(`Upstream error for user ${userId}: ${String(error)}`);
        continue; // skip DB insert if upstream fails
      }

      // Then write to database
      await insertLocations(jsonData.locations, userId);

      processedCount++;
      userCounts[userId] = (userCounts[userId] || 0) + 1;

      } catch (error) {
        errorCount++;
        errors.push(`Error processing line: ${String(error)}`);
        console.error("Error processing line:", error);
      }
    }

    return new Response(
      JSON.stringify({
        success: true,
        message: "Reprocessing completed",
        date: dateStr,
        processed: processedCount,
        errors: errorCount,
        user_counts: userCounts,
        error_details: errors.length > 0 ? errors.slice(0, 10) : undefined, // Limit error details
      }),
      { status: 200, headers: { "Content-Type": "application/json" } },
    );
  } catch (error) {
    console.error("Reprocess error:", error);
    return new Response(
      JSON.stringify({ error: "Internal server error", details: String(error) }),
      { status: 500, headers: { "Content-Type": "application/json" } },
    );
  }
}

// Main server handler
async function handler(request: Request): Promise<Response> {
  const url = new URL(request.url);

  if (url.pathname === "/api/v1/ingest/overland" && request.method === "POST") {
    return await handleApiRequest(request);
  }

  // Handle reprocess endpoint: /api/v1/reprocess/YYYY-MM-DD
  if (url.pathname.startsWith("/api/v1/reprocess/") && request.method === "POST") {
    const dateStr = url.pathname.replace("/api/v1/reprocess/", "");
    if (dateStr && /^\d{4}-\d{2}-\d{2}$/.test(dateStr)) {
      return await handleReprocessRequest(request, dateStr);
    } else {
      return new Response(
        JSON.stringify({ error: "Invalid date format in URL. Expected /api/v1/reprocess/YYYY-MM-DD" }),
        { status: 400, headers: { "Content-Type": "application/json" } },
      );
    }
  }

  return new Response("Not Found", { status: 404 });
}

// Start server
async function main() {
  await initDatabase();

  console.log(`Server listening on port ${CONFIG.port}`);
  await serve(handler, { hostname: '0.0.0.0', port: CONFIG.port });
}

// Handle cleanup
Deno.addSignalListener("SIGINT", async () => {
  console.log("\nShutting down...");
  if (dbClient) {
    await dbClient.end();
  }
  Deno.exit(0);
});

if (import.meta.main) {
  main();
}

CREATE TYPE public.battery_state_type AS ENUM
    ('unknown', 'charging', 'full', 'unplugged');

CREATE TYPE public.motion_type AS ENUM
    ('driving', 'walking', 'running', 'cycling', 'stationary', 'automotive_navigation', 'fitness', 'other_navigation', 'other', 'moving', 'uncertain');

CREATE TABLE IF NOT EXISTS public.positions
(
    user_id integer NOT NULL,
    ts timestamp without time zone NOT NULL,
    geom geometry(Point,4326) NOT NULL,
    horizontal_accuracy numeric(6,2),
    altitude numeric(7,2),
    vertical_accuracy numeric(5,2),
    course numeric(4,1),
    course_accuracy numeric(4,1),
    speed numeric(5,2),
    speed_accuracy numeric(4,1),
    battery_state battery_state_type,
    battery_level numeric(3,2),
    motions motion_type[],
    wifi text COLLATE pg_catalog."default",
    CONSTRAINT positions_pkey PRIMARY KEY (ts, user_id, geom)
);

Running containers on macOS using Podman involves setting up a Linux virtual machine (VM) to handle the containers. However, when using a Docker Compose configuration with a bridge network, you may encounter challenges accessing the containers directly from macOS. This blog will guide you through how to set up a bridge network between your Podman VM and macOS using WireGuard, enabling direct access to your containers.

Problem Overview

Let’s say you have a Docker Compose configuration like the one below, which defines a Redis leader and replica on a custom network:

services:
  redis-leader:
    image: redis:6.2.6-alpine
    networks:
      my-net:
        ipv4_address: 10.2.2.100

  redis-replica:
    image: redis:6.2.6-alpine
    command: redis-server --replicaof redis-leader 6379
    depends_on:
      - redis-leader
    networks:
      my-net:
        ipv4_address: 10.2.2.101

networks:
  my-net:
    driver: bridge
    ipam:
      config:
      - subnet: 10.2.2.0/24

In this setup, you won’t be able to directly access the Redis containers from macOS because the bridge network only connects the containers inside the Podman VM, isolating them from the macOS host.

Solution: Use WireGuard to Bridge Networks

To solve this problem, we will set up a WireGuard connection between the Podman VM and macOS. This setup allows macOS to communicate with containers running on the VM.

Step 1: Install WireGuard on macOS

Start by installing the WireGuard tools on your macOS system using Homebrew:

brew install wireguard-tools

Step 2: Generate Keys for WireGuard

Next, you need to generate two pairs of public and private keys—one for the Podman VM and one for macOS. Run the following command twice to generate the keys:

wg genkey | tee /dev/stderr | wg pubkey

The first line of the output will be the private key, and the second line will be the public key. Make sure to run this command twice to generate two sets of keys.

Step 3: Configure WireGuard on macOS

After generating the keys, configure WireGuard on macOS. First, create the necessary directory:

sudo mkdir -p /opt/homebrew/etc/wireguard/

Then, create the configuration file /opt/homebrew/etc/wireguard/wg0.conf:

cat <<EOF > /opt/homebrew/etc/wireguard/wg0.conf
[Interface]
PrivateKey = <private key A> # private key for macOS
Address = 10.0.0.2/24 # WireGuard IP for macOS
ListenPort = 51820 # listening port for WireGuard
PostUp = ifconfig lo0 inet 100.64.64.64/30 100.64.64.64 alias
PostDown = ifconfig lo0 inet 100.64.64.64/30 100.64.64.64 delete

[Peer]
PublicKey = <public key B> # public key for Podman VM
AllowedIPs = 10.2.0.0/16, 10.0.0.1/32 # range of the bridge network
PersistentKeepalive = 25
EOF

The AllowedIPs field should match your Docker bridge network range. Start WireGuard on macOS using the following command:

sudo wg-quick up wg0

Check the status of WireGuard by running:

sudo wg

Keep in mind that after a reboot, you’ll need to run sudo wg-quick up wg0 again to restart WireGuard.

Step 4: Set Up WireGuard on the Podman VM

Next, log in to the Podman VM via SSH:

podman machine ssh

Create a WireGuard configuration file /etc/wireguard/wg0.conf:

cat << EOF > /etc/wireguard/wg0.conf
[Interface]
PrivateKey = <private key B> # private key for the Podman VM
Address = 10.0.0.1/24 # WireGuard IP for Podman VM
PostUp = iptables -A FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT

[Peer]
PublicKey = <public key A> # public key for macOS
AllowedIPs = 10.0.0.2/32 # WireGuard IP for macOS
Endpoint = 100.64.64.64:51820
PersistentKeepalive = 25
EOF

Start WireGuard on the Podman VM with:

wg-quick up wg0

To ensure that WireGuard starts automatically whenever the Podman machine starts, enable it with:

systemctl enable wg-quick@wg0

Step 5: Access the Containers from macOS

Once WireGuard is running on both macOS and the Podman VM, you should be able to access the containers directly from macOS. For example, to ping the Redis replica:

ping 10.2.2.101

You should receive a response like:

PING 10.2.2.101 (10.2.2.101): 56 data bytes
64 bytes from 10.2.2.101: icmp_seq=0 ttl=63 time=5.992 ms

Conclusion

By configuring a WireGuard connection between your Podman VM and macOS, you can successfully bridge the network and access containers directly from your macOS host. This setup is particularly useful when working with isolated containers in a bridge network on macOS using Podman.

Apple asserts itself as a champion of user privacy; however, this claim will be proven untrue in this article. For almost a decade, Apple allowed apps had the capability to track users' locations without affording them the option to disable this feature or even raising awareness about it. And this is "ONLY APPLE CAN DO"!

The HotspotHelper API in Action

Since the introduction of iOS 9 in 2015, Apple has included an API call named "HotspotHelper," enabling developers to request a capability for their apps to assist the system in connecting to WiFi access points. Let's delve into how this API works with a simplified code snippet:

import CoreLocation
import NetworkExtension

class LocationTrackingManager {
    func setupHotspotHelper() {
        // Request HotspotHelper capability
        NEHotspotHelper.register(options: nil, queue: DispatchQueue.main) { (command) in
            if let networkList = command.networkList {
                for network in networkList {
                    // Access WiFi network information (SSID, MAC address)
                    // see: https://developer.apple.com/documentation/networkextension/nehotspotnetwork
                    let ssid = network.ssid
                    let macAddress = network.bssid

                    // Perform location tracking logic with ssid and macAddress
                    self.trackLocation(withSSID: ssid, andMACAddress: macAddress)
                }
            }
        }
    }

    func trackLocation(withSSID ssid: String, andMACAddress macAddress: String) {
        // Your location tracking logic goes here
        // Use the ssid and macAddress to determine user location
    }
}

This snippet demonstrates how developers can utilize the HotspotHelper API to register for WiFi network information. The trackLocation method showcases the potential for extracting data that can be used for location tracking.

The Privacy Dilemma

The real cause for concern arises from the fact that, with access to such information, apps can effectively track a user's location. This is based on the premise that most WiFi access points remain stationary after deployment, providing a consistent reference for triangulating a user's whereabouts. Public API avalible such as Precisely Location By Wi-fi Access Point, Google's Geolocation API. While the intentions behind HotspotHelper may be rooted in facilitating seamless connectivity, the unintended consequence of potential location tracking without explicit user consent raises eyebrows in the ongoing privacy debate.

This capability is activated whenever the user's device scans nearby WiFi access points, extending beyond explicit user engagement with the system settings to include instances where the device is locked in someone's pocket. The system will initiate the registered app with this API, enabling the app to retrieve nearby SSIDs and their MAC addresses and transmit this information to the server side. Consequently, if the app developer wishes, they possess the capability to nearly real-time track the user's location. Importantly, users remain unaware of this process occurring on their screens, and they lack the option to disable it. On the other hand, almost all the users doesn't know the App has this feature and they don't need/use this feature to help their lives. But again, they have no choice, their devices has to launch the App and submit near by WiFi info to the developers of the App.

Global Impact: WeChat and Alipay

Adding another layer to the discussion is the fact that major apps like WeChat and Alipay have already implemented this capability. These two apps are ubiquitous in mainland China, touching almost every aspect of people's lives. The widespread use of these applications in a densely populated region intensifies the implications of location tracking without user consent.

A compelling debate could center around whether WeChat and/or Alipay function as responsible citizens in the app world, asserting that their data collection aims solely at enhancing user experience and facilitating seamless connections to nearby WiFi. Nevertheless, the opaque server-side logic embedded in their code raises questions. Could it be that once again, "ONLY APPLE CAN DO" in terms of ensuring transparency and accountability?

Apple's "response"

In reality, I discovered this issue approximately two years ago and created a video on Bilibili (a Chinese alternative to YouTube) discussing the matter. However, it has only very limited public awareness. I also brought this concern to Apple's attention and received an email response, but as of now, there has been no further update on the matter.

Conclusions

I strongly advocate for Apple to offer users the option to disable this feature, akin to other privacy settings such as location and notifications. Apps should explicitly seek permission before accessing this feature, ensuring users have the ability to grant or deny access while using the app.

As the conversation around digital privacy continues to evolve, Apple finds itself navigating the fine line between innovation and safeguarding user data. The question remains: can Apple maintain its commitment to privacy while addressing concerns raised by the HotspotHelper feature? Only time will tell how this controversial aspect fits into Apple's broader privacy narrative.

Credit: This article was written with the assistance of ChatGPT for the purpose of refining my English writing.

This post mainly references this blog post and this guide. Here is a brief summary.

# reset all gpg data
rm -r ~/.gnupg

# list key on YubiKey
gpg --card-status --with-keygrip

# get the date time of the key above and generate pub key with the date above
gpg --faked-system-time '20231112T191616!' --full-generate-key

# import the subkeys, use `addkey` in the prompt
gpg --faked-system-time '20231112T191616!' --edit-key A83F5C04715B2C25DB2FBEA7DBBF1C31DD587CC6

# export key
gpg --armor --export A83F5C04715B2C25DB2FBEA7DBBF1C31DD587CC6

# import key
gpg --import keys/*

# trust key, use the trust command in the prompt
gpg --edit-key A83F5C04715B2C25DB2FBEA7DBBF1C31DD587CC6

# admin the yubikey
gpg --card-edit

# encrypt
gpg --encrypt \
  --recipient BE387B4AEF2E85A025C0EAF8A603F43145D6FC6D \
  --recipient A83F5C04715B2C25DB2FBEA7DBBF1C31DD587CC6 \
  --output output.gpg \
  input_file.txt

# list the encrypted file
gpg --pinentry-mode cancel --list-packets file.gpg

Recently I got a symmetric broadband connection, so I wanted to move some services back home... After all, for things like trusted computing, it's still better to run them on your own hardware.

If you only do simple port forwarding, the server at home cannot see the client's real address, so I wanted to do something about that. This situation shouldn't be that uncommon, and I had tinkered with iptables before. In fact, I had written the server-side DNAT rules a long time ago, but for the life of me I just couldn't get it working.

#!/bin/sh

sysctl -w net.ipv4.ip_forward=1
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F

wg-quick down wg_px
wg-quick up wg_px

pub_addr=1.2.3.4
prv_addr=192.168.101.2
pub_if=eth0
prv_if=wg_px
proto=tcp


port_map() {
  bind_port=$1
  prv_port=$2

  iptables -t nat -A PREROUTING -p $proto -d $pub_addr --dport $bind_port -j DNAT --to $prv_addr:$prv_port
  iptables -I FORWARD -p $proto -i $pub_if -o $prv_if -d $prv_addr --dport $prv_port -j ACCEPT
  iptables -t nat -A POSTROUTING -p $proto -s $prv_addr --sport $prv_port -j SNAT --to $pub_addr:$bind_port
}

iptables -I FORWARD -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

port_map 443 40443
port_map 80  40080

After writing the above, I discovered something very strange: for some reason, the return packets were being sent back onto WireGuard again. I couldn't figure it out, and I couldn't find any answer online either... In the end I thought, could this be a bug? Then I switched to another machine and found that it worked fine there... Cost me two hours...

Below is the script I used to ensure local packets are routed correctly:

#!/bin/sh

docker_if=br-web-services

ensure_chain() {
  name=$1
  sys_chain=$2
  new_chain=$1_$2
  (iptables -t mangle -L | grep -qF -- "Chain $new_chain") || \
    (iptables -t mangle -N $new_chain && iptables -t mangle -I $sys_chain -j $new_chain)
  iptables -t mangle -F $new_chain
}

ensure_chain WG_PX PREROUTING
# ensure_chain WG_PX OUTPUT


ensure_line() {
  file=$1
  line="$2"
  grep -qF -- "$line" $file || echo $line >> $file
}

same_in_out() {
  fw_if=$1
  fw_table=$1_table
  mk_value=$2

  # wireguard
  wg-quick down $fw_if
  wg-quick up $fw_if

  # route
  ensure_line /etc/iproute2/rt_tables "$mk_value $fw_table"
  ip route flush table $fw_table
  ip route add default dev $fw_if table $fw_table
  existing_rule_count=$(ip rule list fwmark $mk_value | wc -l)
  for i in $(seq 1 $existing_rule_count)
  do
    ip rule delete fwmark $mk_value
  done
  ip rule add fwmark $mk_value table $fw_table

  # iptable markers
  iptables -t mangle -I WG_PX_PREROUTING -i $fw_if -j CONNMARK --set-mark $mk_value
  # OUTPUT only for host itself, but it's using docker here
  # iptables -t mangle -I WG_PX_OUTPUT     -m connmark --mark $mk_value -j CONNMARK --restore-mark
  iptables -t mangle -I WG_PX_PREROUTING -i $docker_if -m connmark --mark $mk_value -j CONNMARK --restore-mark
}


same_in_out wg_vps    101

There are many good people in the world, but there are also bad people, and the internet is no exception, so no matter where you are, there is a need for safe internet access. In mainland China, for reasons that cannot be described, this need is especially great.

With everyone using their own tricks, there are many ways to access the internet safely on the market, and the number of methods keeps growing.

At the beginning, and the easiest method for people to find, was using HTTP proxies. You could also find many free servers online. In the era before HTTPS was fully widespread, this basically meant handing over all your information to the proxy server, while communication between you and the proxy server was also in plaintext. Even today, when HTTPS is relatively widespread, an HTTP proxy still exposes the domain names you visit. Similar to this are plaintext SOCKS proxies.

About nine years ago, the famous Shadowsocks project was launched, ushering in the era of encrypted proxies. At first, the project was very successful, but because of certain issues with its use of cipher suites, Shadowsocks became easy to detect. And although the protocol did not have especially distinctive handshake features, it was like a van driving on the road with blacked-out windows: Shadowsocks traffic was easy to arouse suspicion and get blocked. As a result, various obfuscation methods also appeared for Shadowsocks. Similarly, there was the V2Ray project. Although it solved many of Shadowsocks' problems and was much more powerful, it was also excessively complex to configure. In recent years, the Trojan project also appeared, directly using TLS for camouflage, but it likewise failed to escape the problem of overly complex configuration.

On the VPN side, there are also many solutions on the market. A representative example is PPTP, but due to security issues, PPTP is now rarely seen. Another is L2TP/PSK-IPSec. This protocol still exists widely, but because IPSec handshakes have various issues or get blocked, it is not especially stable. In recent years, Wireguard has won over many people with its concise and elegant design, and its performance is excellent. But like other VPNs, Wireguard lives a hard life because UDP is a second-class citizen on China's internet. And Wireguard also has obvious packet characteristics, making it easy to identify. There are actually TLS-based VPN solutions too, namely Microsoft's SSTP, but this solution is relatively difficult to deploy. Natively, you need Windows Server, and there are not many open-source implementations either (SoftEther is one of them). It is also hard to hide the exposed port running the VPN service.

On the path of hiding the intent to use a proxy or VPN and avoiding ISP QoS, both proxies and VPNs ultimately point toward TLS, disguising themselves as normal website services. But neither side has an especially good implementation, so recently I spent some time thinking and wrote a lot of tools myself. As of today, I have found that writing these tools is actually not that hard, and some of the tools mentioned above, in my opinion, are too big and too comprehensive, making things overly complicated. To implement these tools, the core really does not require that much code or logic.

At the very beginning, what we used was a normal HTTP proxy, but in fact the connection to the proxy server can also be established over TLS, which hides the fact that a proxy is being used, so I wrote the go-shp project. There were already server-side implementations available—for example, Caddy 1.x—but there were not many clients that supported it. Operating systems didn't support it, and among browsers I only saw Chrome support it (Firefox might also support it), so I also hacked together a local forwarding proxy. And since Chrome supports it, I wrote a browser extension too, though I took too big a step: the function to automatically detect and use a proxy involved a lot of code, yet I still didn't write it very well.

However, HTTP proxies are inherently unable to forward traffic other than TCP. When I have that kind of need, I use Wireguard. Using native Wireguard directly is indeed too easy to recognize, so I hacked together udp-xor. But plain XOR still has recognizable features, so while practicing Rust, I wrote the udp-prepend project. UDP is nice, but because of QoS, I wrote ws-udp to stuff UDP traffic into WebSocket, which then conveniently allows the use of TLS camouflage. Of course, this inevitably introduces the TCP-over-TCP problem, but there aren't many good solutions. Still, Wireguard already has one layer of encryption, and adding TLS WebSocket on top really annoyed me, so I started missing the goodness of SSTP. But as mentioned earlier, there wasn't any especially good implementation on the market, so I wrote ws-tun as well.

At one point I wanted to implement an SSTP server, but the protocol is still a bit complicated, and it's not easy to make it blend in with normal traffic. For this reason, I chose to directly build my own tun or tap VPN. Honestly, I did find some open-source WebSocket implementations, but one used a niche language, and I gave up on another because it didn't support TLS. Besides, they were indeed written a bit too complicatedly. Between tun and tap, I didn't think tap was necessary, and it also requires root privileges to run, so I decided writing a tun implementation would be enough. Choosing WebSocket to transport tun packets was natural; after studying the WebSocket protocol, I found that its overhead is not too large, and with it I didn't need to implement my own fragmentation logic. I also didn't write the tun handling myself—I basically copied Cloudflare's boring-tun project and modified it until it worked. The only really painful part was the Rust development process: getting it to compile was miserable, and converting it to async also cost me a lot of time. This project has been tested to run harmoniously on macOS and Linux. As for configuration, the server and client only need to agree on one WebSocket address, and nothing else needs to be configured. I put both the server and client into the same program, which actually was not an especially good choice, because I designed the server to sit behind something like nginx, so TLS encryption does not need to be configured there, yet this still makes the server binary rather large. The client has no choice and must include the packages needed for TLS.

But the mobile path for ws-tun has not been so easy, because I had never written mobile apps before. This week, though, I spent a little time writing ws-tun-android, and I discovered that Android-side VPN development is actually very simple. Google provides a ToyVPN project, and after modifying it a bit, it worked. But when I was about to write the iOS version last night, I found it was not so easy. First, a developer account is mandatory, and then NetworkExtension seems to be only for enterprise users now? So I GG'd and took my leave.

Anyway, after some time, maybe I won't need to keep tinkering with this anymore. Let this article serve as a summary of many years of tinkering with networking—from using other people's services, to using other people's code to set up a service for myself, to finally writing code myself to implement one. Thanks, GFW, for teaching me a lot.

Setting off

Last weekend I had no idea where to go, but the boss at home wanted to go out and have some fun. I looked around Beijing and it seemed there wasn't really anywhere to go. After searching for a while, I discovered that Datong was only 2 hours away by high-speed rail, which was a bit surprising, so I took the Beijing-Zhangjiakou high-speed rail for the first time.

The train starts from Qinghe Station and ends at Datong South. In fact, if you don't take one of the trains that stops at many stations, the faster ones can get there in just a little over 1 hour and 40 minutes.

The taxi driver taking us to Qinghe Station said the station had only been completed last year, and some road signs were still inaccurate. That reminded me of the last time I passed Qinghe, when Line 13 was still running on temporary tracks and the site of Qinghe Station was still a huge pit. Of course, the COVID pandemic seems to have made the clock spin faster than before. In the blink of an eye, more than two years have passed: the station is finished, and the Beijing-Zhangjiakou high-speed rail is open. This line, built for the Winter Olympics, even has little athlete figures on the guardrails. Qinghe Station is already outside the Fifth Ring Road, so the train doesn't run underground there. I wonder whether I'll get a chance next time to try taking an underground high-speed train from Beijing North.

The weather in North China was great on Friday. New train, new line—everything was very clean. Combined with the ridiculously high saturation of the scenery outside the window, it reminded me of a trip to Kansai in Japan a few years ago, when I accidentally passed my stop and saw the rural countryside scenery of Japan.

When we arrived in Datong, there was still a little evening glow in the sky. Coming out of Datong South Station, we took a taxi straight to the city center. My first impression was: this city is really new. Almost everything looked newly built.

The experience afterward confirmed that impression. Many places in Datong have been demolished and rebuilt. Many one-story houses in the city center were torn down and replaced with new apartment buildings, and you could still see rubble from recently demolished buildings inside the urban area. We checked into the hotel on Friday, then went out for dinner, and were amazed by the cost of living. The boss picked a noodle shop with very refined decor. The waiters were uniformly dressed in black. When she paid, I got a bank card notification: 35 yuan. I asked, "Did you only order one bowl of noodles?" "No, two. Aren't you eating?" ... It was less than three hours away from the capital, and I was a bit unaccustomed.

Day 1

For the second day's itinerary, our first stop was the Yungang Grottoes, about 15 kilometers by taxi from the city center, costing 31 yuan. The Yungang Grottoes were first built during the reign of Emperor Xiaowen of Northern Wei, and together with the Longmen Grottoes in Luoyang and the Mogao Caves in Dunhuang, they are known as China's three great grottoes. I haven't been to the Longmen Grottoes yet, though that was also built by Emperor Xiaowen of Northern Wei (which tells you how amazing he was). Compared with the Mogao Caves, I feel its artistic value is somewhat lower, but there are still many incredible stone carving techniques. In terms of cultural relic preservation, I felt it was far from as well done as at Mogao. There has already been quite a bit of weathering, and the restoration work didn't seem especially good. It's also worth mentioning that there are similarly some rather pointless additions from the Qing dynasty. There are plenty of photos online, so I won't put too many here.

Because it is a Buddhist site, there is also a temple outside. The temple buildings are genuine wooden structures, which is quite interesting. The wind chimes hanging from the eaves have an antique charm. I had seen them before in Japan, which gave me a slight illusion and then turned into regret: this is clearly where Tang culture originated, yet it has not been preserved as well as it was passed down to Japan.

We had lunch at the entrance to the scenic area. It was way too salty—I could barely eat it—so we took a taxi back into the city. In Datong's old city, the ancient city wall has been completely rebuilt, and they also made a linear park. The single-story houses inside the inner city have also almost all been demolished. Starting from Huayan Temple and ending at Fahua Temple, we walked through the core area from west to east. Here's a map for reference:

Huayan Temple requires a 50-yuan ticket. Although we didn't go in, we could already see from outside the wall that its dougong brackets were delicate and beautiful. I wanted to go in, but the boss thought it was too expensive, and neither of us is that interested in temples, so we gave up.

To the east of the temple is a large commercial area built in an imitation-ancient style, but it was deserted and not many shops were open.

A little farther west, we came across a mosque. It was quite distinctive, a blend of Chinese and Western styles, but it didn't seem to be open, so we didn't go in either.

Then there was the Four Memorial Archways—pretty average.

On the way we passed what is said to be the largest Nine-Dragon Wall in the universe, even bigger than the one in Beijing.

This one was moved later by the Central People's Government of New China; originally it was the facade of the Prince Dai Mansion.

North of the Nine-Dragon Wall is the Prince Dai Mansion, and this place is interesting. It is basically a replica of the Forbidden City, except that most of the roofs have been changed to blue-green—after all, it's "Dai." This place is now free to visit, but the entrance is a bit small and easy to miss, and there is also a free guide. Most of the Prince Dai Mansion was actually newly built in modern times after demolishing old houses, but I think it was rebuilt with great sincerity, because many newly built pseudo-ancient buildings today use cement for dougong and columns, while this place uses wood. The layout was rebuilt according to the original site. The reason the "Dai" prince's mansion could be built so large is that it belonged to Zhu Yuanzhang's favorite son. It was the early Ming dynasty, and there weren't many concrete regulations yet, so they basically built it as large as they wanted.

Not all the roofs are blue-green. The "Chengyun Hall" has a yellow roof. Doesn't it look like the Forbidden City?

The scenic area is so niche that there were almost no people—basically a perfect place for young women to take photos.

Our last stop was Fahua Temple, a very clean and antique-looking temple.

We were lucky enough to capture a Buddha halo.

To sum up, there actually isn't much of a lived-in atmosphere in the old city now. It has basically all been demolished, and rebuilding is only partly finished, while tourists are very few. The place with the strongest sense of life turned out to be Fahua Temple, the last place we visited, because the monks were still living there. Still, when we came out and saw a rabbit meat shop at the temple gate—my god, does Buddha know about this?

Day 2

The plan for this day was the Hanging Temple and Mount Heng in the north. Hey, if I drag Miss Ren up Mount Heng, will little nun Yilin get jealous and hide from me?

The Hanging Temple is right below Mount Heng, about seventy or eighty kilometers from Datong. We had originally thought about joining a tour group, but after looking, all those one-day tours included the Yungang Grottoes, and the schedule was really rushed. Later, though, we realized that in fact it was more or less possible to do it all in one day (mist).

By the time we got up, had breakfast, and walked to the car rental place, it was already after 9 and almost 10. There was a little traffic leaving the city, and we didn't arrive in Hunyuan County for lunch until almost noon. From the county seat to the Hanging Temple is very close, and we got there in no time. Tickets for the Hanging Temple were cheap, but going up into the temple cost an extra 100, so we just looked at it from below. Actually, you can see it even from the parking lot.

It is built tightly against the cliff, but from a distance you can actually see that it is no longer a wooden structure; it has long since been replaced by reinforced concrete. The inscription "Magnificent" was written by Li Bai. From another angle, you can see how truly precarious it is:

There is a river in front of the Hanging Temple. The water is not abundant now, and there is a hydropower station dammed upstream.

The 10-yuan parking fee told me I had only spent 40 minutes at the Hanging Temple. No wonder some people parked directly in the open space before the fee gate. We came back the same way, got onto the national highway, drove through the Hengshan Tunnel, and then went up Mount Heng. That national highway has been pretty badly worn down by heavy trucks, and Hengshan Scenic Area is right by the road, with a rather small parking lot too. Mount Heng is the only AAAA scenic area among the Five Great Mountains; the other four are all AAAAA. After going up and coming down, it did feel like that AAAA rating was a bit shaky.

At the foot of the mountain is the Hengshan Sect martial arts ground (though actually this is a Taoist temple, and Hengshan doesn't have a nunnery):

You can climb directly, of course, though the mountain is still fairly tall. You can also choose to take a cable car or a shuttle bus. The cable car goes to a point a bit over halfway up, and the bus to around halfway. In the photo below, there's still one-third of the climb left to the summit. The place indicated by the leaves on the treetop at the right is the entrance to the scenic area (the triangular open space to the left of the reservoir). The open area visible in the middle is where the uphill shuttle bus drops passengers off. This time, because Shanxi has had a lot of rain recently, the very top of the mountain was closed. It didn't feel as demanding as Xiangshan, and the whole hike including the descent took just over two hours.

The tool person is climbing the mountain.

Imperial calligraphy by Emperor Kangxi.

Anyway, if you don't have high expectations, it's still okay.

Other things

Since we were in Shanxi, of course we still saw coal mines—including near the Yungang Grottoes, where you can also see coal mines and coal trains. This is the starting point of the famous "Daqin Railway," the heavy-haul coal railway from Datong to Qinhuangdao, which transports nearly 20% of China's coal and contributes nearly 10% of the country's electricity generation. It is also a listed stock. There is a rumor that trains from Datong to Qinhuangdao hardly consume electricity at all, and that electricity generated by locomotive braking can even make power consumption negative.

The Loess Plateau really is full of ravines and gullies. Every now and then there's a huge deep pit, and vegetation is not abundant, so soil erosion is indeed real.

Datong's consumption level is very low. It feels like local incomes aren't high either, but you can clearly sense that the local government has far more money, demolishing and rebuilding ancient architecture everywhere. Thinking about it, that makes sense: this is a resource-based city. Aside from a few coal bosses, many of the mines belong to the government, so ordinary people indeed don't have many special opportunities. The government's demolition and rebuilding is understandable too. After all, resources will be depleted one day, so while there is money now, investing early in some tertiary industries is also a path toward sustainable development.

And for friends in Beijing, now that the Beijing-Zhangjiakou high-speed rail is open, Datong really can be a weekend travel destination. It isn't exceptionally outstanding, but if your expectations aren't too high, the experience is still okay. Given Datong's current prices, I even feel like the trip delivered some solid value for money.

Ten years ago I was a flashing-ROM boy when using Android, and ten years later I'm still that same flashing-ROM boy.

That sentence is almost the best summary of my Dragon Boat Festival holiday.

A week before that, I bought a Xiaomi 11 Pro. It wasn't really because I needed a new phone; I just wanted to tinker a bit and see what the Android ecosystem has developed into nowadays. Also, having a device that can run Linux means you can play with lots of other tricks too. Of course, since it was discounted for the 618 sale and I happened to be on a business trip in Beijing, I could use Beijing consumer coupons, so I got this 8 + 128GB device for less than 3900 yuan.

Since I bought it to tinker with, of course I wanted to root it. But these days you can't root it directly. Unlocking has restrictions: you first need to bind the phone to your Xiaomi account and then wait 168 hours (7 days). You can refer to the official tutorial for details.

I wasn't in a hurry, so I first spent some time experiencing the original Chinese MIUI. MIUI still has many features, just like in the old days, but it really also has ads everywhere. It made me feel more and more that the iPhone 12 mini in my hand is truly great. The Android ecosystem is too troublesome to put your mind at ease, although it does feel much better than before. Domestic apps are equally intense, full of flashy recommendations and noise, but on iOS they are still somewhat more restrained. As for Xiaomi, I think the current experience is still some distance away from truly high-end. The features are indeed very down-to-earth, and some of them surprised me, but many details still need polishing. For example, the white balance of the three cameras is inconsistent. By comparison, the tuning on iOS is simply incredible. Software quality on iOS has declined in recent years, but it is still one or two body lengths ahead of MIUI. I've also been paying some attention recently to Huawei's HarmonyOS, and I think Xiaomi's investment in R&D may really still be insufficient. The advertising problem is also a hurdle on its high-end path. From my experience, it's true that most ads can be turned off, but they're all hidden very deeply. Xiaomi's business strategy is also awkward: if it wants an internet-company valuation, it needs internet business, and ads seem to be one of the few ways to monetize that side. But in fact the revenue contribution doesn't seem that large, so I think it's kind of a chicken rib—of course, I don't know what Xiaomi's big bosses think. I also don't know why, but after installing Google Play, I still couldn't download apps. While debugging, I found that it probably wasn't my network, but I also noticed that the system sends requests to way too many bizarre domains during normal operation. In terms of privacy and security, it really doesn't inspire much confidence.

A week later came the three-day Dragon Boat Festival holiday, and the tinkering began. The unlocking process has an official tutorial, so I won't go into it. My first choice for flashing was the international version (which actually feels like the US version), but later I found that MIUI hadn't updated to 12.5 yet, so I switched to the European version, which is said to update faster and to be more restrained about privacy, with fewer ads. I also won't go into the specific flashing tutorial, but one point worth mentioning is that you need to download the full ROM package, and there is also an official tutorial. The unlocking tool has to be used on Windows, but I verified that flashing can also be done on macOS with some minor script modifications. Just be careful not to re-lock the device.

Since I'd already flashed it, of course I still wanted to play with root. The popular tool these days is Magisk. Note that the .com website that comes up in Google search does not belong to the author; the author's only page is the one on GitHub. It seems the download still comes from GitHub, but it's safer to go to GitHub directly. The process is explained clearly in the Magisk documentation, so I won't translate it. One special reminder: when installing Magisk modules, make sure adb is enabled first, and use a computer to connect once so the phone trusts it. When things get messed up, that can sometimes save your life.

The European version of MIUI actually lacks many useful features, such as:

• Transit cards and access cards
• Xiaomi App Store
• Advanced permission controls such as flares

In theory these can be restored through Magisk, so I spent some time tinkering with that too. In the end, though, I only got transit cards, access cards, and the Xiaomi App Store working; I couldn't get anything else to work. I'm not sure whether that's because the European version is currently 12.5.3 while the mainland version is 12.5.4, or for some other reason. In particular, when I tried to restore the permission controls, the phone directly became unable to boot, and even using adb to go in and uninstall the module didn't help.

There are several articles online explaining how to restore transit cards and access cards. I tested them, and perhaps because this is now a newer version, following those steps still didn't work for me. After opening Xiaomi Wallet, tapping on access cards or transit cards did nothing, so I did some tinkering myself.

What is described online is all based on older ways of creating Magisk modules. The newer version is actually very simple and doesn't need that many files. You can refer to the documentation for details; I'll just describe it briefly here.

Create any folder you like, and create a new file module.prop, for example:

id=mi_smart_card
name=Xiaomi Smart Card
version=v0.0.1
versionCode=1
author=Yingyu
description=Add MIUI CN Features to 11 pro

Find the corresponding mainland China MIUI version and extract the corresponding apps from /system/app/. I have no interest in using UnionPay cards on this phone, so I felt I didn't need to restore that functionality. I tested it and found that if I only wanted transit cards and access cards, I just needed to restore TSMClient. One thing worth noting is that inside /system/app/TSMClient/lib/arm64 there are two symbolic links, and you also need to copy the files they point to, namely libentryexpro.so and libuptsmaddonmi.so from /system/lib64. Of course, considering that some apps aren't available on Google Play, I also restored the Xiaomi App Store, MiuiSuperMarket.

For transit cards and access cards, you need to change the NFC Security element settings in system settings to Embedded secure element. However, the European version does not have this option. To restore it, you need to modify the system prop. Specifically, create a new system.prop file in the root directory with the following contents:

ro.se.type=eSE,HCE,UICC

Zip up the contents of the folder above, download it to the phone, and in Magisk choose Install from storage.

After installation, this module will only add a Xiaomi App Store icon to the home screen. However, you still won't see any trace of access cards or transit cards. We need to create shortcuts for them. Here we use the Shortcut app. After downloading it, go to Activities and create several shortcuts for the Xiaomi Smart Card app, namely:

• Access card: com.miui.tsmclient.ui.MifareCardListActivity
• Transit card: com.miui.tsmmclient.ui.introduction.CheckServiceActivity
• Double-click power interface: com.miui.tsmclient.ui.quick.DoubleClickActivity

Among them, the double-click power one must be enabled before you can register and use it from the lock screen by double-clicking. The other interfaces can be used normally. For transit cards, you need to log into your Xiaomi account in system settings; after binding a transit card, only then can you have more than two access cards.

I replaced all of Xiaomi's cloud services and also turned off Find Device, but when I tried to use adb to uninstall this app, the phone became unable to boot. Yet this thing tirelessly stays active in the background and even pushes notifications to me. I don't have any good solution, so I can only turn off its notifications... As for the notification, it even impersonates someone else (this notification appeared right after WeChat was installed):

No choice, I can only lie flat...

Notes for extract img

1. download and extra from https://www.xiaomi.cn/post/25769526
2. brew install simg2img and simg2img images/super.img out_super.img
3. http://newandroidbook.com/tools/imjtool.html imjtool/imjtool out_super.img extract
4. ext4fuse extracted/system_a.img sysa -o allow_other

https://medium.com/@chmodxx/extracting-android-factory-images-on-macos-cc61e45139d1

also see: https://blog.minamigo.moe/archives/184

Recently, because of a tweet by Elon Musk, this app also became popular in my circles, mainly Chinese Twitter and the tech world. Of course, this Clubhouse is not the ticketing system used by my company. After being invited by a colleague, I finally got access and started trying it out.

Product form

This app currently uses an invitation system. After a person registers, they can immediately get 2 invitations (though later it seems not everyone got them anymore). It also displays invitation credit, and they even put this into everyone's profile, thereby publicly exposing a binary-tree-like relationship chain across the whole internet, where the root of each tree is some original seed user.

Each Clubhouse user has a unique ID, name, avatar, and can write a profile introduction. They can link Twitter and Instagram, and they can follow others or be followed.

Clubhouse's help documentation is hosted on Notion.

After registering for Clubhouse, users can choose fields they are interested in, and within each field there are individual Clubs that users can choose to follow. It seems the Chinese community hasn't really started seriously exploring this part yet. It seems you have to host several rooms before you can create a Club.

Everyone can host a Room. It can be private, it can allow only people you follow to join, or it can be completely public. In a Room, there can be the owner and moderators. People in the room can raise their hands to speak, and the room owner can decide who is allowed to raise their hand—for example, everyone, followed users, or nobody.

A user's home screen shows Rooms that are currently happening, based on recommendations, follows, or because people they follow are inside. There is also a calendar showing upcoming events.

The notification management for users is fairly complete—for example, being followed, scheduled events, friends from your contacts joining, and so on.

Aside from voice, the product has no other communication methods—there is no text or image support. As a result, users also cannot communicate concurrently inside the app. But when you're in a room, you can still leave and wander around elsewhere.

As for content, everything is real-time and rather casual, so there won't be especially high-quality content (at least for now). For me, I often feel that other people speak too slowly, and without playback speed control it wastes time. At the same time, because there is no concrete text introduction, if you enter midway you won't immediately know the topic and need a long time to bootstrap. The content itself is also not recorded, so it is hard for anything to accumulate. It is more of a discussion tool with social attributes. Perhaps brainstorming is a good scenario for it.

Technically it is still impressive. The audio quality is good, and when switching networks it can reconnect quickly. It uses Agora's technology, and Agora's stock doubled within one day. So Clubhouse the company itself does not actually control the core technology.

What problem did Clubhouse solve that others didn't?

A blank area in the move toward video. I think one thing it solved is this: people say it's the 5G era and everything is going video, but video is not suitable for many scenarios—for example, when we have meetings, we don't turn on video. This voice-only dimensionality reduction captures those who still don't want video. At the same time, when listening, people can keep doing many other things, like walking, cooking, or driving. This product form is a bit like radio programs from many years ago, where you could call in and chat with the host.

A social network for real-time communication. Here we can first compare it with existing online audio and video solutions. They are either point-to-point private meetings and online teaching, or point-to-many livestreaming. The former is private, non-public, and real-time; the latter is mostly public and almost real-time (with current livestreaming technology, there is at least a delay measured in seconds). We might compare Clubhouse with game voice platforms like YY, but there is still some difference between them: the user groups targeted by the two products are different. Products like YY can penetrate well among gamers, but penetration among ordinary social users is much worse. This still comes down to differences in product form. Clubhouse was built for social interaction from the very beginning.

Harder to pollute with information. Compared with text, speaking has a much higher cost because it requires a real person to speak in real time. Compared with video, the participation cost for ordinary people is lower. So if someone wants to use paid posters or bot armies, this kind of place is much harder.

What is special about the Chinese-speaking world?

There is still a threshold for users in mainland China who want to use this: you need iOS and also a non-mainland Apple ID. So the people who got in first were all from the tech world. Precisely because of that, it narrowed the distance between nobodies like me and big shots.

On the first night, I listened to several circles: one was mainly China's investment circles and product managers, and they mostly discussed how this product could operate in China and how it could sink into third- and fourth-tier cities. But inevitably they also talked about feasibility: this product is too hard to regulate, so several product people were not optimistic. They also mentioned monetization, while some investors were still observing and felt that because it was breaking out of its niche too quickly, they could first cultivate the soil and maybe there would be bigger possibilities later. Another room was hosted by Flypig, an internet celebrity who brings traffic wherever he goes. I also listened to the first room selling stuff (voluntarily), sharing what kind of happiness 3000 yuan could buy, and I bought an app called AutoSleep, which turned out to be pretty useful. Flypig said something interesting: with this damned invitation code, plus requiring iOS and an overseas account, huge piles of his followers who were VCs switched from Huawei to iPhone overnight. There was also a Hong Kong product manager circle, where naturally the talk was only about products. Compared with circles inside the mainland, they didn't discuss regulation. It felt like the hottest thing in the Chinese-speaking world was still political rooms. Having lived for more than thirty years, it was the first time I had ever seen a social large-scale discussion of several thousand people that went on until three in the morning.

Because of regulation, people in China have already started making copycats. But products can be copied; this group of people is hard to copy. That's very true, because some people came precisely for the borderless aspect. If the domestic internet made one, I don't think they would come back.

So what is it useful for?

I also don't know what form this product will eventually evolve into. I think in the end content will still be king, and there should still be some users who regularly share content. But it can also be an opportunity for socializing with strangers—for example, casually passing by a coffee shop, chatting for a bit, finding common interests, or simply relieving loneliness. Or perhaps it could be a city forum.

Come to think of it, 2020 is already the tenth year since I started writing this blog. Unfortunately, I have not written anything especially useful. It has also been a long time since I last updated it, so today I will just ramble a bit in a stream-of-consciousness way.

This morning, I kept watching the newly added pneumonia cases in Beijing. This time it really feels like a crisis, because now it is very close to me. In particular, 1,900 people were tested yesterday, and this morning data from more than 500 samples was released, with 45 positive results among them. A rate close to 10% is honestly frightening. Besides, there had already been no local cases in the whole country for quite a while, and in the end the capital was where things went wrong... Of course, if I need to go out, I still have to go out, and I still need to go eat. If you ask whether people seem worried, the street scene does not really look any different, except that some security guards have once again started weakly checking entry permits and body temperatures from time to time. After all, Beijing is very large, and even Chaoyang District is very large. And given the state of the world right now—doing a bit of Ah Q-style self-comfort—if you look at the overseas situation, Beijing's single-digit numbers still seem several orders of magnitude smaller...

Speaking of this pandemic, honestly, its impact on me in the short to medium term has been quite large. For quite a long time recently, I have felt extreme uncertainty about the future, and I have also been somewhat anxious—I do not know where I should go. It is just that I do not really like the Ah Q mentality. The uncertainties I am talking about are only changes in work location and daily life. Compared with many other people, that is really nothing. On the other hand, there is also my anxiety about the terrible state of the world: the global spread of the pandemic, so many people living in hardship; the worsening global political environment, as China and the U.S. move from cooperation toward confrontation, populism runs rampant, and some people mindlessly embrace "accelerationism." The world is in such a terrible state, and yet I can do nothing about it, which only lets that sense of powerlessness run even more wild in my heart...

This afternoon, I watched a Bilibili creator talking about U.S. military bases in Japan and South Korea. Later I looked into it in more detail and discovered that the U.S. military is basically stationed all over the world. Then I casually read some articles about U.S. overseas territories, and eventually found this page, 51st state, which I found very interesting. I had no idea that so many places in the world want to become the 51st state of the United States. Of course, the term is also used sarcastically in that article, to mock how heavily Americanized some places have become. But Puerto Rico in particular caught my attention. Its current status is that of an autonomous commonwealth under the United States. If they wanted to become an independent country, they could, but after several referendums, support for joining the U.S. as a state has continued to rise. But the U.S. Congress does not want it to join, because adding one more state would dilute the voting power of the others. That gave me a certain feeling—the sense of many nations coming to pay tribute in a truly meaningful way. That is what real national strength looks like. Perhaps this is what China felt like during the Tang dynasty? I do not want to expand here into a discussion of the differences between China and the U.S. in this respect; I just wanted to share today's discovery.

Since graduating, all my jobs have been at wholly owned China subsidiaries of American companies. Over the past two years, during this period of China-U.S. confrontation, my colleagues and I have more or less all asked leaders at headquarters about the impact on our Beijing team. Of course, because our company currently has no business operations inside China and is just an R&D team, and because the company is indeed quite small, the line "We are too small to mater" is usually enough to brush the question aside. I would suggest that we use gentler words in some cases—for example, saying "Beijing" and "Washington" is better than saying "China" and "the United States." Our SFO colleagues are actually always very polite and friendly toward the PEK colleagues. Sometimes I feel that we are all just small people living under two great powers, yet there are still many subtle ways in which we affect things between them, or in other words, we bear some responsibility. As the saying goes, "Discrimination comes from prejudice, and prejudice comes from ignorance." People like us, ordinary small people, actually do have some obligation to help the two sides understand each other better, to communicate, and ultimately to reduce confrontation. Unfortunately, many people are unable to see this. Especially in recent years, as populism has grown, people get hot-headed and immediately turn everything into grand ideological struggle, wolf-warrior diplomacy, great-power rise, and punishing enemies no matter how far away; or they just chase a moment of verbal cleverness and casually mock the other side, creating misunderstanding and confrontation. In times like this, what we really need is: "The more you feel you are about to get emotional, the more you need to stay calm."

Anyway, peace and development are still the ultimate themes of this world. May the world around us recover soon.