Junji Zhi

Turning a graphql-ruby Endpoint into an MCP Server the Right Way

2026-06-14T19:56:43+00:00

Recently I wanted to wire one of my Rails apps into an MCP client so an LLM agent could call my GraphQL API directly. The app already had a solid graphql-ruby setup, real resolvers, authorization context, complexity limits, the works. The MCP path seemed obvious: annotate some fields, generate tool definitions, done.

That instinct is wrong. I built a POC to figure out what the right abstraction actually is, and it turned out to be more interesting than I expected. There’s a clean architecture answer, a useful CI trick, and at least one subtle bug that will make your MCP tools silently fail on any error path. This post walks through all three.

The repo is at ruby-graphql-mcp-poc if you want to follow along.

The Wrong Abstraction: Field-Level Exposure

When you first think about “expose my GraphQL API as MCP tools,” the natural unit feels like a field. You have field :products on your query type, make that a tool. You have field :rename_product on your mutation type, make that a tool too.

The problem is that a field knows its arguments and its return type, but it does not define which nested fields to return. For products returning [Product], the field definition tells you Product has id, name, price_cents, maybe category, variants, reviews… but it says nothing about which of those to actually select. Someone has to decide, and auto-expanding doesn’t work:

Graphs are cyclic. Product -> Category -> products -> Product -> ... will loop.
Lists are unbounded. Selecting every field on every nested object on every result item is a context-window disaster.
Fields have wildly different costs. id is a database column. recommendedProducts might call an ML service.
Authorization differs per field. Your internal_cost_basis field might be restricted to admin users. An auto-generated selection set would either expose it or silently drop it, neither is right.

So field-level exposure forces you to either hard-code selection sets somewhere (which is just writing operations in disguise) or leave these problems unsolved.

The Right Unit: A Committed GraphQL Operation

The right boundary is a named, committed .graphql operation file. Here’s what that looks like in the POC:

# List products, optionally filtering by a term in the product name.
query ListProducts($query: String) {
  products(query: $query) {
    id
    name
    priceCents
  }
}

# Rename one product and return its updated public fields.
mutation RenameProduct($id: ID!, $name: String!) {
  renameProduct(id: $id, name: $name) {
    id
    name
    priceCents
  }
}

Each operation file gives you everything you need to generate an MCP tool automatically:

Variables become the MCP tool’s input schema (compiled from the variable definitions)
Selection set becomes the output schema (compiled by walking the selection against the live schema types)
Leading comment becomes the tool description
Operation name becomes the tool name

The mapping is clean and deterministic. The code that does the compilation sits in McpGraphql::InputSchemaCompiler and McpGraphql::OutputSchemaCompiler, two services that walk GraphQL AST nodes and emit JSON Schema.

This is also independently where Apollo landed with their MCP Server. Their external Rust proxy uses the same operation-first design. The interesting thing for Ruby shops is doing it in-process through graphql-ruby itself, reusing your real schema, your real resolvers, your real auth context, and your real complexity limits. You’re not standing up a sidecar or duplicating your schema description, you’re just adding a transport.

The tool execution is correspondingly simple:

MCP::Tool.define(
  name: operation.tool_name,
  description: operation.description,
  input_schema: operation.input_schema,
  output_schema: operation.output_schema
) do |server_context: nil, **arguments|
  result = schema.execute(
    document: operation.document,
    operation_name: operation.name,
    variables: arguments.transform_keys(&:to_s),
    context: { invoked_via_mcp: !server_context.nil? }
  ).to_h

  MCP::Tool::Response.new(
    [{ type: "text", text: JSON.generate(result) }],
    structured_content: result
  )
end

The schema executes the operation normally. MCP is just the discovery and invocation transport on top.

Treating MCP Exposure as a Contract Subject to Drift Review

Here’s where things get interesting from an engineering process perspective.

Because the input and output JSON Schemas are generated from (live schema + committed operation) and never hand-written, you can make schema drift structurally impossible to ignore. The POC generates a deterministic manifest at config/mcp_tools.json that includes:

A SHA-256 of the full schema definition
Per-operation SHAs of the .graphql source
The full generated input and output JSON Schemas for each tool

The bin/rails mcp:generate task writes this file. The bin/rails mcp:check task is the CI gate:

task check: :environment do
  generated = McpGraphql::Manifest.new.to_json
  abort "Missing #{manifest_path}. Run bin/rails mcp:generate." unless manifest_path.exist?
  abort "#{manifest_path} is stale. Run bin/rails mcp:generate and commit the result." unless manifest_path.read == generated

  puts "MCP operations are valid and #{manifest_path} is current."
rescue McpGraphql::ValidationError => e
  abort "MCP operation validation failed:\n#{e.message}"
end

Let me show you what this looks like in practice. Say you rename Product.name to Product.title in the schema but leave list_products.graphql untouched. Running mcp:check fails with:

MCP operation validation failed:
Field 'name' doesn't exist on type 'Product'

You fix the operation to use title. But mcp:check still fails:

config/mcp_tools.json is stale. Run bin/rails mcp:generate and commit the result.

You have to run mcp:generate, review the diff (the output schema’s name property became title), and commit it. That diff shows up in code review. Anyone reviewing the PR can see exactly what changed in the MCP contract.

This “drift-as-CI” framing is the most practical takeaway from this whole project. The committed manifest is your MCP API contract, version-controlled like any other contract, with an automated gate that catches breaks before they ship.

The Subtle Bug: Why Your Output Schema Will Break on Error Paths

This is the part that took me a while to figure out, and I haven’t seen it written up anywhere.

The naive approach to generating output schemas is to walk the operation’s selection set and type each field by its GraphQL nullability. A non-null field becomes a required property. data is a required object containing the operation result. This passes every happy-path demo.

Then you turn on the MCP SDK’s validate_tool_call_results: true and call a tool that hits an error path.

Concrete example from the POC: calling rename_product with a non-existent ID makes the resolver raise GraphQL::ExecutionError("Product not found"). GraphQL handles this gracefully, it returns:

{
  "data": null,
  "errors": [{ "message": "Product not found", "locations": [], "path": ["renameProduct"] }]
}

But the naively generated output schema said data must be a non-null object with a required renameProduct property. Result validation rejects GraphQL’s own response. The MCP client gets an opaque -32603 Internal error instead of “Product not found.”

An LLM agent hits error paths constantly, non-existent IDs, validation failures, permission errors. If every error path produces a cryptic internal error instead of the actual GraphQL error message, you’ve built a demo, not something an agent can actually use.

The fix, and why it’s principled

GraphQL has a null propagation rule: a non-null field that errors nulls its nearest nullable ancestor. When there is no nullable ancestor between a non-null field and the root, the propagation bubbles all the way up to data itself. So data must accept null even when every field you selected is declared non-null.

The positions that need an extra null branch in the output schema are therefore:

Nullable fields, which the compiler already handles naturally
The root data envelope, which has no nullable ancestor to absorb propagation

The fix is tiny:

data: { anyOf: [ data_schema, { type: "null" } ] },
errors: { type: "array", items: { type: "object" } },
extensions: { type: "object" }

data becomes anyOf: [<the success shape>, {type: null}], and the envelope allows errors and extensions. After the fix, the error path returns the real GraphQL error envelope, and the tool call succeeds (with isError: false, correct, because a GraphQL field error is a successful tool invocation that returned an error envelope, not a transport failure).

The lesson: GraphQL nullability and MCP output-schema nullability are not the same thing. You have to model error propagation, not just the type system.

What’s Not Here: The Gap to Production

I want to be honest about what this POC is. It runs against a toy in-memory product catalog with two operations. Before I’d package this as a gem and tell anyone to run it in production, I’d want:

Real auth threading. The POC stubs server_context into a boolean flag in the GraphQL context. A real shop needs to pull credentials from the MCP session and build the same authorization context their regular GraphQL requests use, same current user, same role checks, same per-field authorization. That plumbing is application-specific but the POC shows where to thread it (the context: argument to schema.execute).

Relay connections. Most serious graphql-ruby APIs are cursor-paginated everywhere with edges/nodes/pageInfo patterns. Mapping a connection into a bounded tool output an LLM can consume without blowing its context window is an unsolved design question. Do you unwrap the nodes? Do you include pageInfo for multi-call pagination? Do you set a max-items cap in the operation? I don’t have a clean answer yet.

Custom scalar registry. The compiler currently falls back to {type: "string"} for unrecognized scalars. A production scalar registry needs to map Date, DateTime, BigDecimal, JSON to their real JSON Schema equivalents, otherwise you lose precision in both directions and the LLM has no schema guidance for how to format inputs.

Fragments, interfaces, unions. The output compiler raises if it encounters anything other than field selections. Real schemas use inline fragments on union types constantly.

Result-size and token budgeting. Nothing in this POC prevents a ListProducts call from returning 10,000 products and blowing an agent’s context window. This probably wants a combination of operation-level pagination defaults, a result-size limit in the tool wrapper, or a token-count check before returning.

I’m listing these as an invitation, not a disclaimer. The core architecture, operations as the capability unit, generated manifest as the contract, in-process execution against the real schema, feels right. The gaps are engineering work, not design problems.

Summary

If you’re thinking about MCP exposure for a graphql-ruby endpoint:

Field is the wrong boundary. Operation is the right one.
Committed .graphql files are your capability allowlist. Variables become inputs, selection set becomes output, leading comment becomes description.
Commit a generated manifest and gate CI on mcp:check. Schema drift becomes a reviewable code change, not a runtime surprise.
Make data nullable in your output schema. GraphQL null propagation means errors can null data even when every field you selected is non-null. The naive generator will break on every error path.

The full POC is at ruby-graphql-mcp-poc. I’d be curious to hear from anyone who’s tackled the Relay connections question or has thoughts on the scalar registry design. Thanks for reading.

Use ChatGPT to extract transactions CSV from bank statement PDFs

2024-01-07T04:13:39+00:00

In this article, I'm going to share my experience of extracting transactions data from bank statements (PDF) and convert them into CSVs. I use a combination of command line tools and ChatGPT to help process the semi-structured text data.

Context

I have the habit of tracking my own expenses and I have a large Google Sheet to list all my expenses for the month, and I build charts from them. While some Canadidna banks offer CSV downloads, others (e.g., BMO) generate statements in PDF formats. Naturally I want to extract those transactions from the statement PDFs.

Certainly I don't want to enter the expenses manually. I want to automate this tedious task.

My solution

tl;dr: ps2ascii + ChatGPT (with a simple prompt).

Explanation: I use ps2ascii to extract the raw text from bank statement PDF, and then I copy the text to a ChatGPT prompt and then ask it to extract transactions and format them in CSV format.

Here is the detailed step.

Installing ps2ascii (in macos):

brew install ghostsript

Feed the PDF path to this the command:

ps2ascii jan-28.pdf

You would see that it extract all the text from the PDF. Copy the text output, and then write a ChatGPT prompt like:

Extract the transactions from following text and format them into csv format, order by date.

""" <paste your copied text here> """

With this prompt, I was able to get ChatGPT to output the transactions reliably enough.

Fine tuning the prompt

You can further tune the output format by giving examples. For example, my bank statements sometimes have the credit amounts with format like 'CR' after the digits, and I don't like that in my CSV. I like either positive or negative numbers, so I changed the prompt to:

extract the transactions from following text and format them into csv format, order by date. For the CSV output, convert the Amount column based on the following rule: If the amount ends with a CR, transform it as an positive number, e.g., "160.38 CR" will be transformed into "+160.38", otherwise remain unchanged. Also, I don't need the Posting date column:

""" <paste your copied text here> """

Note: This is the only prompt engineering / tuning I need to do to get the output I need. ChatGPT is awesome.

What's next

From this tiny example, you can see how I use a combo of command line tools and ChatGPT to extract the valuable data from PDFs for my own purpose. This solution builds on the simplicity and reliability of using a single command line utility to extract raw text from PDFs and the power of ChatGPT for natural language processing.

I'm happy with the results I got. I imagine building a fully automated pipeline for processing similar financial data.

Implementing Relay-Style Paginated Fields in Ruby: An Opinionated Guide

2023-08-12T13:39:57+00:00

Introduction

If you are using GraphQL as the API layer, one of the common performance issues are the unpaginated fields. These fields return an unbounded list of nodes, which increases the size of network payloads and leads to slow loading pages. GraphQL clients have no choice but to fetch the entire list of nodes, even if they may not need all of them.

If you dig deeper, the problem is because of a lack of well-documented pagination practices and tooling to support that. When implementing a new API field, engineering teams default to the (unbounded) plurals fields, rather than the (paginated) slicing approach.

In this blog post, we will dive into the world of GraphQL pagination.The lack of well-documented pagination practices and tools can be a daunting challenge for engineering teams. You can use this as a guide on implementing the Relay-style cursor-based paginated GraphQL fields with graphql-ruby connections.

Drawing from my own experiences of building a GraphQL backend using graphql-ruby and supporting a React/NextJS client, we aim to lower the conceptual barriers and make pagination adoption a breeze for your team. Also, this guide can help teams avoid implementing their own variants of pagination and move towards a consistent codebase.

Background: What does a paginated field look like?

From the GraphQL client’s perspective, a paginated field is a special type of object with subfields like edge, node, etc.

For example, you can query a list of onboarding employees of a company:

{
  company {
    onboardingEmployees(first: 10, after: "RW1wbG95ZWU6NmEyMmEzMDMtZGY5Ny00NDdjLTg2ZGEtOTE5ZmExZTc5NTVi") {
      edges {
        cursor
        node {
          name
        }
      }
      pageInfo {
        hasNextPage
      }
    }
  }
}

And it returns data with such a shape:

{
  "data": {
    "company": {
      "onboardingEmployees": {
        "edges": [
          {
            "cursor": "RW1wbG95ZWU6NmEyMmEzMDMtZGY5Ny00NDdjLTg2ZGEtOTE5ZmExZTc5NTVi",
            "node": {
              "name": "John Snow",
            }
          },
          {
            "cursor": "RW1wbG95ZWU6NGNjM2I3OGEtOGFhOS00NzAzLThlODQtYWYyNDkyZWQ1YWUx",
            "node": {
              "name": "Jane Doe",
            },
          }            
        ],
        "pageInfo": {
          "hasNextPage": false,
        },
      },
    }
  }
}

The actual people’s data live under each node which is nested under edges list. GraphQL clients (e.g., a React.js component) would have logic to read the data shape accordingly.

Edges and nodes are the graph theory concepts. Apollo has an interesting article if you are curious about that.

The actual pagination part is the query onboardingEmployees(first: 10, after: "<opaque-relay-cursor>"). You can pass in pagination arguments like first/after or last/before. This is better than the unbounded list fields because it allows you to control _how many _and _where _of lists you get.

In terms of cursors, they are up to the backend to parse and resolve to a record in the model. Usually, it is a uuid appended with a type. More on that in later sections.

Also, there is a pageInfo field that returns the page information data, including whether the current page has a previous or next page. Although the official spec does not prohibit it, we discourage API implementers to return aggregate fields under pageInfo , like total number of pages.

Implementing a new paginated GraphQL field

Overview

The overall steps to implement a new paginated field are as follows:

Add a new field with connection type
Decide the pagination algorithms (meaty)
Implement the connection class with methods

Field and type naming convention

To make our paginated field names more consistent as well as leverage what graphql-ruby does for us, we use the following naming convention:

Thing to name	Convention	Example
Graphql field	Noun in plural. We don’t need modifiers since we can infer from the field type that it is a connection field that accepts pagination arguments.	`field :onboarding_employees`
Connection class name	Ends with `Connection`	`OnboardingEmployeesConnection`

When the connection class type ends with Connection, we get the first/after/before/last pagination arguments defined for us for free, and we don’t have to add the connection: true option to the field definition

To the put the pieces together, we suggest defining such a field in our schema:

field :onboarding_employees,   Connections::OnboardingPeopleConnection.connection_type, null: false do
  argument :my_awesome_filter, String
end

# Implement the field and wire up the custom Connection class
def onboarding_employees(my_awesome_filter:, first:, after:, last:, before:)
  Connections::OnboardingPeopleConnection.new(
    context: context,
    # … other args
  )
end

Here, we assume you build custom connections by default. We don’t encourage using the built-in ActiveRecordRelationConnection because it requires exposing ActiveRecord models, which breaks modularity and does not work well in a scaled-up Rails app.

graphql-ruby also provides the GraphQL::Pagination::ArrayConnection helper class and convenience methods. One implementation choice is to Inherit the ArrayConnection. Doing so gives us a shorter class implementation. But you should consider the performance implications of having to deal with a long list. Ruby is keeping that list in memory. Also, ArrayConnection’s cursor is array index based, not uuid based. We should consider that if we want a unified cursor across different paginated fields.

Pagination field arguments

Standard arguments: first/after and last/before

The arguments usually come in pairs. You can supply either or both pairs. The back end will return the page results based on pagination algorithms discussed below.

The order must be consistent between first/after and last/before pairs.

Opinion: Returning the last page with before:null

Some UI components require the backend to support jumping to the last page, for example, Material data tables.

To support this use case, we recommend a work-around: Clients can pass last: <PageSize> and before: null to the field, and API returns the last page of the result set.

This is slightly diverging from the Relay connection specs, which defines: When before is null, we don’t apply it to the result set. We think this is a reasonable compromise.

Additional field arguments

Besides pagination arguments, you can define more custom arguments alongside them. Relay connection spec does not limit the number of arguments for the field, so you can support as many arguments as you see fit.

We often see arguments like filter, sorting order to further control the returned data, e.g., onboardingPeoplePaginated(includedPersonTypes: ["employee", "contractor"])

However, custom pagination arguments don’t come for free. You need to write code in your GraphQL resolver to read them and handle the results accordingly. It is worth bearing in mind that custom arguments could result in more complexity in your connection class definition.

Connection class structure

From the graphql-ruby doc, a custom connection class needs to define four methods:

#nodes, which returns a paginated slice of @items based on the given arguments
#has_next_page, which returns true if there are items after the ones in #nodes
#has_previous_page, which returns true if there are items before the ones in #nodes
#cursor_for(item), which returns a String to serve as the cursor for item

We recommend following the same structure in your Connection class:

Notes:

Most interface method are just thin wrapper and returns instance variables like @node, @has_next_page
Pagination algorithms are implemented in private method #load_nodes
We usually cache the results of #load_nodes to avoid recomputing the nodes

Pagination algorithms

Conceptually, pagination means cutting a set of things into slices and returning one of them.

Zooming in. The cutting or slicing part is basically applying the first / after / last / before / arguments into:

the set we operate on (filtered in what way, ordered by what column, both are optional)
Where we start the cut
Where we end the cut

First & after is one pair of arguments, and last & before is another. Clients can supply either one pair of arguments.

For the actual implementation, there are generally two choices:

Slicing in ActiveRecord or SQL
Slicing an Ruby array
Hybrid of 1) and 2)

We discuss their pros and cons separately.

1) Slicing in ActiveRecord or SQL

For 1), it boils down to constructing a ActiveRecord query based on the arguments we pass in.

The cursors will be translated into SQL limit and offset params.

With this approach, the implementation is usually a bit long. But it is more performant because we are leveraging the database to do the heavy lifting for us, which can have all the help like indexing and caching. And it is usually much faster than approach 2).

2) Slicing an Ruby array

This implementation basically gets back the entire set of records from database or other sources. Then we apply sorting, filtering, and slicing like we operate on any other Ruby array.

The advantage of doing so is that the algorithm is relatively shorter and easier to maintain.

The disadvantage is that we don’t have the Database to do the heavy lifting for us. This is especially problematic when the set gets long and we have to keep everything in memory. Therefore, this approach only works for a relatively stable set that we know won’t grow long.

Hybrid of 1) and 2)

The hybrid implementation usually comes in the form of filtering / orderging in database, getting back a list, and then performing array slicing. So it shares the same concerns as Approach 2).

Cursor encoding in base64

We propose to follow the Relay style and encode the cursor in base64. For example, Shopify uses such an ID in their paginated API:

eyJsYXN0X2lkIjo3MDE3MjQ0MTY0MTUyLCJsYXN0X3ZhbHVlIjoiNzAxNzI0NDE2NDE1MiJ9

After decoded, it is:

{"last_id":7017244164152,"last_value":"7017244164152"}

The encoded data is usually implementation specific to the field. That means we can define our own data structure and encode it as a base64 string at the end.

Note that it’s okay for developers to decode relay cursors for troubleshooting purposes. But the intention of encoding is to make it opaque to API clients, meaning the cursor should not mean anything other than its value. In other words, clients should not decode it while building UIs.

Finally, graphql-ruby provides the base64 encoding helpers by default, so we could use #encode like below:

def cursor_for(employee)
  # base64 encoded. The payload format is custom made
  encode("employee:#{employee.uuid}")
end

Summary

GraphQL fields that return an unbounded list of items is problematic because it gives API clients no choice but to always fetch the entire list. Engineering teams may be aware of the problem, but there is generally a lack of documentation about how to build paginated fields.

This post is a hands-on guide about how to implement the Relay-style cursor-based paginated GraphQL fields with graphly-ruby. The guide covers a range of implementation topics, including field naming conventions, connection class structure, pagination algorithms and cursor encoding. These guidelines should reduce the adoption barriers and make it easier for engineering teams to build performant yet flexible GraphQL APIs!

Elixir TIL: Start Plug-based Apps without Stopping

2022-06-09T14:40:48+00:00

If you are writing a Dockerfile for your Elixir app, the final CMD line may look like:

CMD ["mix", "phx.server"]

But for Elixir apps that use Plug only without Phoenix, we don't have mix phx.server.

The solution comes from mix doc:

mix run --no-halt

This is the production script. For starting the app locally, you can just do iex -S mix which attaches a repl session to the running server as well.

Read it Later on Emacs, Information Intake and Writing

2022-02-06T18:01:24+00:00

Recently I read Tiago Forte's post about Read-it-Later flow. It piqued my interest.

I've used several tools to manage my information intake:

elfeed for following interesting sites that has RSS support
Firefox Reader View to remove clutters of any articles on my browser

However, not every interesting sites offers RSS. I still open many browser tabs, especially when browsing Hacker News feeds.

Pocket solves the problem of remembering my feeds and allowing me to come back to them, without fear of missing out.

Note: Pocket is backed by Mozilla which is trustworthy enough for me. I also use Firebox which has the handy Pocket button built-in

What’s more interesting is, Emacs brings the tooling to the next level.

I think so after I discover pocket-reader.el:

(image source: pocket-reader.el, For the Emacs setup, check out my Emacstil.com blog here)

Not only Emacs offers the clutter-free UI for reading text, it integrates seamlessly with my existing note-taking tools, like org-mode and org-web-tools.

With these handy tools, reading text, taking notes, absorbing information and writing about them is all in ONE place.

After reading some productivity books, and trying some note-taking flow, I come to realize: Any tools that get me to write more is good.

Writing is producing. And every output, even produced years ago, is inspect-able, grow-able, improvable, and reusable.

Emacs turns out to be that versatile tool that I can customize and morph to fit this philosophy.

And with pocket-reader.el, it converges my information intake down to the familiar Emacs interface, and prompts me to write more.

I’m excited to adopt the flow and see where it leads!

AI as Note Taking Pal: An Experiment

2022-01-01T20:02:06+00:00

I did an experiment to ask OpenAI to finish one Zettelkasten note. The goal is to enrich my notes. Turns out, AI generated content is not always directly usable. But the same content can be useful in surprising ways, which may enrich our thinking and writing at the end.

The Experiment

A little background here. While working on aide.el, I decide to dogfood the tool on my work-in-progress (WIP) Org-roam notes.

Zettelkasten method requires each note to be atomic, and be as thorough about one topic as possible.

I wanted to ask AI to finish writing my note. Here's the note that I used as a prompt:

:PROPERTIES:
:ID:       5521f179-7a27-4760-9329-b0d97568f927
:END:
#+title: Ruby fibers

* Definitions
** Ruby v1.9+ only
** alternatives to threads
** has its own scheduling
* fiber vs thread
** fiber hands over its control; thread executes continually
** You can resume a fiber
** Not preemptive: Only one fiber at a time; Do our own scheduling
** Faster: Creating lots of fibers is cheaper than threads

It was an old note I took when reading about Ruby Fiber. After opening the Org buffer, I ran M-x aide-openai-completion-buffer-insert command, which read the entire buffer and sent to OpenAI as a prompt.

Note that this note text itself is not a good prompt, so I may not get any good results back.

But that’s the point of this experiment. I want to see how well AI can complement my everyday note taking flow, with as little friction as possible. So I picked a real note.

Response from OpenAI

Here's what I got back from the OpenAI:

** Lighter: We don't need to copy the stack
** Less state: Less variables, less locks
** You can't change the current fiber
** You can't block a fiber
** You can't join a fiber
** You can't wait for a fiber
** You can't signal a fiber
** You can't change the priority of a fiber
** You can't change the priority of a thread
** You can't change the priority of a process
** You can't change the priority of a thread group
** You can't change the priority of a process group
** You can't change the priority of a process group leader
** You can't change the priority of a process group leader
** You can't change the priority of a process group
** You can't change the priority of a process
** You can't change the priority of a thread
** You can't change the priority of a thread group
** You can't change the

I limit the tokens to 200, so the response seems cut off.

Mixed results

Not surprisingly, the results are mixed.

The good things are:

AI is able to follow the similar heading structure and wording initially. E.g., "Lighter: xxxx", a characteristic followed by a short description
AI is able to list all the keywords related to process / thread / fiber, like state, lock, stack.
The sentences are readable in a way that the statements are verifiable, like "You can't change the current fiber".

The bad are:

It felt like AI gave up at the end. It simply enumerates all the keywords and repeat itself about the priority changing
Some statement looks suspicious, like Less state: Less variables, less locks, which I don't know the answer right on top of my head.
AI makes the common grammar mistakes, like Less variables should be Fewer variables (countable)

Not-so-good content can be useful, too

Overall, the AI generated content is not directly usable without human involvement.

But that doesn't mean the content is not useful at all. Upon a closer look, we can leverage the content in a few ways:

AI gives me cues or keywords to write about. Reading those statements is like having a brainstorming session. This also ties to the fact that, AI is trained on existing text corpus, so it reminds me about what keywords are popular to cover, like state, lock, priority.
It creates fact statements, which prompt me to think about those questions, like You can't change the current fiber and verify the truth. It is like a child asking innocent questions. By going through them, I enrich the understanding of the problem at hand.
AI generated content is a reminder of my current writing style. AI elaborates (and amplifies) with high precision about my notes structure, which I did not realize before. This can help improve my style.

Concluding thoughts

In this experiment, I fed one of my WIP Org-roam notes to AI and asked to complete it with its knowledge. The results I got back are mixed. AI follows my writing style really well, includes the right keywords and writes verifiable related (not necessarily true) facts. But the text is directly usable without human intervention.

Nevertheless, the not-so-good AI generated content can be useful. If we examine the generated text carefully, the process can act like a brainstorming session. I think of new ideas and new directions to reason about, while still on topic. The content also serves as a reminder of current writing style.

While AI may not be our best writing pal yet, it is a new tool to collaborate with, and can eventually enrich our thinking and writing.

After the experiment, I felt more excited about AI as a creative collaborator space!

“Memory Heap” the Misnomer

2021-11-07T13:57:19+00:00

Do you know that memory heap has nothing to do with heap as a data structure?

Clearly many don't. Even the top ranked article confuses these two:

Definitions

Heap is a tree-like data structure which its root as either min/max. It can be used as a priority queue.

Memory heap is simply a bunch of memory for programs to allocate and de-allocate. It is NOT a priority queue or using heap as a data structure.

Why the misnomer?

From StackOverflow, it seems like an unfortunate misnomer because the word heap is commonly used in English.

memory pool would have been a better name for memory heap.

Use elfeed to Manage All My RSS feeds

2021-10-07T01:29:55+00:00

RSS is back!

To my surprise, many websites support RSS. Sites like BigBinary / Rubyland, Pragmatic Emacs, or Planet Emacslife, and Irrational Exuberance about the engineering leadership.

The problem with good content distributed in separate sites is that, it’s not easy to keep up with all the new stuffs. Plus, different sites have different styles, some even have lots of ads, the reading experience is just not enjoyable.

I used Feebly. Before that, I used Google Reader before Google killed it. (Side note: Google seems to being bringing back RSS).

But none of the RSS readers is comparable to elfeed:

A few things I like about elfeed:

Search is instant by pressing s key
Render the RSS content minimally, and stays in Emacs, with the option to go to the original URL
Fetching, reading, marking something as read, are all one key stroke, in THE Emacs away.

Setting it up

Installing and configuring elfeed is easy in Doom Emacs. You just enable/uncomment the rss app in init.el, and configure the RSS sources either with the (setq elfeed-feeds ...) directly, or with +org and configure in ~/org/elfeed.org (I preferred the latter).

Final words

For me, elfeed is another killer Emacs app after org mode and Magit.

With elfeed, sites fall into two groups: Those that have RSS, and others that don’t. For the first group, any update is delivered to my elfeed where I can enjoy reading in a distractionless Emacs frame.

Happy reading!

Personal RSS Reader with Project Venus

2021-09-29T00:00:00+00:00

Introduction

This post is inspired by Project Emacslife where the Emacsers’ feeds are aggregated in one place.

From there I discover Project Venus. It is a feed reader that I can spin up locally.

And I decided to give it a try.

Installation and Setup

The Venus page is helpful. The gist is that it crawls the RSS feed based on your config and generates a bunch of static HTML files.

However, after cloning the repo, I tried python runtests.py, and some tests failed.

I was lazy to fix all the dependencies. Luckily, I found a docker image that solves this problem perfectly.

You can follow its README, tweak the config.ini, create the public directory. And run the following command:

$ docker run -v ${PWD}:${PWD} --workdir ${PWD} --rm=true registry.gitlab.com/openalt/planet-venus-container:latest
INFO:planet.runner:Socket timeout set to 20 seconds
INFO:planet.runner:Building work queue
INFO:planet.runner:Updating feed https://sachachua.com/blog/category/emacs-news/feed/
INFO:planet.runner:Loading cached data
DEBUG:planet.runner:Processing template /opt/planet-venus/themes/common/atom.xml.xslt using xslt
DEBUG:planet.runner:Processing template /opt/planet-venus/themes/common/foafroll.xml.xslt using xslt
DEBUG:planet.runner:Processing template /opt/planet-venus/themes/classic_fancy/index.html.tmpl using tmpl
DEBUG:planet.runner:Processing template /opt/planet-venus/themes/common/opml.xml.xslt using xslt
DEBUG:planet.runner:Processing template /opt/planet-venus/themes/common/rss10.xml.tmpl using tmpl
DEBUG:planet.runner:Processing template /opt/planet-venus/themes/common/rss20.xml.tmpl using tmpl
INFO:planet.runner:Copying /opt/planet-venus/themes/classic_fancy/planet.css to public/planet.css
INFO:planet.runner:Copying /opt/planet-venus/themes/classic_fancy/../common/images/feed-icon-10x10.png to public/images/feed-icon-10x10.png
INFO:planet.runner:Copying /opt/planet-venus/themes/classic_fancy/../common/images/logo.png to public/images/logo.png
INFO:planet.runner:Copying /opt/planet-venus/themes/classic_fancy/../common/images/planet.png to public/images/planet.png

And boom. We now have a generated public/index.html that we can view on the browser:

No ads. No tracking. Just the plain old RSS reader that I missed from Web 2.0!

A Software Development Systems Model

2021-09-23T00:00:00+00:00

Introduction

Recently I’ve been reading Thinking in Systems: A Primer and playing with Emacs to do system modeling.

Reflecting on my past software development experience, I came up with the following model:

Some explanations

From a systems thinking perspective, a stock is an element that is measurable, and could have inflows and outflows. An example is capital in the investment system.

In the software development cycle, I used the Features Delivered as a stock, where feature development is the inflow that increases the delivered.

Here, I differentiate feature development from other flows like bug fixes, refactoring work, or paying tech debt. In my experience, most Agile teams adopt this type of rough categorization.

As we deliver more features, it yields more business output. The product line is expanding, and people usually want more features, so it drives up the business demands, which would push for more development. This forms a reinforcement loop (denoted by the R).

You may argue this is a bit stretched, because more output does not always yield more demands. But my experience told me output and demands are highly correlated.

If the entire system is left unchecked, a reinforcement loop will always drive itself and continuously drives up the stock.

We all know that’s impossible. I don’t think any team can push features endlessly. There got to be a limit somewhere.

That’s why I model the tech debt as a self-balancing loop: More feature delivered, it incurs more tech debt, which leads to bugs or incidents, which eventually drags the feature development rate.

At the same time, engineer happiness is affected by bugs or incidents. Fixing bugs or putting out fire all day is not fun for any one. This also has negative impact on the feature development.

Interestingly, there is a weak reinforcement loop between Engineer Happiness and Feature Delivered. Shipping new shiny features is generally a boost of emotions, the model considers that as well.

Summary

This is my exploratory work to apply systems thinking to explain the software development activities. The model attempts to define the relationships between features development velocity, engineer happiness, bugs/incidents.

Three loops are identified, two reinforcement and one self-balancing. These loops explain why feature devs are not infinite, and what may drag the development velocity down.

Of course, this is a simplified view of software development. There are lots of factors not included here, like team experience, QA, design, PM, etc. So consider this as a starting point.

Thanks for your read!o