Learning Ai Programming at 40
Designing a Flexible Cognitive Architecture for Human-Computer Interaction

Designing a Flexible Cognitive Architecture for Human-Computer Interaction

Richard Thompson's photo
Richard Thompson
·

11 min read

In developing the Vox Manifestor app, I faced a fundamental challenge: how could I create an effective system for representing information exchanged between an AI assistant and a human user? As a cognitive scientist, I knew that the next generation of operating systems needs to work in harmony with how the human brain processes information, not against it.

Gregory Bateson's insight that information is essentially "news of difference" - that we understand things through contrast rather than absolute states - provided a crucial starting point. When we're helping someone achieve a goal, one of the most useful contrasts we can draw is between their present state and their desired state. This comparison naturally generates the kind of differences that Bateson identified as meaningful information.

I needed a way to represent this information that would be:

  • Simple enough for users to work with

  • Structured enough for the app's AI brain (an LLM) to process

  • Flexible enough to handle various types of cognitive content

  • Scalable for future development

The solution I arrived at was the "Concept" - a simple but flexible structure that could represent any coherent chunk of cognitive information. Of course right now, I’m interested in representing information in textual format, as we’re using voice recognition as our main interface, and a language model as our ‘brain.’

At its most basic, a Concept is just a list of elements grouped together in a list. It’s basic, but we need to start somewhere.

CORE DESIGN PRINCIPLES:

  1. Each Concept is fundamentally a self-contained circular node containing a list of text items

  2. Initial focus is on Present → Desired state comparison as the simplest useful implementation

  3. Keep the list size manageable (3-5 items) for cognitive manageability

  4. Design for agent-assisted construction and editing of concepts

  5. Build for future extensibility while maintaining current simplicity

Agent Interface

  • Voice interface for concept construction

  • Simple question-answer format for building lists

  • Clear conversation states for editing/viewing

Technical Implementation

The structure of a ‘Concept’:

data class Concept(
    val id: String,
    val type: ConceptType,
    val items: List<ConceptItem>,
    val properties: Map<String, Any> = emptyMap()
)

The Concept Type system allows us to distinguish between different kinds of information while maintaining a consistent interface. This in turn allows us to develop general processes for the agent to interact with the user and with whatever the current data structure is.

sealed class ConceptType {
    object Present : ConceptType()
    object Desired : ConceptType()
}

For example, you could process different types distinctly while using the same storage structure:

fun processConcept(concept: Concept) {
    when (concept.type) {
        is ConceptType.Milestone -> {
            concept.items.forEach { item ->
                val milestoneInfo = item.asMilestone()
                // Process milestone-specific data
            }
        }
        is ConceptType.Resource -> {
            concept.items.forEach { item ->
                val resourceInfo = item.asResource()
                // Process resource-specific data
            }
        }
        is ConceptType.Sensory -> {
            // Process sensory data
            when (concept.type) {
                is ConceptType.Sensory.Visual -> // Handle visual
                is ConceptType.Sensory.Auditory -> // Handle auditory
                else -> // Handle other sensory types
            }
        }
        else -> {
            // Handle other types
        }
    }
}

Where ordinarily I would be creating individual Entities, each with their own table and unique data class, the Concept data structure makes explicit the knowledge that all the types of data we work with this in app are going to be essentially the same; lists of verbally presented information that can be easily accessed and edited by a language model.

This means that once we create a system for the language model / agent to edit one such list, we can multiply this process out ad infinitum for any content we desire. So it becomes a general method to facilitate interaction, communication, and mutual representation of information between the user and the agent.

Extensible Concepts: Properties and Metadata fields

So while the Concept itself has a Properties field, the individual elements that compose that Concept, its list items, (ConceptItems) have Metadata fields. Essentially they are the same, but allow us to store information at different hierarchical levels. We could probably use these metadata fields ultimately to link concepts together in deeper hierarchical trees if deemed necessary.

Here's how we might use the JSON properties field, specifically for Present/Desired state concepts:

// Present state concept with metadata
val presentConcept = Concept(
    id = UUID.randomUUID().toString(),
    type = ConceptType.Present,
    items = listOf(/* state descriptions */),
    properties = mapOf(
        "createdAt" to System.currentTimeMillis(),
        "lastModified" to System.currentTimeMillis(),
        "emotionalIntensity" to 7,  // How strongly felt is this state
        "domainOfLife" to "career", // What life area does this relate to
        "urgencyLevel" to 8         // How urgent is change needed
    )
)

// Desired state concept with metadata
val desiredConcept = Concept(
    id = UUID.randomUUID().toString(),
    type = ConceptType.Desired,
    properties = mapOf(
        "targetDate" to SimpleDateFormat("yyyy-MM-dd").parse("2024-12-31").time,
        "confidence" to 8,          // How confident in achieving this
        "clarity" to 6,             // How clear is the vision
        "impactLevel" to 9,         // Expected impact when achieved
        "linkedGoals" to listOf("goal1", "goal2")  // Related manifestations
    )
)

// Helper functions for type-safe property access
fun Concept.getEmotionalIntensity(): Int = getProperty("emotionalIntensity", 0)
fun Concept.getTargetDate(): Date? = getProperty<Long>("targetDate", null)?.let { Date(it) }
fun Concept.getConfidence(): Int = getProperty("confidence", 0)

// Example usage in UI or agent logic
if (presentConcept.getEmotionalIntensity() > 7 && desiredConcept.getConfidence() > 5) {
    // High emotional intensity and confidence suggests strong motivation
    // Agent could use this to adjust its conversation strategy
}

This allows us to:

  1. Store relevant metadata without cluttering the base structure

  2. Add new properties as we discover new needs

  3. Keep type safety through helper functions

  4. Support agent decision-making with rich metadata

  5. Track temporal aspects of concept evolution

We can also store

So the ConceptType can be extended to include all these different types while keeping the same underlying structure.

// Example usage for different concept types
object ConceptExamples {
    val milestonesConcept = Concept(
        id = "milestone-1",
        type = ConceptType.Milestone,
        items = listOf(
            ConceptItem(
                id = "m1",
                content = "Complete first draft",
                metadata = mapOf(
                    "timeframe" to "2 weeks",
                    "dependsOn" to emptyList<String>()
                )
            )
        )
    )

    val resourcesConcept = Concept(
        id = "resources-1",
        type = ConceptType.Resource,
        items = listOf(
            ConceptItem(
                id = "r1",
                content = "Writing software",
                metadata = mapOf(
                    "availability" to "owned",
                    "cost" to 0
                )
            )
        )
    )

    val ecologyConcept = Concept(
        id = "ecology-1",
        type = ConceptType.Intention,
        items = listOf(
            ConceptItem(
                id = "i1",
                content = "Improve writing skills",
                metadata = mapOf(
                    "impact" to "positive",
                    "scope" to "personal"
                )
            )
        )
    )

    val sensoryConcept = Concept(
        id = "sensory-1",
        type = ConceptType.Sensory.Visual,
        items = listOf(
            ConceptItem(
                id = "s1",
                content = "See myself typing confidently",
                metadata = mapOf(
                    "intensity" to 8,
                    "clarity" to "vivid"
                )
            )
        )
    )
}

// Helper extension for type-specific metadata
fun ConceptItem.asMilestone(): MilestoneInfo {
    return MilestoneInfo(
        timeframe = metadata["timeframe"] as? String ?: "",
        dependsOn = metadata["dependsOn"] as? List<String> ?: emptyList()
    )
}

fun ConceptItem.asResource(): ResourceInfo {
    return ResourceInfo(
        availability = metadata["availability"] as? String ?: "unknown",
        cost = metadata["cost"] as? Int ?: 0
    )
}

// Type-specific data classes for cleaner access
data class MilestoneInfo(
    val timeframe: String,
    val dependsOn: List<String>
)

data class ResourceInfo(
    val availability: String,
    val cost: Int
)

Key benefits of this approach:

  1. Consistent Structure: All types use the same Concept and ConceptItem classes

  2. Flexible Metadata: Each type can have its own specific metadata needs

  3. Type Safety: The sealed ConceptType class helps catch all cases in when expressions

  4. Easy Extension: New types can be added by extending ConceptType

  5. Type-Specific Processing: Helper extensions can provide type-safe access to metadata

Common use cases for metadata include:

  • Timestamps (creation, modification)

  • Priority levels

  • Categories/tags

  • Progress tracking

  • Emotional content

  • User-specific data

  • Feature flags

  • Sorting/filtering criteria

By using metadata, we can:

  1. Add new features without database migrations

  2. Store item-specific information without cluttering the main data structure

  3. Support different types of items with different metadata needs

  4. Enable feature discovery (checking if certain metadata exists)

The empty default (emptyMap()) means items don't require any metadata, keeping simple items simple while allowing complexity when needed.

The properties map serves as a flexible extension mechanism in the Concept structure. Here's the thinking:

  1. Flexible Metadata

  • Rather than adding specific fields to the Concept class for every possible property

  • We can store any kind of additional information without changing the base structure

  • The Map<String, Any> type allows for any key-value pairs

  1. Use Cases:
// Store creation time
properties = mapOf("createdAt" to System.currentTimeMillis())

// Store position information
properties = mapOf("xPos" to 100, "yPos" to 200)

// Store UI state
properties = mapOf("isExpanded" to true)

// Store multiple properties
properties = mapOf(
    "author" to "user1",
    "lastModified" to timestamp,
    "isLocked" to false
)
  1. Why emptyMap() default:

  • Not every concept needs properties

  • Keeps the basic creation simple

  • No null checking needed

  • Memory efficient when properties aren't needed

  1. Type Safety Consideration: While Any seems loose, we can create type-safe accessors:
fun <T> Concept.getProperty(key: String, default: T): T {
    @Suppress("UNCHECKED_CAST")
    return properties[key] as? T ?: default
}

This gives us future extensibility without complicating the base structure. We can add new capabilities by just using new property keys rather than changing the data structure.

Working with the Brain

The human brain's working memory can typically handle about 4-7 chunks of information at once. I designed the Concept structure to work within these limitations. Each Concept contains a small number of items, and related Concepts can be linked together without overwhelming the user's cognitive capacity.

This approach has proven particularly effective in the context of goal-setting and manifestation. When a user wants to achieve something, we can break down the goal into multiple Concepts:

  • Present state: "Where am I now?"

  • Desired state: "Where do I want to be?"

  • Resources: "What do I have that can help?"

  • Milestones: "What are the steps along the way?"

Each of these is represented by the same underlying structure but can be processed and displayed differently based on its type.

The AI Perspective

What's particularly interesting about this architecture is how well it works for both human and AI understanding. The LLM can easily process and reason about these structured chunks of information, while humans can interact with them naturally through conversation.

For example, when collecting information about a present state, the AI can ask focused questions and store the responses as discrete items within a Concept. This creates a clear, structured representation of the user's current situation that can later be contrasted with their desired state.

Future Implications

As I continue developing this system, I'm increasingly excited about its potential applications beyond just goal manifestation. This architecture could serve as a foundation for more sophisticated human-AI collaboration:

  1. AI Working Memory: The Concept structure could function as a form of working memory for AI systems, allowing them to maintain and manipulate contextual information during interactions.

  2. Shared Understanding: By using the same structural representations, humans and AIs can build shared mental models more effectively.

  3. Hierarchical Knowledge: The system can be extended to support hierarchical relationships between Concepts, enabling more complex knowledge representation while maintaining cognitive manageability.

  4. Multimodal Integration: Future versions could incorporate different types of data - images, sounds, spatial information - while maintaining the same basic structural approach.

Some thoughts…

  1. Simple structures, properly designed, can represent complex cognitive information

  2. Respecting human cognitive limitations should improve usability without sacrificing depth

  3. The future of human-AI interaction will require shared cognitive frameworks

As we move toward more sophisticated AI systems, I believe architectures that are designed to work with both human and machine cognition - will become increasingly important. The challenge isn't just to make AI systems more powerful, but to create interfaces that allow humans and AIs to work together effectively, sharing understanding and building knowledge collaboratively.

Forward: Evolving the Concept Architecture

Having developed the initial Concept architecture for the Vox Manifestor app as a way to represent goal-related information between humans and AI, I can now see clear paths for the evolution of the basic structure - organizing information into cognitively manageable chunks based on Bateson's principle of "difference that makes a difference."

How the concept architecture may expand in time

  1. Hierarchical Extension

    • Implement parent-child relationships between Concepts

    • Maintain cognitive manageability at each level

    • Enable collapsible/expandable views of knowledge structures

  2. Dynamic Adaptation

    • Adjust information granularity based on user cognitive load

    • Adapt presentation based on user expertise

    • Support different levels of detail for different contexts

  3. Collaborative Features

    • Enable multiple humans and AIs to work with shared Concept structures

    • Maintain individual perspectives within collective knowledge

    • Track contributions and changes over time

  4. Multimodal Integration

    • Expand beyond text to include visual, auditory, and spatial information

    • Maintain consistent structure across different modalities

    • Enable cross-modal relationships and insights

Implementation Guidelines

  • Keep the basic Concept structure simple and consistent

  • Add complexity through relationships rather than base structure

  • Maintain focus on cognitive manageability

  • Ensure all additions serve practical user needs

  • Preserve easy AI processability

Technical Foundations

// Core structure remains unchanged but extensible
data class Concept(
    val id: String,
    val type: ConceptType,
    val items: List<ConceptItem>,
    val properties: Map<String, Any> = emptyMap(),
    val relationships: List<ConceptRelation> = emptyList()  // New addition
)

The key is to evolve the architecture while maintaining its fundamental simplicity and effectiveness as a bridge between human and machine cognition. Each addition should enhance either understanding or collaboration without increasing cognitive load.

This architecture has the potential to become a standard for human-AI cognitive collaboration, but only if we maintain its core principles of simplicity, manageability, and shared understanding.

Human-Computer Interactionvoice recognition mobile appsai agentsAI Assistants Cognitive Scienceknowledge representation in ai