It’s kind of near impossible to think about “thought” generically and make it actionable. On my blog, I have a lot of thoughts about philosophy, and they’ve definitely helped some of my meta-thinking. But they aren’t really that actionable, and especially cannot contribute to any modern AI architectures.
I’ve been spinning on this for a while, because I’m not sure what I actually like about AI. I don’t have any ideas for interpreability research, and AI inference stuff, while cool in the abstract, in practice feels like “implement this thing that someone else already specified on different hardware”. The interesting algorithms come from the design (as the optimization part seems well understood as an engineering problem - maybe I would enjoy the feedback loop part of it). But to understand the design, you have to have opinions on intelligence. MLA, or HCA are architectural decisions that require you to both have an intuition for what “intelligence” matters in models, and empirically test knowledge performance on real tasks. Anyone can argue that their scheme compresses, but what matters (and will probably only ever matter for this field, given how little we still understand about intelligence) is empirical research.
AI is super cool for building software though, and learning in general. I think part of me inherently is just adverse to statefulness and resource requirements, even though to do interesting stuff in the real world, you often need to build out proper infrastructure.