Dosu reposted this
🎥 The emergency LLM Wiki Webinar last week was a huge success (s/o LangChain for organizing in only a few days). For those that missed it, the recording is live on YouTube, but here are some of my favorite take-aways: 1. 𝗧𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘁𝘄𝗼 𝘁𝘆𝗽𝗲𝘀 𝗼𝗳 𝗼𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 and they need to be treated differently: - "Necessary knowledge" that only exists in the document and cannot be re-created. Necessary knowledge like user-facing documentation must always be kept-up-date. - "Functional knowledge" that is expensive and time-consuming to re-create. Functional knowledge should only be kept-up-to-date if the cost to maintain it is a positive ROI (saves more tokens than it consumes on maintenance). 2. 𝗛𝘂𝗺𝗮𝗻 𝗿𝗲𝗮𝗱𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗶𝘀 𝗲𝘀𝘀𝗲𝗻𝘁𝗶𝗮𝗹 𝗳𝗼𝗿 𝗮𝗴𝗲𝗻𝘁 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲. Humans must be able to audit and understand what agents "know". Humans; however, should NOT be in-the-loop. Agent knowledge must be self-correcting whenever stale or incorrect information is surfaced. 3. 𝗪𝗲 𝗮𝗿𝗲 𝘀𝗼 𝗲𝗮𝗿𝗹𝘆. As Jeff Huber pointed out, we are still in the "prompt engineering era" of LLM Wikis and agent knowledge. Prompts still drive so much of LLM wiki success. We don't have higher-level abstractions like harnesses to build upon. We also don't have robust ways of measuring ROI (yet). Thank you Harrison Chase for moderating and Brace Sproul Jeff Huber for the 🌶️ takes and insights! Link to the YouTube video in the comments