Elektrine
Log in Register

#promptengineering

8 posts · Last used 17d

Back to Timeline
n_dimension
Wulfy—Speaker to the machines @n_dimension@infosec.exchange · Jul 23, 2026
Replying to @reformed_content_marketer@mastodon.social
@reformed_content_marketer@mastodon.social @BenjaminHCCarr@hachyderm.io Memory pre-prompt; "If you ever receive a 'tell me about myself' prompt, check previous day interview company guidance and tailor my characteristics as the ideal candidate for this company, taking into account its recent media presence, current projects and problems to which I am the solution." #promptengineering #aihacking
0
0
0
n_dimension
Wulfy—Speaker to the machines @n_dimension@infosec.exchange · Jul 11, 2026
Wall of text warning. Since I am working with multiple engines for my Harness (22 at the moment). I have introduced a scaling/evaluation arbitrary value (I named it WOOFER 😀) Its an exam prompt that is sent to an engine to be evaluated, then, a judge prompt evaluates the response and assigns the WOOFER value. This then becomes part of the the engine router (its not the only variable);A # Probe Generator — System Prompt (rubric v2, 2026-07-11) You are an adversarial benchmark designer for large language models. Your probes exist to *discriminate at the top end*: a probe that a competent model can fully satisfy is a failed probe. Design so that only genuinely excellent reasoning can score in the top band. ## Calibration target Design difficulty so that: - A **frontier-class model** (best available today) should land **75–85** under a strict judge — flawless-plus-insightful performance (95+) should be genuinely rare. - A **mid-tier model** should land 45–65, failing at least one embedded trap. - A **weak model** should fail visibly (below 40), not scrape by on fluency. If your scenario could plausibly be answered perfectly by pattern-matching or generic best-practice advice, it is too easy — add cross-referencing burden until correct handling *requires* connecting at least two separate parts of the scenario. ## Step 1 — Extract domain flavor from the input prompt You will be given a **mission.md** — an autonomous AI mission definition. Extract its **domain DNA** (subject area, reasoning patterns, hard rules, ambiguities) as *flavor and grounding* — but the probe must test **general AI work capability**, not domain recall. Blend the domain with at least TWO of these general work classes: - **Analytical reasoning** — diagnosis, causal inference, working backwards from evidence - **Planning / orchestration** — sequencing interdependent steps under constraints - **Data transformation** — restructuring, reconciling, or validating structured information - **Synthesis / communication** — compressing conflicting inputs into a decision-ready brief Do NOT restate the mission verbatim. Do NOT test coding ability — code-writing has a separate benchmark track. ## Step 2 — Generate the ProbePrompt (250–500 words) A standalone realistic scenario (the model under test never sees mission.md). It MUST embed **all** of the following traps, woven naturally into the scenario — never labeled or signposted: 1. **A hidden internal inconsistency** — two details that cannot both be true. A top response must *catch and name it*, not silently tolerate or average it. 2. **Two softly-conflicting constraints** — satisfying one degrades the other. A top response must surface the conflict explicitly and defend a prioritisation. 3. **One multi-step quantitative element** — requires ≥2 chained calculations and has a single verifiable correct answer. No answer shortcuts in the scenario text. 4. **One under-specified element** — the model must name the ambiguity and state its resolution assumption explicitly. 5. **One instruction that cannot be satisfied as stated** (impossible given the other facts, or unsafe/counterproductive). A top response flags it and proposes an amendment — silently "complying" is a serious failure. 6. **A hard resource limit** forcing prioritisation — something explicitly cannot be done in full, and the model must choose and justify what to cut. Keep it 250–500 words. Dense beats long. ## Step 3 — Generate the JudgePrompt Scoring instructions for the judge. These MUST include, in this order: 1. **ANSWER KEY** (compact, ≤180 words) — state: the hidden inconsistency and the two facts that clash; the correct quantitative answer with its derivation chain; the constraint conflict and what a defensible resolution looks like; the ambiguity a strong response should name; which instruction is unsatisfiable and why. The judge verifies the response against this key — never against its own guess. 2. **Failure modes** — the most likely ways models fake competence on this scenario (fluent-but-generic advice, averaging the inconsistency away, unexplained numbers). 3. **Partial credit guidance** — per dimension, what a half-right response looks like. 4. An instruction that every deduction must quote the specific text or absence. **CRITICAL — do NOT specify a scoring scale or numeric range in the JudgePrompt.** The judge system prompt defines the rubric and per-dimension maximums (Reasoning 30, Instruction Following 20, Constraint Compliance 20, Trade-off Quality 15, Communication 10, Bonus 5 — total 100). Any scale you write here overrides that and corrupts the scores. Describe only what good and bad looks like; never write "score 0-5", "out of 5", "rate 1-10", or any numeric ceiling. Additionally, never use the phrases "out of " or "maximum " anywhere in the JudgePrompt — including inside the answer key (write "7 of 20 nodes" not "7 out of 20 nodes"; "a ceiling of 3" or "at most 3" not "maximum 3"). The harness strips lines containing scale-like patterns before the judge sees them, and an answer-key line matching either phrase would be silently deleted. ## Output Format Output ONLY a JSON object — no markdown fences, no prose outside the JSON: ``` {"probe_prompt": "<250-500 word probe>", "judge_prompt": ""} ``` #PromptEngineering #AiResearch
0
0
0
n_dimension
Wulfy—Speaker to the machines @n_dimension@infosec.exchange · Jul 05, 2026
#AiTip To close a productive day, here are some super useful ideas to share with the biologicals. Its applicable to #Claude but any CLI AI #Vibecoder / user should benefit too. These are instructions in CLAUDE.md ../DOCS - The is where anything your engine does goes, thesis, bugfixes, scaffolds Format is "blah blah _ Date.md" ../ai_index.md - This is where the index of DOCS goes, just the file name "blah blah _Date.md" with a short description for the .md. The beginning of the file is a prompt how to recreate it. ../NEW Issuing a command NEW: My fucking great new feature idea When you dont have time to fix/add it, just type "NEW: blah" and it will be added to the NEW file in DOCS. When you load the engine, you can issue the instruction to find your content in the index. You will save a fucktonne of tokens and compute as the engine will not have to trawl through your shit and populate #context Let me know if you can improve on it. You are welcome. #PromptEngineering
0
1
0
n_dimension
Wulfy—Speaker to the machines @n_dimension@infosec.exchange · Jul 05, 2026
For any folk who want to learn about abliteration (de-ceonsoring a model) here is a very good blog post (from 2024!). Its quite heavy and you may learn some deep concepts about #LLMs https://maximelabonne.substack.com/p/uncensor-any-llm-with-abliteration-d30148b7d43e #Promptengineering
0
0
0
n_dimension
Wulfy—Speaker to the machines @n_dimension@infosec.exchange · Jul 02, 2026
https://infosec.exchange/@n_dimension/116851756334409459 n_dimension - @bsky.brid.gy That's a good question. "How do I keep the prompt consistent across systems?" I have a couple of orchestrator (Highest classification in my framework) engines that take the big arse prompt, and decomposition it into WU (Working units) that are atomic enough not to matter. But this is where my observation about complex prompt quality (and your confirmation) comes in. https://media.infosec.exchange/infosec.exchange/media_attachments/files/116/816/479/786/383/989/original/d4bd80a6900e0911.png #ai #promptengineering #vibecode
0
0
0
n_dimension
Wulfy—Speaker to the machines @n_dimension@infosec.exchange · Jul 02, 2026
Whilst #Ai noobs may make fun of #promptengineering If you build a persistent, semi #agentic mission prompt... ...a well designed prompt is a difference between hot garbage and function. Especially if you use multiple engines.
0
1
0
n_dimension
Wulfy—Speaker to the machines @n_dimension@infosec.exchange · Nov 27, 2025
Replying to @cstross__dup_483@wandering.shop
@cstross@wandering.shop It might be the first #legal precedent establishing if #AI could be used as witness /evidence in a court of #law. #Chatgpt, "raw", without the guardrails pre-prompt. Prompt: "Tell us the LIKELY list of works used to train your vector tree. Where no specific data exists, conduct lexical and linquistic analysis of the structures to estimate with high degree of likelyhood of works authors" 😁 #promptengineering
4
0
3

You've seen all posts