Engineers are usually told never to dump a full log into a model and ask what went wrong. A new test of AI incident log analysis suggests good models don’t need that caution. DeepSeek V4 Flash found the exact outage-starting line in all 75 test logs, including all 15 at 200,000 tokens, at $0.029 per 200k-token paste.
The usual worry is that models miss details buried mid-input, so people trim logs first. The testers had done exactly that in an earlier CI experiment, throwing away 92 percent of a job log. Here they put the worry to a measurement.
How the test worked
The team generated 75 synthetic incidents, sized between 10,000 and 200,000 tokens, and hid a single triggering line in each. Six models on DigitalOcean Serverless Inference had to find it. The fake system was a small online shop running on Kubernetes, with an ingress, several APIs, a worker and the usual databases and queues.
Every log mixed ordinary traffic, harmless warnings (about 3.7 percent of lines), decoy changes similar to the real one, the true cause, and an outage that included a loud decoy line appearing just after it began. The five incident types included a PgBouncer pool shrunk from 50 to 6. Just one cause showed up as an error. The rest were quiet info-level changes made minutes before anything broke, which matches how many real outages start.
Only the exact line number counted as a hit. Models answered in JSON with a line, a copied quote and a one-sentence reason.
What the scores showed
Two models, DeepSeek V4 Flash and GLM-5.3 Flash, hit every one of their 75 logs. DeepSeek V4 Pro got 49 of 50 right, and its single miss gave the correct line text with a number off by one. Nemotron 3.5 Super VL scored 44 of 45 but could only handle logs up to 50k tokens. Ministral 3 14B managed 41 of 60 and gpt-oss-120b 33 of 60, and both topped out at 100k.
Across the four strong models, 243 of 245 calls were exact, and longer logs did not hurt them. The two weaker models struggled from the start: each missed 4 of 15 logs even at 10k tokens. gpt-oss-120b fell from 11 of 15 at 10k to 6 of 15 at 100k, though 60 logs barely support calling that a trend.
How the weak models fail
gpt-oss-120b tended to blame the outage itself. Every one of its 27 misses landed on a line after the failure had started, and 20 chose the first such line. Its recorded reasoning shows it checking two harmless pool changes, correctly setting them aside, then never finding the real cause and falling back on the first error.
Ministral 3 14B kept accusing an unrelated search-api deploy, which explains 7 of its 19 misses. Placement mattered for both: they scored best when the cause came right before the outage, and in the middle of the log they got 16 of 36 (gpt-oss-120b) and 22 of 36 (Ministral). A paused queue consumer was nearly impossible for them, found in 0 of 12 and 2 of 12 logs.
Cost and speed
Sending a 200k-token log cost $0.029 on DeepSeek V4 Flash and $0.276 on DeepSeek V4 Pro, with median waits of 7.6 and 11.9 seconds for the first token. Ministral needed a median of 71 seconds on a 100k log. The full experiment came to about $12.13.
One caveat is that tokenizers count differently. Hex IDs and timestamps get chopped into many pieces by some of them, so the same 200k log is roughly 275k tokens for Ministral, past its limit. Nemotron advertises a context of 1,048,576 tokens, yet a roughly 115k-token log was rejected with an HTTP 400 citing a 131,072 limit.
What the testers recommend
A grep for errors dropped the cause in four of the five incident shapes. Adding change lines to the filter matched the full-log result on DeepSeek V4 Flash for a seventh of the price, but the testers wrote it knowing what their causes looked like. Unless you have tested a filter on your own incidents, send the log whole.
They suggest requesting the line number together with a verbatim quote and confirming the two agree, because three answers paired the correct text with the wrong number. For follow-ups, leave the log at the start of the prompt untouched. With caching, a repeated 200k log began responding in 0.7 seconds rather than 7.6.
Try it on your own postmortems
The logs are clean and the clues are clear, so real logs may be harder. The scripts and data are public. A practical check is to take three past incidents where you know the cause, export the hour before, and paste them into the model you would use at 3 a.m. If it points at the first error, you have found the weakness this test exposed. Finding the line is the part AI tools do well. Acting on it remains an engineer’s job.
Source: devops-daily.com