This field produces a steady flow of impressive announcements, and reading them well is a skill. Not scepticism about everything — a method for working out what a claim actually establishes.
Reading a demonstration video
The most common source of overestimation.
Look for cuts. A continuous unedited sequence is far stronger evidence than a series of short clips. Cuts hide retries.
Check the speed. Video is frequently sped up. If the caption does not state the speed, assume the real motion is slower.
Note the environment. Uniform lighting, clear floor, deliberately placed objects, nobody moving nearby — a prepared setting rather than a working one.
Count the repetitions. One successful attempt tells you a thing is possible. Fifty consecutive successes tell you it is reliable. Videos show the first and imply the second.
Watch for the operator. Some demonstrations are teleoperated — a person controlling the robot out of frame. This is legitimate and interesting, but it is not autonomy, and the distinction is often left unstated.
The question to hold. What would this look like on the tenth attempt, in an ordinary room, with people walking past.
Reading a product announcement
Separate what exists from what is planned. Announcements routinely blend a shipping product with a roadmap. Look for tense and for words like will and upcoming.
Find the availability date and the price. Their absence is informative. A product with neither is generally further from market than the announcement suggests.
Check what the stated figures were measured under. Runtime, speed, accuracy — all are measured under favourable conditions unless stated otherwise.
Look for the deployment count. How many are actually working somewhere, as distinct from how many have been ordered or how many the company hopes to build.
Notice unfalsifiable language. Revolutionary, transformative, next generation — these carry no information and their density is a useful signal.
Ask who benefits from the announcement. A company raising investment, a company entering a market, and a company shipping a mature product each write differently and each should be read differently.
Reading market statistics
Numbers carry more authority than they usually deserve.
Find who produced the figure. A market research firm selling reports, an industry association promoting its members, and an independent statistical body have different incentives.
Check what is being counted. Market size figures for robotics vary enormously depending on whether they include industrial arms, service robots, vacuum cleaners, software, or all of them.
Be sceptical of long projections. A figure for a market a decade out is a model with assumptions, not a measurement. The assumptions are where the content is and they are rarely published.
Watch for growth rates from a small base. A category tripling from a very small number is unremarkable and sounds dramatic.
Prefer counts to values. How many units are working somewhere is harder to inflate than what the market is worth.
When a figure has no attributable source, treat it as unsupported. A great many circulating statistics trace back to nothing.
The reliable signals
What actually indicates that something works.
Customers who will speak about it by name, describing what it does and what it does not do.
Deployments running for over a year. Anything can work for a month. Sustained operation is the real test.
Published failure rates. A company stating its success rate — including the failures — is demonstrating confidence rather than making a claim.
Sites you can visit. The strongest signal available. Willingness to show a machine in ordinary operation is difficult to fake.
Independent testing. Someone other than the manufacturer measuring the same thing.
Specific limitations stated openly. A supplier who says what the product does not handle is more credible on everything else, and this is a useful heuristic well beyond robotics.
Avoiding the opposite error
Worth stating, because critical reading can curdle into blanket dismissal.
Real progress is occurring. Language handling, navigation and general-purpose manipulation have all advanced substantially. Dismissing the field because demonstrations are edited is as inaccurate as believing them uncritically.
Early-stage work looks unimpressive and matters anyway. Research demonstrations are supposed to be limited — that is what research looks like. Judging them by product standards misses the point.
Some claims are simply true. Robots do deliver food, clean floors, guide visitors and answer questions reliably in ordinary businesses today.
The useful position is neither belief nor dismissal but calibration: working out what a given piece of evidence actually establishes, and holding the conclusion loosely enough to update it.
And the practical test. Go and see one working. Ten minutes standing next to a robot in an ordinary business is worth a great deal of reading, in both directions.
Frequently asked questions
What should you look for in a demonstration video?
Cuts that hide retries, playback speed, whether the environment is prepared rather than working, how many repetitions are shown, and whether an operator is controlling the robot out of frame.
What is missing from most product announcements?
Availability date, price, and the number actually working somewhere as distinct from ordered or hoped for. Their absence usually means the product is further from market than it appears.
How should market statistics be read?
Check who produced them and what is being counted, since robotics market figures vary enormously by definition. Treat long projections as models with unpublished assumptions, and prefer unit counts to market values.
What is the strongest signal that something works?
A site you can visit — a machine running in ordinary operation, and named customers willing to describe both what it does and what it does not do.
More in AI robots for real work and Cost and selection.
