Case Study · Industrial / Remote operation

Kalmar: Chooyu 32″ PoC for Remote-Operated Cranes

A remote-operation view with depth restored

Finnish port-equipment company Kalmar, together with telecom operator Elisa and MOPIC, completed a PoC in which stereo camera video from a crane cabin is sent over 5G and viewed in 3D, without glasses, at a remote operation console. The console stayed as it was; only one monitor was swapped for a Chooyu 32″ glasses-free 3D display. Follow-up testing with crane operators is now under way.

Ports are increasingly operating container cranes from remote operation stations instead of from the cabin. It moves operators out of high cabins and hazardous areas into an office, and lets one person take turns handling several machines. But all a remote operator sees is the video from cameras mounted on the crane.

The problem is that this video is flat. When picking up or setting down a container, the operator has to judge the distance between the spreader and the container, the stack height, and the clearance to nearby obstacles, and on a flat image that sense of distance disappears. The depth a cabin operator perceives naturally with two eyes has to be replaced with multiple screens and indirect cues.

3D can bring back that sense of distance, but the conditions are demanding. Operators work shifts and watch screens for hours, so 3D glasses or headsets are not a realistic option, and if the image lags behind reality it becomes dangerous, so latency is a precondition too. With the crane and the operation station far apart, the video has to travel over a network and still arrive in real time.

Kalmar, Elisa and MOPIC split the PoC into clear roles. Kalmar provided the crane, the stereo camera and the remote operation console; Elisa provided the 5G network; MOPIC provided the Chooyu 32″ display and the playback software. The crane and its control logic stayed unchanged; MOPIC's components came in at the last step, where the video reaches the operator — the display.

How it fits together

A remote operation station suits glasses-free 3D well. The operator's seat and the distance to the screen are fixed, and since one person watches the screen, Eye-tracking can follow their viewpoint to raise 3D quality. It is the same condition as Renault's rear-view system; the difference is that the video arrives over 5G and the cloud rather than through a cable inside the vehicle.

After the PoC, follow-up testing with real operating staff is under way. The solution is first tested in a virtual environment with an interim evaluation, then on real cranes at the Kalmar Automation test yard in Tampere, Finland. Technicians, crane operators and crane trainers use it directly, with the goal of confirming higher user acceptance and less eye fatigue than other solutions. The evaluation criteria are initial setup complexity, general user experience, eye fatigue, latency and safety (150 ms or less), added value, and accuracy / failure rate.

Live 3D remote-operation view over 5G, verified

The PoC confirmed that stereo camera video from the crane cabin can travel over 5G and the cloud and appear live on the Chooyu 32″ at the remote console. Operators could see the crane's working area in 3D from their seat, without glasses.

Console unchanged, one monitor replaced

The setup was complete once one monitor was replaced with a Chooyu 32″ and the playback software was installed, with no changes to the crane or the console structure. That keeps the cost of adding 3D to a remote operation system already in service low.

On to real-world validation

Through virtual-environment testing and on-site testing at the test yard, the user experience, eye fatigue and latency threshold are being validated with real operating staff. The same setup can move beyond port cranes to other remote operation systems where equipment is controlled from camera video.

It is not limited to port cranes. Any remote operation where someone moves equipment while watching camera video can use the same structure.